Emulex OCE10102 cards

Hello,
Well if you have seen my coworkers blog we have had some issues with Emulex cards and the Nexus 1000V.  So tonight I am going to talk about the Emulex OCE10102 cards and what we saw and what fun we had fixing them.

Part 1 The Cards:
First with the Emulex OCE10102 cards, these are the converged network/FCoE cards, makes absolutely sure you match the driver and firmware version.  If you do not you will run into weird issues with the cards, for instance we have had the card come up on the Nexus1000V but refuse to transmit any traffic on non system vlans.  I know we come from a world of update the driver and everything will be fine.  This is ABSOLUTELY NOT TRUE with these cards, when checking for drivers check for firmware.  VMWARE will show the drivers on their driver page but not the firmware.  Here is the link to Emulex's download site for these converged adapters.  For VMWare they will redirect you to VMWare's site for the driver, but the firmware (or boot code as they call it) will be there.  They now have an installer that works in Windows, ESX 4.1, and ESX 4.0 to update the firmware.  If you are having trouble and are not on the newest drivers and firmware go back, beat yourself, and update them both now.

A special note on the Emulex OCE10102:
Apparently these cards are not compatible with the Dell R900 series of servers.  If you configure one on Dell's website they will sell you the card with it.  So we have 4 Exchange 2003 servers and at least 3 ESX 4.1 servers that are running on the Dell R900 hardware with these Emulex cards.  Our Exchange servers are Blue Screen of Deathing about once a week, if you look in the bios log of the server we see a fatal error in the PCI bus.  Our exchange servers are running Windows 2003 with the latest service pack and patches.  The only response from Dell we can get is these cards are not supported even though they will sell me a R900 with the card in it.  We have used this card in IBM servers and Dell R810s and had no problem after we fixed the driver issue.  Our ESX servers purple screen of death less often but still do. I may have to go to Dell and kick someone soon.

Part 2 configuring converged adapters on a Nexus 5000:

First make sure you have all the drivers installed on your server for the Emulex OCE10102 card.  There are 3 possible drivers, the Ethernet driver, the ISCSI driver, and the FCoE driver.  Your card, depending on the flavor you bought, will either do Ethernet and FCoE or Ethernet and ISCSI, as far as I know it will not do all three. 

ISCSI version:
For Ethernet and ISCSI there is no real special configuration on the switch side for this other than a trunk that allows all vlans to that the card needs.

Example:
This example config puts the ISCSI vlan as 20 and the Ethernet vlan as 30. And we will say the server is plugged into port 1/1.  The funny part about these cards is that if you do not specifically configure the Ethernet drivers for dot1q trunking, it will send out all packets untagged, on the other hand the HBA side of the card will tag the packets.

Nexus 5000 config:
vlan 20
  name ISCSI
  state active
vlan 30
  name ethernet
  state active
interface Ethernet 1/1
  description To ISCSI Ethernet CNA
  switchport mode trunk
  switchport trunk allowed vlan 20,30
  switchport trunk native vlan 30
  no shutdown

Most people using ISCSI want jumbo frames, so I will include the config here to turn them on.  This is stolen directly from this link on Cisco's web site.  We are removing our ISCSI devices in favor of fiber-channel and FCoE and nfs devices so I do not have a working config and will have to trust Cisco got it right.

Jumbo Frames on the nexus 5000:
policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216
    exit
  exit
system qos
  service-policy type network-qos jumbo

What is nice about this config is that you can use Port-Channels to add redundancy and multiple links for throughput.

Now for the config for the FCoE version of the card:
This one is fun and comes with some caveats.  First, the card is not supported in a Port-Channel.  You have to dig around Cisco's website to find it, but it is there.  You will have to create a VLAN and assign it to a VSAN the VLAN has to be local to the switch.  The VSAN can cross multiple devices but as of yet the VLAN has to be local to the device you are working on, I hear rumblings on that changing but we aren't there yet.  You will need to create a VSAN even if you do not have the fiber channel add in cards for the Nexus 5000. So now on to the fun part the configuration.  If you have Cisco's Fabric Manager Server and the Enterprise License on a Cisco MDS series switch you can do some of this in there, after the config I will explain what you can do in the Fabric Manager.

First we need to turn on the FCoE feature, this is a licensed feature so make sure you have the license file loaded and installed.

Nexus 5000 command:
feature fcoe

Next we need to create a vlan and a vsan.

Nexus 5000 commands:
vlan 2
 fcoe vsan 2
 name FCoE_VLAN
 state active
vsan database
 vsan 2 name "FCOE_VSAN"

Now we need to create a virtual interface and bind it to a physical interface, in this example I am again using interface ethernet 1/1.  The bind command will fail if the interface is in a port-channel.

Nexus 5000 commands:
interface vfc1
 bind interface ethernet1/1
 no shutdown

We now need to add the interface vfc1 to the vsan database.

Nexus 5000 commands:
vsan database
 vsan 2 interface vfc1

Finally we need to configure the interface.  Make sure you make this interface a trunk and allow the san vlan and the ethernet vlan as configured above the san vlan is 2 so we will use that here, we will stick with the first example's ethernet vlan being 30.  In this case the above config makes sure that the FCoE packets get on to the proper vlan.  We are assuming the ethernet side of the card is still not configured to tag packets.

Nexus 5000 commands:
interface Ethernet 1/1
 description To CNA with Ethernet and FCoE
 switchport mode trunk
 switchport trunk native vlan 30
 switchport trunk allowed vlan 2,30

Once you have this all up you need to create zones.  I strongly recommend you find the WWN of the cards.  This means if you ever, heaven for bid, move the server from one port to another you will not have to change the zone.  The OCE10102 cards have the WWN of each port on a sticker on the card.  My WWNs are fake.  So you have source and destination. You can only activate one zone on a particular VSAN so you will see I use the AllZones zone to activate my individual zone.  It is good practice to have only 1 initiator per zone, so separate out those zones people, please.  You will save yourself the trouble and in many cases get better performance.

Nexus 5000 commands:
zone name NewZone vsan 2
 member pwwn 11:22:33:44:55:66:77:88
 member pwwn 88:77:66:55:44:33:22:11
zone name AllZones vsan 2
 member NewZone
zoneset activate name AllZones vsan 2

Yay you now have a FCoE Converged CNA configured, this will work for any converged CNA I have been able to test.

Here is the complete command list in one place:
feature fcoe
vlan 2
 fcoe vsan 2
 name FCoE_VLAN
 state active
vsan database
 vsan 2 name "FCOE_VSAN"
interface vfc1
 bind interface ethernet1/1
 no shutdown
vsan database
 vsan 2 interface vfc1
interface Ethernet 1/1
 description To CNA with Ethernet and FCoE
 switchport mode trunk
 switchport trunk native vlan 30
 switchport trunk allowed vlan 2,30
zone name NewZone vsan 2
 member pwwn 11:22:33:44:55:66:77:88
 member pwwn 88:77:66:55:44:33:22:11
zone name AllZones vsan 2
 member NewZone
zoneset activate name AllZones vsan 2

Ok finally in this blog that has become a wall of text I promised what parts you could do in the Cisco Fabric Manager.  The physical interface has to be configured manually.  In the device configuration of the fabric manager you can create and configure the vfc interface and assign it to the VSAN.  I as of yet have not figured out how to noshut the interface, and have to go into the switch command line to do it.  Once you have all the first parts done, sit back for a minute and allow the fabric manager to detect the new WWNs you put on the network.  Once it does you can use the zone manager to configure the individual zones, add it to the zone group, and activate it.  In a later blog post I'll post some screen shots of doing all this.

Happy networking.
--Chris