Preparing the Switch for Configuring FCoE

Information About FCoE

FCoE provides a method of transporting Fibre Channel traffic over a physical Ethernet connection. FCoE requires the underlying Ethernet to be full duplex and to provide lossless behavior for Fibre Channel traffic.


Note

Lossless behavior on Ethernet is provided by using a priority flow control (PFC) mechanism that prevents packet loss during congestion conditions.

The Cisco NX-OS software supports T11-compliant FCoE on all 10-Gigabit and 40-Gigabit Ethernet interfaces.

FCoE and FIP

FCoE Initiation Protocol

The FCoE Initialization Protocol (FIP) allows the switch to discover and initialize FCoE-capable entities that are connected to an Ethernet LAN. Cisco NX-OS switches support the Converged Enhanced Ethernet Data Center Bridging Exchange (CEE-DCBX) protocol for T11-compliant Generation 2 CNAs.

The following switches do not support Pre-FIP on Generation 1 CNAs:

  • Cisco MDS 9700

  • Cisco MDS 9500

  • Cisco MDS 9250i

FIP Virtual Link Instantiation

FIP is used to perform device discovery, initialization, and link maintenance. FIP performs the following tasks:

  • FIP VLAN discovery— Discovers the FCoE VLAN that will be used by all other FIP protocols as well as by the FCoE encapsulation for Fibre Channel payloads on the established virtual link. FIP VLAN discovery occurs in the native VLAN used by the initiator or target to exchange Ethernet traffic. The FIP VLAN discovery protocol is the only FIP protocol running on the native VLAN; all other FIP protocols run on the discovered FCoE VLANs.

  • FIP FCF discovery—When a FCoE device is connected to the fabric, it sends a Discovery Solicitation message. A Fibre Channel Forwarder (FCF) or a switch responds to the message with a Solicited Advertisement that provides an FCF MAC address to use for subsequent logins.

  • FCoE virtual link instantiation— FIP defines the encapsulation of fabric login (FLOGI), fabric discovery (FDISC), logout (LOGO), and exchange link parameter (ELP) frames along with the corresponding reply frames. The FCoE devices use these messages to perform a fabric login.

  • FCoE virtual link maintenance— FIP periodically sends maintenance messages between the switch and the CNA to ensure that the connection is still valid.

FCoE Frame Format

FCoE is implemented when the switch encapsulates a Fibre Channel frame in an Ethernet packet with a dedicated Ethernet type, 0x8906. The packet has a 4-bit version field. The other header fields in the frame (the source and destination MAC addresses, VLAN tags, and frame markers) are all standard Ethernet fields. Reserved bits pad the FCoE frame to the IEEE 802.3 minimum packet length of 64 bytes.

A Fibre Channel frame consists of 36 bytes of headers and up to 2112 bytes of data for a total maximum size of 2148 bytes. The encapsulated Fibre Channel frame has all the standard headers, which allow it to be passed to the storage network without further modification. To accommodate the maximum Fibre Channel frame in an FCoE frame, the class-fcoe is defined with a default Maximum Transmission Unit (MTU) of 2240 bytes.

VLAN Tagging for FCoE Frames

The Ethernet frames that are sent by the switch to the adapter include the IEEE 802.1Q tag. This tag includes a field for the class of service (CoS) value used by the priority flow control (PFC). The IEEE 802.1Q tag also includes a VLAN field.

The switch expects frames from a FIP T11-compliant CNA to be tagged with the VLAN tag for the FCoE VLAN. Frames that are not correctly tagged are discarded.


Note

You cannot map VLAN 1 or the native VLAN to an FCoE VSAN.

FIP Ethernet Frame Format

FIP is encapsulated in an Ethernet packet with a dedicated EtherType, 0x8914. The packet has a 4-bit version field. Along with the source and destination MAC addresses, the FIP packet also contains a FIP operation code and a FIP operation subcode. The following table describes the FIP operation codes and subcodes.

Table 1. FIP Operation Codes

FIP Operation Code

FIP Subcode

FIP Operation

0x0001

0x01

Discovery Solicitation

0x02

Discovery Advertisement

0x0002

0x01

Virtual Link Instantiation Request

0x02

Virtual Link Instantiation Reply

0x0003

0x01

FIP Keepalive

0x02

FIP Clear Virtual Links

0x0004

0x01

FIP VLAN Request

0x02

FIP VLAN Notification

DCBX

Data Center Bridging Exchange Protocol

The Data Center Bridging Exchange (DCBX) protocol is an extension of the Link Layer Discovery Protocol (LLDP). DCBX end points exchange request and acknowledgment messages. For flexibility, parameters are coded in a type-length-value (TLV) format. Cisco NX-OS switches support the Converged Enhanced Ethernet Data Center Bridging Exchange (CEE-DCBX) is supported on all T11-compliant Generation 2 CNAs.

DCBX runs on the physical Ethernet link between the switch and the CNA. By default, DCBX is enabled on Ethernet interfaces. When an Ethernet interface is brought up, the switch automatically starts to communicate with the CNA.

During the normal operation of FCoE between the switch and the CNA, DCBX provides link-error detection.

DCBX is also used to negotiate capabilities between the switch and the CNA and to send configuration values to the CNA.

The CNAs that are connected to a switch are programmed to accept the configuration values that are sent by the switch, allowing the switch to distribute configuration values to all attached CNAs, which reduces the possibility of configuration errors and simplifies CNA administration.

Lossless Ethernet

Standard Ethernet is a best-effort medium which means that it lacks any form of flow control. In the event of congestion or collisions, Ethernet drops packets. The higher level protocols detect the missing data and retransmit the dropped packets.

To properly support Fibre Channel, Ethernet has been enhanced with a priority flow control (PFC) mechanism.

Logical Link Up/Down

On a native Fibre Channel link, some configuration actions (such as changing the VSAN) require that you reset the interface status. When you reset the interface status, the switch disables the interface and then immediately reenables the interface.

Caution:

If an Ethernet link provides FCoE service, do not reset the physical link because this action is disruptive to all traffic on the link.

The logical link up/down feature allows the switch to reset an individual virtual link. The logical link down is signaled with a FIP Clear Virtual Link message.

Caution:

If the CNA does not support the logical link level up/down feature, the CNA resets the physical link, which means that all traffic on the Ethernet interface is disrupted.

Converged Network Adapters

Cisco NX-OS switches support the following CNA types:

  • Hardware adapter

    • Works with the existing Fibre Channel host bus adapter (HBA) driver and Ethernet Network Interface Card (NIC) driver in the server.

    • Server operating system view of the network is unchanged; the CNA presents a SAN interface and a LAN interface to the operating system.

  • FCoE software stack

    • Runs on existing 10-Gigabit Ethernet adapters.

The following Cisco NX-OS series and platforms support Generation 2 CNAs that use the FIP to exchange information about its available capabilities and to negotiate the configurable values with the switch:

  • Cisco MDS 9700

  • Cisco MDS 9500

  • Cisco MDS 9250i

To reduce configuration errors and simplify administration, the switch distributes the configuration data to all the connected adapters.

STP Lite

FCoE does not require full Spanning Tree Protocol (STP) because FCoE has no bridging functionality, which means that no STP loops are created in the network. STP Lite on FCoE interfaces ensures rapid convergence across the network by sending an agreement Bridge Protocol Data Unit (BPDU) whenever it receives a proposal BPDU. The FCoE link sends the identical agreement BPDU in response to either an Multiple Spanning Tree (MST) or a Per VLAN Rapid Spanning Tree Plus (PVRST+) proposal BPDU. Additionally, STP Lite suppresses the MAC address flushing function for FCoE VLANs.

STP Lite is enabled automatically by default across the entire device for FCoE VLANs as soon as the first FCoE VLAN comes up. At the same time, the system automatically converts all FCoE links as the STP-type normal ports. This feature runs only in FCoE VLANs.

Default Settings for FCoE

This table lists the default settings for FCoE parameters.

Table 2. Default FCoE Parameter Settings

Parameters

Default

FCoE feature

Not installed, disabled

FC-Map

0E.FC.00

Fabric priority

128

Advertisement interval

8 seconds

Guidelines and Limitations

FCoE

  • You cannot enable FCoE on the default VLAN.

  • FCoE is already enabled on a Cisco MDS switch.

  • The QoS policy must be the same on all Cisco FCoE switches in the network.

  • Cisco MDS 9700 Series, MDS 9500 Series, and 9250i switch do not support LAN traffic and only accepts and processes FCoE frames.

  • On Cisco MDS 9500 Series, you must install a Supervisor-2A module to configure FCoE.

  • You can install an FCoE module in a fabric that uses FC-Redirect services but do not install the FCoE module in the same switch where FC-Redirect services application modules are enabled.

  • SME does not support any FCoE connected devices including devices connected through the MDS FCoE linecard (DX-X9708-K9).

  • In Cisco MDS NX-OS Release 5.2(x), you cannot install a FCoE module in a switch that is running DMM, SME, or IOA.

  • Cisco MDS 9250i supports a maximum number of up to 32 VSANs.

  • MDS 9250i does not support FCoE Switched Port Analyzer (SPAN).

  • MDS 9250i does not support FCoE port diagnostics, such as generic online diagnostics (GOLD) and Online Health Management System (OHMS).

  • Cisco MDS 9500 switches support only the 8 port 10 Gbps FCoE Module (DS-X9708-K9). Cisco MDS 9700 switches support the following modules:

    • 48 port 10 Gbps FCoE Module (DS-X9848-480K9)

    • 24 port 40 Gbps FCoE Module (DS-X9824-960K9)

  • Cisco MDS 9000 does not support 10-Gigabit or 40-Gigabit FEX connectivity on FCoE modules.

  • We recommend that you use a minimum of five fabric modules (FABs) to receive complete throughput of the MDS 40-G line cards across 24 ports. Usage of three of four fabric modules might result in the oversubscription of ports.

  • When you configure a multi-hop FCoE1, ensure that you use the same no-drop classes on both sides. Priority flow control does not work when you use different no-drop classes. Use the show interface priority-flow-control command to verify the priority flow control operation.

Multi-Hop FCoE Distance Configuration

In Multi-Hop FCoE, when a device sends a pause, the interface that generates the pause frame must have an ingress queue with a buffer space large enough to buffer twice the link distance. This is because, when the pause is generated the wire might get congested. By the time the adjacent device receives or processes the generated pause frame, the wire might get congested again. Therefore, the device that generates the pause must have the ability to buffer twice the link distance.

The 48 port 10 Gbps FCoE Module (DS-X9848-480K9) line cards support a long haul lossless distance of up to 80 kilometers. In Cisco NX-OS Release 6.2(9) and later, you can change the ingress queuing buffer configuration.

The 24 port 40 Gbps FCoE Module (DS-X9824-960K9) line cards support long haul lossless distance of up to 40 kilometers. In Cisco NX-OS Release 7.3(x) and later, you can change the ingress queuing buffer configuration.

Enabling and Disabling FCoE

Enabling FCoE

The Cisco MDS 9700 Series, MDS 9500 Series, and MDS 9250i switches automatically enables the FCoE feature when an FCoE-capable module is present in the chassis. The Cisco MDS 9250i switches already have fixed FCoE ports. These FCoE ports can be enabled only if a minimum of two Power Supply Units (PSUs) are online.

Before you begin

Do not enable FCoE on VLAN 1 or the native VLAN.

Procedure

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

(Optional) feature lacp

Example:

switch(config)# feature lacp

Enables the Link Aggregation Control Protocol (LACP) feature. This feature is considered a best practice for FCoE operation.

Step 3

system qos

Example:

switch(config)# system qos

Enters Quality of Service (QoS) configuration mode.

Step 4

service-policy type network-qos default policy-name

Example:

On Cisco MDS 9700 Series switches: 
switch(config-sys-qos)# service-policy type network-qos default-nq-7e-4q8q-policy

Example:

On Cisco MDS 9500 Series switches: 
switch(config-sys-qos)# service-policy type network-qos default-nq-7e-policy

Example:

On Cisco MDS 9250i Multiservice switch: 
switch(config-sys-qos)# service-policy type network-qos default-nq-4e-2q4q-policy

Enables the QoS policy that supports FCoE traffic. The policy-name default is different for each Cisco MDS switch.

Step 5

(Optional) show feature

Example:

switch(config-sys-qos)# show feature

Displays information about the enabled features.

Step 6

(Optional) copy running-config startup-config

Example:

switch(config-sys-qos)# copy running-config startup-config

Copies the running configuration to the startup configuration.

Disabling FCoE

FCoE is enabled by default on supported Cisco MDS 9000 series switches. You cannot disable FCoE on these switches.

Configuring the FC-Map

You can prevent data corruption due to cross-fabric talk by configuring an FC-Map that identifies the Fibre Channel fabric for this switch. When the FC-Map is configured, the switch discards the MAC addresses that are not part of the current fabric. An FCF can assign Fabric Provide MAC Addresses (FPMA) to the CNAs consisting of the FC-Map Value for the Fabric and the Fibre Channel ID (FCID) assigned during Fabric Login


Note

In most deployments, changing the fc-map is not needed. We recommend that you use VSAN to VLAN mapping. For more information, see the Configuring FCoE VLANs and Virtual Fibre Channel Interfaces chapter.

Procedure

Step 1

configure terminal

Example:

fcoe# configure terminal
fcoe(config)#

Enters configuration mode.

Step 2

fcoe fcmap fabric-map

Example:

fcoe(config)# fcoe fcmap 0x0efc2a

Configures the global FC-Map. The default value is 0x0EFC00. The range is from 0x0EFC00 to 0x0EFCFF. Use the no fcoe map command to reset to the default value.

Example

This example shows how to configure the global FC-Map on a Cisco MDS 9700, or MDS 9500, or an MDS 9250i switch: 

switch# configure terminal
switch(config)# fcoe fcmap 0x0efc2a

Configuring the Fabric Priority

The FCoE switch advertises its priority. The priority is used by the CNAs in the fabric to determine the best switch to connect to.

Procedure

Step 1

configure terminal

Example:

fcoe# configure terminal
fcoe(config)#

Enters configuration mode.

Step 2

fcoe fcf-priority fabric-priority

Example:

fcoe(config)# fcoe fcf-priority 42

Configures the global fabric priority. The default value is 128. The range is from 0 (higher) to 255 (lower). Use the no fcoe fcf-priority command to reset the global fabric priority to the default value.

Example

This example shows how to configure the global fabric priority for a Cisco MDS 9700, MDS 9500, and MDS 9250i switch: 

switch# configure terminal
switch(config)# fcoe fcf-priority 42

Setting the Advertisment Interval

You can configure the interval for Fibre Channel fabric advertisement on the switch.

Procedure

Step 1

configure terminal

Example:

fcoe# configure terminal
fcoe(config)#

Enters configuration mode.

Step 2

fcoe fka-adv-period interval

Example:

fcoe(config)# fcoe fka-adv-period 8
fcoe#

Configures the advertisement interval for the fabric. The default value is 8 seconds. The range is from 4 to 60 seconds.

Example

This example shows how to configure the advertisement interval for the fabric on a Cisco MDS 9000 switch: 

switch# configure terminal
switch(config)# fcoe fka-adv-period 42

Disabling LAN Traffic on an FCoE Link

You can disable LAN traffic on an FCoE link.

DCBX allows the switch to send a LAN Logical Link Status (LLS) message to a directly-connected CNA. Enter the shutdown lan command to send an LLS-Down message to the CNA. This command causes all VLANs on the interface that are not enabled for FCoE to be brought down. If a VLAN on the interface is enabled for FCoE, it continues to carry SAN traffic without any interruption.

Procedure

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters configuration mode.

Step 2

interface ethernet slot/port

Example:

switch(config)# interface e 2/1
switch(config-if)#

Specifies an interface to configure, and enters interface configuration mode. Use ? to view a list of supported interfaces.

Step 3

shutdown lan

Example:

switch(config-if)# shutdown lan

Shuts down Ethernet traffic on the interface. If the interface is part of an FCoE VLAN, the shutdown has no impact on the FCoE traffic. Use no shutdown lan to reenable Ethernet traffic on this interface.

Step 4

(Optional) show interface

Example:

switch(config-if)# show interface

Displays information about the interface.

Step 5

(Optional) copy running-config startup-config

Example:

switch(config-if)# copy running-config startup-config

Copies the running configuration to the startup configuration.

Additional References for FCoE

Related Documents

Related Topic Document Title

Command reference

Cisco NX-OS FCoE Command Reference Guide, Nexus 7000 and MDS 9500

Configuration guide

Cisco MDS 9000 Series NX-OS Quality of Service Configuration Guide

Cisco NX-OS licensing

Cisco MDS 9000 Series Licensing Guide

Standards and RFCs

Standard/RFC Title
T11 FC BB-5 Fibre Channel Backbone 5

MIBs

MIB MIBs Link

To locate and download MIBs for selected platforms, Cisco IOS releases, and feature sets, use Cisco MIB Locator found at the following URL:

http://www.cisco.com/go/mibs

Technical Assistance

Description Link

The Cisco Support website provides extensive online resources, including documentation and tools for troubleshooting and resolving technical issues with Cisco products and technologies.

To receive security and technical information about your products, you can subscribe to various services, such as the Product Alert Tool (accessed from Field Notices), the Cisco Technical Services Newsletter, and Really Simple Syndication (RSS) Feeds.

Access to most tools on the Cisco Support website requires a Cisco.com user ID and password.

http://www.cisco.com/support

1 A Multihop FCoE switch continues to subscribe to the appropriate fibre channel traffic engineering, domain creation, and forwarding.