Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Configuration

Network Diagram

Physical Network

Logical VSAN View

Platform Configurations

Nexus 5000

Quality of Service (QoS) Configuration

UCS Manager

NetApp

Verification

Check the State of VFC Interfaces

Verify NetApp and UCS are Logged Into the Fabric

Troubleshoot

Features Configured

VLAN and VSAN Configuration

Spanning Tree and FCoE VLANs

VPC and FCoE VLANs

Priority Flow Control Status

VSAN Membership

Known and Common Problems

Related Information

Introduction

This document describes a Nexus 5000 in a Virtual Port Channel (VPC) configuration to a NetApp storage to achieve Fibre Channel over Ethernet (FCoE).

Prerequisites

Requirements

Cisco recommends that you have knowledge of Unified Computing System (UCS) and Nexus 5000.

Components Used

The information in this document is based on these software and hardware versions:

- 2x Nexus 5548 - runs NXOS 5.2. Called bdsol-n5548-05 and -06-.

- 2x UCS 6248 fabric interconnects (FIs) - runs 2.2 software. Called bdsol-6248-03-, A and B.

- 2x NetApp 3220 - runs version 8.1. Called bdsol-3220-01-, A and B.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document describes the configuration of the Nexus 5000 switch in a Virtual Port Channel (VPC) configuration to a NetApp storage in order to achieve Fibre Channel over Ethernet (FCoE).

Configuration

Network Diagram

All of the diagrams in this document were created with App Diagrams .

Physical Network

This configuration is meant to be a simple reference. It utilizes single links even though the port channels mentioned can be easily extended with more links to provide more bandwidth.

Between the UCS FIs and the Nexus 5000s there is only a single link on each side. Between each Nexus 5000 and each NetApp there is also one link bound to associated VPCs.

Logical VSAN View

In this case, VSAN 70 is used for the left side of the fabric and VSAN 80 for the right side. Respectively, VLANs 970 and 980 are the FCoE VLANs.

Platform Configurations

The majority of the complexity of this setup is concentrated on the Nexus 5000 platform. The UCS and NetApp configuration is pretty straightforward.

Nexus 5000

This configuration has a lot of similarities to the Nexus 5000 NPIV FCoE with FCoE NPV Attached UCS Configuration Example.

Enable the Features

On both Nexus 5000s:

feature npiv

feature fcoe

feature lacp

Create the Virtual Storage Area Network (VSAN)

bdsol-n5548-05

vsan database

vsan 70

bdsol-n5548-06

vsan database

vsan 80

Add New VLANs to Carry FCoE Traffic

bdsol-n5548-05

vlan 970

fcoe vsan 70

bdsol-n5548-06

vlan 980

fcoe vsan 80

Physical Connectivity Configuration

This type of configuration allows you to have FCoE and another VLAN carried in a single trunk. In this case VPCs carry VLAN 3380, configured as Network File System (NFS) VLAN on the NetApp side as seen later.

The configuration is taken from bdsol-n5548-05.

Configure VPC to NetApp-B:

interface Ethernet1/5

switchport mode trunk

switchport trunk allowed vlan 970,3380

spanning-tree port type edge trunk

channel-group 105 mode active

interface port-channel105

switchport mode trunk

switchport trunk allowed vlan 970,3380

vpc 105

Configure VPC to NetApp-A:

interface Ethernet1/6

switchport mode trunk

switchport trunk allowed vlan 970,3380

spanning-tree port type edge trunk

channel-group 106 mode active

interface port-channel106

switchport mode trunk

switchport trunk allowed vlan 970,3380

vpc 106

Configure Port Channel that Leads to UCS:

interface Ethernet1/3

switchport mode trunk

switchport trunk allowed vlan 970

spanning-tree port type edge trunk

spanning-tree bpdufilter enable

channel-group 103 mode active

interface port-channel103

switchport mode trunk

switchport trunk allowed vlan 970

This configuration is reflected on the bdsol-n5548-06 switch. The only difference is that VLAN 980 is carried in trunks and not 970.

Create and Bind the VSAN Configuration

Now that physical connectivity is configured, the newly configured Virtual Fibre Channel (VFC) interfaces must be bound to the port channels created previously.

Note: This configuration pertains to the UCS when it operates in Fabric End Host mode, not Fibre Channel (FC) switching. When you run FC Switching mode on your UCS, remember to change the VFC mode from default Trunking Fabric (TF) to Trunking E_port (TE).

The bdsol-n5548-05 configuration to NetApp:

interface vfc105

bind interface port-channel105

switchport trunk allowed vsan 70

no shutdown

interface vfc106

bind interface port-channel106

switchport trunk allowed vsan 70

no shutdown

The bdsol-n5548-05 FCoE configuration to UCS:

interface vfc103

bind interface port-channel103

switchport trunk allowed vsan 70

no shutdown

As mentioned in the introduction to this document, this configuration depicts a single link in port channel between UCS and Nexus switches. It is used for reference only and can be extended to additional links.

The previous configuration is reflected on bdsol-n5548-06. The only change is to use VSAN 80 instead of VSAN 70.

Peer Link Port Channel

In this configuration each side of the fabric has only one FCoE VLAN. The VLANs cannot trunked between Nexus 5000s.

Dependent upon the configuration standards and best practices, this can be completed by:

- No additional changes - since VLANs are not created on both Nexus 5000s they cannot be trunked. Those VLANs are seen under "Vlans Err-disabled on Trunk".

- Remove VLANs from the trunk via the switchport trunk allowed vlan except command.

- Explicitly mention which VLANs are allowed -do not mention the FCoE VLANs.

Add the VFCs to the VSAN Database

The newly configured VFCs are added to VSAN database.

bdsol-n5548-05:

vsan database

vsan 70 interface vfc103

vsan 70 interface vfc105

vsan 70 interface vfc106

bdsol-n5548-06:

vsan database

vsan 80 interface vfc103

vsan 80 interface vfc105

vsan 80 interface vfc106

Quality of Service (QoS) Configuration

This configuration has a simple QoS profile, that is meant as a reference.

In order for Priority Flow Control (PFC) to be enabled, which allows loss-less transmission that is necessary for FCoE, the QoS configuration needs to match.

In this case only two classes are used. One for FC/FCoE and one for NFS traffic that allows jumbo frames.

bdsol-n5548-05# show running-config ipqos

class-map type qos class-fcoe

class-map type queuing class-fcoe

match qos-group 1

class-map type network-qos class-fcoe

match qos-group 1

policy-map type network-qos jumbo

class type network-qos class-fcoe

pause no-drop

mtu 2158

class type network-qos class-default

mtu 9216

multicast-optimize

system qos

service-policy type network-qos jumbo

service-policy type queuing input fcoe-default-in-policy

service-policy type queuing output fcoe-default-out-policy

service-policy type qos input fcoe-default-in-policy

UCS Manager

A step-by-step guide that shows you how to set up FCoE Uplinks can be found in UCS FCoE Uplink Configuration Example.

As mentioned previously, in this example a single link is used only, but this configuration can be extended to multiple links in a port channel.

Port Channel Configuration

The previous example shows the UCS point of view. Each side of the fabric has a port channel, number 2, based on port ethernet1/21.

In this example the only VSANs which exist are 70 and 80, so it is safe to assume they are the ones trunked.

bdsol-6248-03-A(nxos)# show vlan fcoe

Original VLAN ID        Translated VSAN ID      Association State

----------------        ------------------      -----------------

970                       70               Operational

4048                      1                Operational

interface Ethernet1/21

description UF: UnifiedUplink

pinning border

switchport mode trunk

switchport trunk native vlan 4049

switchport trunk allowed vlan 1,970,4048-4049

udld disable

channel-group 2 mode active

no shutdown

interface port-channel2

description UF: UnifiedUplink

switchport mode trunk

switchport trunk native vlan 4049

switchport trunk allowed vlan 1,970,4048-4049

pinning border

speed 10000

VLAN 970 is trunked via port channel 2.

QoS Configuration

As mentioned, in the QoS configuration on the Nexus 5000 an end-to-end QoS configuration allows PFC to work.

In this example, only two classes are enabled. The default class which carries jumbo frames (MTU 9216) and the FC class.

This matches what was configured on the Nexus side in previous steps. Other classes are present, but not enabled.

NetApp

In this example NetApp operates in 7-mode. A step-by-step guide on how to set up the NetApp side can be found in the NetApp Storage Deployment Guide.

License

At least the FCP (FC/FCoE) license must be present in order to enable the FCoE to work.

Configuration

The FCoE VLANs do not need to be added anywhere on the NetApp. They are discovered in the VLAN discovery stage of FCoE negotiation.

As mentioned previously, during Nexus 5000 configuration the NFS is handled by same Uplinks also.

bdsol-3220-01-A> rdfile /etc/rc

ifgrp create lacp dvif -b ip e1a e1b

vlan create dvif 3380

ifconfig e0M `hostname`-e0M flowcontrol full netmask 255.255.255.128 partner

e0M mtusize 1500

ifconfig dvif-3380 `hostname`-dvif-3380 netmask 255.255.255.0 partner dvif-3380

mtusize 9000 trusted

route add default 10.48.43.100 1

routed on

The configuration shows that LACP portchannel (ifgroup) is created from interface e1a and e1b and the virtual interface for VLAN 3380 is added.

The same configuration from the GUI shows:

Interface Group Configuration

bdsol-3220-01-A> ifgrp status

default: transmit 'IP Load balancing', Ifgrp Type 'multi_mode', fail 'log'

dvif: 2 links, transmit 'IP Load balancing', Ifgrp Type 'lacp' fail 'default'

Ifgrp Status Up Addr_set

up:

e1a: state up, since 28Sep2014 00:48:23 (142+11:23:01)

mediatype: auto-10g_twinax-fd-up

flags: enabled

active aggr, aggr port: e1b

input packets 766192514, input bytes 2560966346135

input lacp packets 410301, output lacp packets 410438

output packets 615632, output bytes 81875375

up indications 9, broken indications 5

drops (if) 0, drops (link) 1

indication: up at 28Sep2014 00:48:23

consecutive 0, transitions 14

e1b: state up, since 28Sep2014 00:48:22 (142+11:23:02)

mediatype: auto-10g_twinax-fd-up

flags: enabled

active aggr, aggr port: e1b

input packets 246965410, input bytes 702882508932

input lacp packets 410301, output lacp packets 410442

output packets 615646, output bytes 81876343

up indications 9, broken indications 5

drops (if) 0, drops (link) 2

indication: up at 28Sep2014 00:48:22

consecutive 0, transitions 15

Verification

Note: Only registered Cisco users can access internal Cisco tools and information.

Use this section to confirm that your configuration works properly. The Cisco CLI Analyzer  supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.

The Verify and Troubleshoot sections of this document center around Nexus 5000 switches since they are central to this configuration.

Check the State of VFC Interfaces

bdsol-n5548-05# show int vfc 105

vfc105 is trunking

Bound interface is port-channel105

Hardware is Ethernet

Port WWN is 20:68:00:2a:6a:28:68:7f

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port mode is TF

Port vsan is 70

Trunk vsans (admin allowed and active) (70)

Trunk vsans (up) (70)

Trunk vsans (isolated) ()

Trunk vsans (initializing) ()

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

65 frames input, 6904 bytes

0 discards, 0 errors

65 frames output, 9492 bytes

0 discards, 0 errors

The previous output shows that VSAN 70 is up and trunks on VFC and that it does indeed operate in the expected mode - TF.

Verify NetApp and UCS are Logged Into the Fabric

Check the flogi database on the switches.

bdsol-n5548-05# show flogi database

--------------------------------------------------------------------------------

INTERFACE        VSAN    FCID           PORT NAME               NODE NAME

--------------------------------------------------------------------------------

vfc103           70    0xa00020  22:c8:54:7f:ee:f2:d6:bf 20:46:54:7f:ee:f2:d6:81

vfc105           70    0xa00000  50:0a:09:81:98:fd:66:a5 50:0a:09:80:88:fd:66:a5

[bdsol-3220-01-B-1a]

vfc106           70    0xa00001  50:0a:09:81:88:fd:66:a5 50:0a:09:80:88:fd:66:a5

[bdsol-3220-01-A-1a]

The previous output shows that each VFC has one device logged in. This is expected in a very basic setup.

In case there is doubt, you can check the status on the UCS side. In end host mode the NPV facility can be used.

bdsol-6248-03-A(nxos)# show npv status

npiv is enabled

disruptive load balancing is disabled

External Interfaces:

====================

Interface: vfc713, State: Trunking

VSAN: 1, State: Waiting For VSAN Up

VSAN: 70, State: Up, FCID: 0xa00020

Note:The FCID (0xa00020) in NPV matches the one seen by the Nexus switch.

VSAN1 is not up on the trunk nor configured, so it can be ignored. Here is verification of the VFC 713 on the UCS.

bdsol-6248-03-A(nxos)# show interface vfc 713

vfc713 is trunking (Not all VSANs UP on the trunk)

Bound interface is port-channel2

Hardware is Ethernet

Port WWN is 22:c8:54:7f:ee:f2:d6:bf

Admin port mode is NP, trunk mode is on

snmp link state traps are enabled

Port mode is TNP

Port vsan is 1

Trunk vsans (admin allowed and active) (1,70)

Trunk vsans (up) (70)

Trunk vsans (isolated) ()

Trunk vsans (initializing) (1)

1 minute input rate 2523680 bits/sec, 315460 bytes/sec, 163 frames/sec

1 minute output rate 1114056 bits/sec, 139257 bytes/sec, 69 frames/sec

134861480 frames input, 159675768364 bytes

0 discards, 0 errors

405404675 frames output, 792038498124 bytes

0 discards, 0 errors

This output shows that VSAN 70 is up and that port operates in Node port in NPV mode (NP). This is expected since UCS operates in End Host host mode.

NetApp

As mentioned previously, this setup runs in 7-mode. The cluster mode command can be different.

The FCP status on interfaces is:

bdsol-3220-01-A> fcp topology show

Switches connected on adapter 1a:

Switch Name: bdsol-n5548-05

Switch Vendor: Cisco Systems, Inc.

Switch Release: 5.2(1)N1(1b)

Switch Domain: 160

Switch WWN: 20:46:00:2a:6a:28:68:41

Port Count: 5

(...)

Switches connected on adapter 1b:

Switch Name: bdsol-n5548-06

Switch Vendor: Cisco Systems, Inc.

Switch Release: 5.2(1)N1(1b)

Switch Domain: 35

Switch WWN: 20:50:00:2a:6a:28:6e:41

Port Count: 5

Troubleshoot

This section provides information you can use to troubleshoot your configuration. The Cisco CLI Analyzer  supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.

Note: Only registered Cisco users can access internal tools and information.

Features Configured

Check to ensure that all required features are enabled.

bdsol-n5548-05# show feature | i npiv|fcoe|lacp

fcoe 1 enabled

fcoe-npv 1 disabled

lacp 1 enabled

npiv 1 enabled

VLAN and VSAN Configuration

Check the existence of the FCoE VLAN and where it is sent.

show vlan fcoe

show vlan id X

An example for the left side of the fabric is shown here:

bdsol-n5548-05# show vlan fcoe

Original VLAN ID        Translated VSAN ID      Association State

----------------        ------------------      -----------------

970                       70               Operational

bdsol-n5548-05# show vlan id 970

VLAN Name                             Status    Ports

---- -------------------------------- --------- -------------------------------

970  VLAN0970                         active    Po1, Po103, Po105, Po106, Po107

Po202, Po203, Po204, Eth1/3

Eth1/5, Eth1/6, Eth1/7, Eth1/8

VLAN 970 exists and is trunked to the ports outlined in the configuration, as well as confirmation that it carries VSAN 70.

Spanning Tree and FCoE VLANs

FCoE VLANs are a special case and must not exist in the spanning tree.

bdsol-n5548-05# show spanning-tree vlan 970

Spanning tree instance(s) for vlan does not exist.

A special case exists for Multiple Spanning Tree (MST) and FCoE VLANs. An instance of MST needs to be created in order to carry FCoE VLANs, and not other VLANs. See MST Instances For Dual Fabric FCoE Deployments.

VPC and FCoE VLANs

In the previous example, FCoE VLANs are added on VPCs and VPCs are on two different devices. The configuration for trunking is slightly different for each VPC.

The FCoE VLANs cannot show up as allowed VLANs in the VPC consistency check.

bdsol-n5548-05# show vpc consistency-parameters vpc 105

Legend:

Type 1 : vPC will be suspended in case of mismatch

Name                        Type  Local Value            Peer Value

-------------               ----  ---------------------- -----------------------

Shut Lan                    1     No                     No

STP Port Type               1     Default                Default

STP Port Guard              1     None                   None

STP MST Simulate PVST       1     Default                Default

lag-id                      1     [(1, 2-a0-98-40-ba-fa, [(1, 2-a0-98-40-ba-fa,

1, 0, 0), (7f9b,       1, 0, 0), (7f9b,

0-23-4-ee-be-35, 8069, 0-23-4-ee-be-35, 8069,

0, 0)]                 0, 0)]

mode                        1     active                 active

Speed                       1     10 Gb/s                10 Gb/s

Duplex                      1     full                   full

Port Mode                   1     trunk                  trunk

Native Vlan                 1     1                      1

MTU                         1     1500                   1500

Admin port mode             1

Allowed VLANs               -     3380                   3380

Local suspended VLANs       -     -                      -

This is expected.

In order to verify that the FCoE VLANs are trunked, verify per-interface trunking.

bdsol-n5548-05# show interface port-channel 105 trunk

--------------------------------------------------------------------------------

Port          Native  Status        Port

Vlan                  Channel

--------------------------------------------------------------------------------

Po105         1       trunking      --

--------------------------------------------------------------------------------

Port          Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Po105         970,3380

--------------------------------------------------------------------------------

Port          STP Forwarding

--------------------------------------------------------------------------------

Po105         3380

As mentioned, FCoE VLANs do not participate in spanning tree and are not visible as an STP forwarding VLAN. However they can show up as trunking.

Priority Flow Control Status

Priority flow control is crucial for the operation of FCoE. In order to have the PFC operational, the QoS configuration must match end to end.

In order to verify the state on a per interface basis:

bdsol-n5548-05# show interface ethernet 1/3 priority-flow-control

============================================================

Port Mode Oper(VL bmap) RxPPP TxPPP

============================================================

Ethernet1/3 Auto On (8) 2852 13240218

The previous output shows that PFC is operational (On) and that Per Priority Pause (PPP) frames have been both received and transmitted.

For a device-wide view the administrator can enter the show int priority-flow-control command:

bdsol-n5548-05# show int priority-flow-control

============================================================

Port Mode Oper(VL bmap) RxPPP TxPPP

============================================================

Ethernet1/1 Auto Off 4 0

Ethernet1/2 Auto Off 4 0

Ethernet1/3 Auto On (8) 2852 13240218

Ethernet1/5 Auto On (8) 0 0

Ethernet1/6 Auto On (8) 0 0

(...)

The interfaces which were outlined in this document, Ethernet 1/3, 1/5 and 1/6, have PFC enabled and operational.

VSAN Membership

The associated configured VFCs need to be a part of the right VSAN. This example shows that the configured virtual interfaces are part of the correct VSAN.

bdsol-n5548-05# show vsan membership

vsan 1 interfaces:

fc2/15

vsan 70 interfaces:

fc2/16            vfc103            vfc105            vfc106

FCoE Initiation

FCoE Initiation Protocol goes through several steps in order to bring up the link between the end host and the switch.

What is important to remember is that to bring up the interface properly, a couple of different actions need to happen as depicted in this diagram.

Enode is an end node, while FCF is the switch.

In order to verify the steps from one of the devices, the fcoe_mgr (FCoE manager) facility can be used. In this case it is vfc 105.

bdsol-n5548-05# show platform software fcoe_mgr info interface vfc 105

vfc105(0x841e4c4), if_index: 0x1e000068, VFC RID vfc105

FSM current state: FCOE_MGR_VFC_ST_PHY_UP

PSS Runtime Config:-

Type: 3

Bound IF: Po105

FCF Priority: 128 (Global)

Disable FKA: 0

PSS Runtime Data&colon;-

IOD: 0x00000000, WWN: 20:68:00:2a:6a:28:68:7f

Created at: Sat Sep 27 22:45:05 2014

FC Admin State: up

Oper State: up, Reason: down

Eth IF Index: Po105

Port Vsan: 70

Port Mode: F port

Config Vsan: 70

Oper Vsan: 70

Solicits on vsan: 70

Isolated Vsan:

FIP Capable ? : TRUE

UP using DCBX ? : FALSE

Peer MAC : 00:c0:dd:22:79:39

PSS VN Port data&colon;-

FC ID 0xA00000 -

vfc index 503316584 vfc name vfc105

vsan id 70

enode_mac 00:c0:dd:22:79:39

vfc wwn 50:0a:09:81:98:fd:66:a5

The output is continued.

The output contains a few useful pieces of information. It verifies current status, shows the actual binding between VFC and portchannel and the actual state of the VFC on the VSAN. It shows transitions between different states outlined in the diagram before the interface comes up. VLAN discovery is performed and FIP solicitation is received and answered which allows the VFC to come up.

Continuation of the previous output:

17) FSM:<vfc105> Transition at 554094 usecs after Sat Sep 27 22:48:06 2014

Previous state: [FCOE_MGR_VFC_ST_PHY_UP]

Triggered event: [FCOE_MGR_VFC_EV_BRING_UP]

Next state: [FSM_ST_NO_CHANGE]

18) FSM:<vfc105> Transition at 685820 usecs after Sat Sep 27 22:48:07 2014

Previous state: [FCOE_MGR_VFC_ST_PHY_UP]

Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]

Next state: [FSM_ST_NO_CHANGE]

19) FSM:<vfc105> Transition at 686781 usecs after Sat Sep 27 22:48:07 2014

Previous state: [FCOE_MGR_VFC_ST_PHY_UP]

Triggered event: [FCOE_MGR_VFC_EV_FIP_SOLICITATION]

Next state: [FSM_ST_NO_CHANGE]

Curr state: [FCOE_MGR_VFC_ST_PHY_UP]

Known and Common Problems

There a couple of known and common problems with the FCoE configuration that the Data Center Networking Solutions Technical Assistance Center team sees reoccur.

- Trunk protocol - The trunk protocol needs to be up.

bdsol-n5548-05# show trunk protocol

Trunk Protocol is enabled.

- NetApp 8.2.2 and Nexus 5500/5600 that run NX-OS 7.x software. At the time that the document was written, a known problem between these two versions exists and is under investigation by Cisco and NetApp. Versions 6.x and 5.x of NX-OS work fine with NetApp 8.2.2.

Related Information

- Interface Modes (Port Descriptions)

- Cisco Technical Support & Downloads