Contents

Introduction

Prerequisites

Requirements

Components Used

Background Infomation

Configure

Example

Verify

Troubleshoot

Related Information

Introduction

This document describes the use of  Embedded Logic Analyzer Module (ELAM) tool within the Unified Computing System (UCS) 4th Generation Fabric Interconnect (FI) 6454, and how to best use it.

Prerequisites

There are no prerequisites for this document.

Requirements

Cisco recommends that you have knowledge of these topics:

- UCS 6454 Fabric Interconnect

Components Used

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Infomation

UCS 4th Gen FI has the capability to run ELAM captures. An ELAM capture comes embedded on the ASIC.

ELAM tool allows real time view of the packets being forwarded at the ASIC level. You can view the details of a packet such as:

- Ingress and Egress Interface

- Maximum Transmission Unit (MTU) Size

- VLAN Tag

- MAC and IP Address of the Source and Destination Device

- Packet Drop and The Cause

- Quality of Service (QoS) Marking

ELAM provides details of packet forwarding. It is nondisruptive to the data plane.

Configure

Log in to UCS via Command Line Interface (CLI).

Run these commands:

#connect nxos a|b

#attach module 1

#debug platform internal tah elam asic 0

#trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1

#set outer ...

#start

#report

Note: 4th Gen FI is a single rack unit with one single module (module 1) with one ASIC (asic 0) and one slice (slice 0). See the output below.

RCH-SV-FFAIII-A(nx-os)# show hardware internal tah interface ethernet 1/30

#########################################

IfIndex: 436222464

DstIndex: 6028

IfType: 26

Interface name Ethernet1/30

Asic: 0

Asic: 0     <<<<<

AsicPort: 49

SrcId: 98

Slice: 0    <<<<<

PortOnSlice: 49

Table entries for interface Ethernet1/30

For ELAMs where the trigger is based on packet attributes "lu-a2d 1" is used. Value 6 and 1 will be used for "in-select" and "out-select" respectively for out interest

The "set outer" command is our filter, this is where we define and tell the FI what packet we want to capture, there are a bunch of options and we can be as granular as needed:

module-1(TAH-elam-insel6)# set outer ?

arp   ARP Fields

fcoe  FCoE Fields

ipv4  IPv4 Fields

ipv6  IPv6 Fields

l2    All Layer 2 Fields

l4    L4 Fields

module-1(TAH-elam-insel6)# set outer l2 ?

cfi            CFI Setting

cntag_vld      CNTag Information Valid

cos            Class of Service

dst_mac        Destination MAC Address

qtag_vld       VLAN Tag Information Valid

snap_vld       SNAP Header Information Valid

src_mac        Source MAC Address

vlan           VLAN Id (Present only in case of FEX)

vntag_dvif     VNTAG Destination vif

vntag_looped   VNTAG Header Looped Valid

vntag_pointer  VNTAG Header Pointer Valid

vntag_svif     VNTAG Source vif

vntag_vld      VNTAG Information Valid

module-1(TAH-elam-insel6)# set outer ipv4 ?

checksum       Checksum

dscp           Diff. Serv. Code Point

dst_ip         Destination IP Address

ecn            Explicit Congestion Ntfn

fragment-off   Fragments Offset

header-len     Header Length

more-frags     More Fragments Available

next-protocol  Next Protocol

packet-len     Packet Total Length

pyld-len       Payload Length

src_ip         Source IP Address

ttl            Time to Live

version        Version

Once the filters have been defined, run the command start to run the ELAM tool. If nothing that fullfills the filter condition has been captured then, this is seen:

module-1(TAH-elam-insel6)# report

ELAM not triggered yet on slot - 1, asic - 0, slice - 0

Note: The "set" command will survive across ELAMs, a good practice is to run a "reset" command everytime we intend to capture traffic with different IP, MAC, etc.

Example

1. Ping from VM 172.16.35.31 to gateway 172.16.35.126:

RCH-SV-FFAIII-A(nx-os)# attach module 1

module-1# debug platform internal tah elam asic 0

module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1

param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1

module-1(TAH-elam-insel6)# set outer ipv4 src_ip 172.16.35.31 dst_ip 172.16.35.126

module-1(TAH-elam-insel6)# start

GBL_C++:  [MSG] rocky_elam_wrapper_init:54:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1

GBL_C++:  [MSG] rocky_elam_wrapper_enable:149:asic type 8 inst 0 slice 0 a_to_d 1

GBL_C++:  [MSG]   - writing data=0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000005608118F800000

000000000000000000560811BF00000000000000000000001000000000000000000000000000000000000000000000000000000000001

GBL_C++:  [MSG]   - writing mask=0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000007FFFFFFF800000

0000000000000000007FFFFFFF80000000000000000000003800000000000000000000000000000000000000000000000000000000001

module-1(TAH-elam-insel6)# report

HOMEWOOD ELAM REPORT SUMMARY

slot - 1, asic - 0, slice - 0

============================

Incoming Interface: Eth1/33

Src Idx : 0x1002, Src BD : 35

Outgoing Interface Info: dmod 1, dpid 4

Dst Idx : 0x604, Dst BD : 35

Packet Type: IPv4

Dst MAC address: 8C:60:4F:CD:FD:7C

Src MAC address: 00:25:C5:00:00:1E

.1q Tag0 VLAN: 35,  cos = 0x1

Dst IPv4 address: 172.16.35.126

Src IPv4 address: 172.16.35.31

Ver     =  4, DSCP    =    0, Don't Fragment = 0

Proto   =  1, TTL     =   64, More Fragments = 0

Hdr len = 20, Pkt len =   84, Checksum       = 0x5f19

L4 Protocol  : 1

ICMP type    : 8

ICMP code    : 0

Drop Info:

----------

LUA:

LUB:

LUC:

LUD:

Final Drops:

Verify

The packet with src_ip 172.16.35.31 and dst_ip 172.16.35.126 is on VLAN 35 and arrived on port 1/33 (incoming interface) and was destined (outgoing interface) to interface "dpid4" ...the what? "dpid" is the ASIC port internal identifier and the mapping can be found with "show interface hardware-mappings":

Incoming Interface: Eth1/33

Src Idx : 0x1002, Src BD : 35

interface Ethernet1/33

description S: Server, Port-channel 1025

no pinning server sticky

switchport mode fex-fabric

priority-flow-control mode on

fex associate 1

channel-group 1025

no shutdown

Outgoing Interface Info: dmod 1, dpid 4

Dst Idx : 0x604, Dst BD : 35

RCH-SV-FFAIII-A(nx-os)# show interface hardware-mappings

Legends:

SMod  - Source Mod. 0 is N/A

Unit  - Unit on which port resides. N/A for port channels

HPort - Hardware Port Number or Hardware Trunk Id:

HName - Hardware port name. None means N/A

FPort - Fabric facing port number. 255 means N/A

NPort - Front panel port number

VPort - Virtual Port Number. -1 means N/A

Slice - Slice Number. N/A for BCM systems

SPort - Port Number wrt Slice. N/A for BCM systems

SrcId - Source Id Number. N/A for BCM systems

------------------------------------------------------------------------

Name       Ifindex  Smod Unit HPort FPort NPort VPort Slice SPort SrcId

------------------------------------------------------------------------

.

Eth1/13    1a001800 1    0    4     255   48    -1    0     4     8

This "dpid 4" also corresponds to what the "show hardware internal tah interface ethernet 1/13" suggests:

RCH-SV-FFAIII-A(nx-os)# show hardware internal tah interface ethernet 1/13

#########################################

IfIndex: 436213760

DstIndex: 6096

IfType: 26

Interface name Ethernet1/13

Asic: 0

Asic: 0

AsicPort: 4     <<<<<

SrcId: 8

Slice: 0

PortOnSlice: 4  <<<<<

The packet was identified as an Internet Control Message Protocol (ICMP) by the ELAM Layer 4 (L4) Protocol. Refer to list of IANA protocol numbers. You can also filter with a specific MTU size.  ELAM triggers only when the exact MTU is hit.

module-1(TAH-elam-insel6)# set outer ipv4 src_ip 172.16.35.31 dst_ip 172.16.35.126 packet-len 1500

Dst IPv4 address: 172.16.35.126

Src IPv4 address: 172.16.35.31

Ver     =  4, DSCP    =    0, Don't Fragment = 1

Proto   =  1, TTL     =   64, More Fragments = 0

Hdr len = 20, Pkt len = 1500, Checksum       = 0x1758

L4 Protocol  : 1

ICMP type    : 8

ICMP code    : 0

ARP request from the Virtual Machine (VM) to upstream network, with the MAC addresses set as filter:

RCH-SV-FFAIII-B(nx-os)# attach module 1

module-1# debug platform internal tah elam asic 0

module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1

param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1

module-1(TAH-elam-insel6)# set outer l2 src_mac 00:25:c5:00:00:1e dst_mac ff:ff:ff:ff:ff:ff

module-1(TAH-elam-insel6)# start

GBL_C++:  [MSG] rocky_elam_wrapper_init:36:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1

GBL_C++:  [MSG] rocky_elam_wrapper_enable:95:asic type 8 inst 0 slice 0 a_to_d 1

GBL_C++:  [MSG]   - writing data=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

00000015820463E004B8A00003C0000000000000000000002000000000000000000000000000000000000000000000000000000000001

GBL_C++:  [MSG]   - writing mask=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000001FFFFFFFFFFFFFFFFFFFE0000000000000000000003800000000000000000000000000000000000000000000000000000000001

module-1(TAH-elam-insel6)# report

HOMEWOOD ELAM REPORT SUMMARY

slot - 1, asic - 0, slice - 0

============================

Incoming Interface: Eth1/33

Src Idx : 0x1002, Src BD : 35

Outgoing Interface Info: dmod 1, dpid 4

Dst Idx : 0x604, Dst BD : 35

Packet Type: ARP

Dst MAC address: FF:FF:FF:FF:FF:FF

Src MAC address: 00:25:C5:00:00:1E

.1q Tag0 VLAN: 35,  cos = 0x1

Target Hardware address: 00:00:00:00:00:00

Sender Hardware address: 00:25:C5:00:00:1E

Target Protocol address: 172.16.35.110

Sender Protocol address: 172.16.35.31

ARP opcode: 1

Drop Info:

----------

LUA:

LUB:

LUC:

LUD:

Final Drops:

The packet is identified by the system as an ARP, this is especially useful when there is an incomplete ARP entry either at the VM or gateway level.

If applicable, the TCP/UDP ports will be listed as well, SSH is tested here:

RCH-SV-FFAIII-B(nx-os)# attach module 1

module-1# debug platform internal tah elam asic 0

module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1

param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1

module-1(TAH-elam-insel6)# set outer ipv4 src_ip 172.16.35.126 dst_ip 172.16.35.31

module-1(TAH-elam-insel6)# start

GBL_C++:  [MSG] rocky_elam_wrapper_init:36:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1

GBL_C++:  [MSG] rocky_elam_wrapper_enable:95:asic type 8 inst 0 slice 0 a_to_d 1

GBL_C++:  [MSG]   - writing data=0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000560811BF0000

000000000000000000005608118F80000000000000000000001000000000000000000000000000000000000000000000000000000000001

GBL_C++:  [MSG]   - writing mask=0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000007FFFFFFF8000

000000000000000000007FFFFFFF80000000000000000000003800000000000000000000000000000000000000000000000000000000001

module-1(TAH-elam-insel6)# report

HOMEWOOD ELAM REPORT SUMMARY

slot - 1, asic - 0, slice - 0

============================

Incoming Interface: Eth1/14

Src Idx : 0x604, Src BD : 35

Outgoing Interface Info: dmod 1, dpid 44

Dst Idx : 0x1002, Dst BD : 35

Packet Type: IPv4

Dst MAC address: 00:25:C5:00:00:1E

Src MAC address: 8C:60:4F:CD:FD:7C

.1q Tag0 VLAN: 35,  cos = 0x0

Dst IPv4 address: 172.16.35.31

Src IPv4 address: 172.16.35.126

Ver     =  4, DSCP    =    0, Don't Fragment = 0

Proto   =  6, TTL     =   64, More Fragments = 0

Hdr len = 20, Pkt len =   60, Checksum       = 0x27f5

L4 Protocol  : 6

TCP Dst Port : 22

TCP Src Port : 15067

Drop Info:

----------

LUA:

LUB:

LUC:

LUD:

Final Drops:

Troubleshoot

Drops are recorded as well. The FI drops an ARP request:

RCH-SV-FFAIII-B(nx-os)# attach module 1

module-1# debug platform internal tah elam asic 0

module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 1

param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 1

module-1(TAH-elam-insel6)# set outer l2 src_mac 00:25:c5:00:00:1e dst_mac ff:ff:ff:ff:ff:ff

module-1(TAH-elam-insel6)# start

GBL_C++:  [MSG] rocky_elam_wrapper_init:54:asic type 8 inst 0 slice 0 a_to_d 1 insel 6 outsel 1

GBL_C++:  [MSG] rocky_elam_wrapper_enable:149:asic type 8 inst 0 slice 0 a_to_d 1

GBL_C++:  [MSG]   - writing data=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000971400007BFFFFFFFFFFFC00001

GBL_C++:  [MSG]   - writing mask=0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

00000000000000000000000000000000000000000000000000000000000000000000000000000003FFFFFFFFFFFFFFFFFFFFFFFC00001

module-1(TAH-elam-insel6)# report

HOMEWOOD ELAM REPORT SUMMARY

slot - 1, asic - 0, slice - 0

============================

Incoming Interface: Eth1/18

Src Idx : 0x603, Src BD : 35

Outgoing Interface Info: dmod 0, dpid 0

Dst Idx : 0x0, Dst BD : 35

Packet Type: ARP

Dst MAC address: FF:FF:FF:FF:FF:FF

Src MAC address: 00:25:C5:00:00:1E

.1q Tag0 VLAN: 35,  cos = 0x1

Target Hardware address: 00:00:00:00:00:00

Sender Hardware address: 00:25:C5:00:00:1E

Target Protocol address: 172.16.35.99

Sender Protocol address: 172.16.35.31

ARP opcode: 1

Drop Info:

----------

LUA:

LUB:

LUC:

LUD:

MC_RPF_FAIILURE

SRC_VLAN_MBR

Final Drops:

MC_RPF_FAIILURE

SRC_VLAN_MBR

The FI receives an ARP request on port 1/18 (this is an uplink port) with a source MAC of 00:25:c5:00:00:1e which is learned locally on a virtual Ethernet (vEth) ports. This condition triggers a Reverse Path Forward (RPF) drop. Notice that the Outgoing Interface Info reports dpid 0 which is a drop.

VLAN 35 is not allowed on port 1/18 and this also triggered drop SRC_VLAN_MBR.

RCH-SV-FFAIII-A(nx-os)# show run interface ethernet 1/18

interface Ethernet1/18

description U: Uplink

pinning border

switchport mode trunk

switchport trunk allowed vlan 1

channel-group 105 mode active

Related Information

- ELAM Overview

- Technical Support & Documentation - Cisco Systems