Troubleshooting Packet Flow Issues

Packet Flow Issues

Packets could be dropped for the following reasons:

  • Software-switched packets could be dropped because of Control Plane Policing (CoPP).

  • Hardware-switched packets could be dropped by the hardware because of a bandwidth limitation.

Beginning with Cisco NX-OS Release 10.3.(1)F, the following CLIs are supported on Cisco Nexus 9300 and 9500 Cloud Scale switches.

  • show hardware internal statistics module-all all : Displays the statistics of active modules.

  • show hardware internal statistics module <module-no> all : Displays the statistics of a particular active module from supervisor.

Packets Dropped Because of Rate Limits

Use the show hardware rate-limit command to determine if packets are being dropped because of a rate limit.

switch(config)# show hardware rate-limit module 1 

Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters

Rate Limiter Class                       Parameters
------------------------------------------------------------
access-list-log                         Config    : 100            
                                        Allowed   : 0                   
                                        Dropped   : 0                   
                                        Total     : 0

Packets Dropped Because of CoPP

Use the show policy-map interface control-plane command to determine if packets are being dropped because of CoPP.

switch# show policy-map interface control-plane
   class-map copp-system-p-class-exception (match-any)
      match exception ip option
      match exception ip icmp unreachable
      match exception ttl-failure
      match exception ipv6 option
      match exception ipv6 icmp unreachable
      match exception mtu-failure
      set cos 1
      police cir 200 pps , bc 32 packets 
      
      module 27 :
        transmitted 0 packets;
        dropped 0 packets;

      module 28 :
        transmitted 0 packets;
        dropped 0 packets;

Monitoring Inband Packet Statistics

Use the show hardware internal cpu-mac inband counters command to display inband packet statistics for supervisor modules, fabric modules, and line cards.

switch# show hardware internal cpu-mac inband counters
eth2 counters:
eth2      Link encap:Ethernet  HWaddr 00:00:00:01:1b:01  
          BROADCAST MULTICAST  MTU:9400  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth3 counters:
eth3      Link encap:Ethernet  HWaddr 00:00:00:01:1b:01  
          inet6 addr: fe80::200:ff:fe01:1b01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9400  Metric:1
          RX packets:425432 errors:0 dropped:0 overruns:0 frame:0
          TX packets:352432 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:253284953 (241.5 MiB)  TX bytes:249647978 (238.0 MiB)

ps-inb counters:
ps-inb    Link encap:Ethernet  HWaddr 00:00:00:01:1b:01  
          inet6 addr: fe80::200:ff:fe01:1b01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9400  Metric:1
          RX packets:128986 errors:0 dropped:0 overruns:0 frame:0
          TX packets:129761 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:221538103 (211.2 MiB)  TX bytes:227158091 (216.6 MiB)

switch# slot 22 show hardware internal cpu-mac inband  counters
inband0 counters:
inband0   Link encap:Ethernet  HWaddr 00:00:00:01:16:03  
          inet addr:127.2.2.22  Bcast:127.2.255.255  Mask:255.255.0.0
          inet6 addr: fe80::200:ff:fe01:1603/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9676  Metric:1
          RX packets:147425 errors:0 dropped:0 overruns:0 frame:0
          TX packets:147470 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:532 
          RX bytes:15479625 (14.7 MiB)  TX bytes:14898335 (14.2 MiB)
          Interrupt:10 

knet0_0 counters:
knet0_0   Link encap:Ethernet  HWaddr 02:10:18:e1:6f:50  
          inet6 addr: fe80::10:18ff:fee1:6f50/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9400  Metric:1
          RX packets:36 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3816 (3.7 KiB)  TX bytes:0 (0.0 B)

knet0_1 counters:
knet0_1   Link encap:Ethernet  HWaddr 02:10:18:e1:6f:51  
          inet6 addr: fe80::10:18ff:fee1:6f51/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9400  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Fabric Connectivity Commands

Cisco NX-OS provides the following commands to display information and statistics related to fabric connectivity:

  • show system internal fabric connectivity [module module-number] —Displays connectivity information for all fabric modules or a single module.

    switch# show system internal fabric connectivity
    HiGIG Link-info Linecard slot:4
    --------------------------------------------------------------
     LC-Slot  LC-Unit   LC-HGLink   FM-Slot   FM-Unit   FM-HGLink 
    --------------------------------------------------------------
        4        0       HG02         22        0        HG09
        4        0       HG03         22        1        HG09
        4        0       HG06         24        0        HG09
        4        0       HG07         24        1        HG09
        4        1       HG02         22        0        HG10
        4        1       HG03         22        1        HG10
        4        1       HG06         24        0        HG10
        4        1       HG07         24        1        HG10
        4        2       HG02         22        0        HG11
        4        2       HG03         22        1        HG11
        4        2       HG06         24        0        HG11
        4        2       HG07         24        1        HG11
    
    HiGIG Link-info Fabriccard slot:22
    --------------------------------------------------------------
     FM-Slot  FM-Unit   FM-HGLink   LC-Slot   LC-Unit   LC-HGLink 
    --------------------------------------------------------------
       22        0       HG09          4        0        HG02
       22        0       HG10          4        1        HG02
       22        0       HG11          4        2        HG02
       22        1       HG09          4        0        HG03
       22        1       HG10          4        1        HG03
       22        1       HG11          4        2        HG03
    
    HiGIG Link-info Fabriccard slot:24
    --------------------------------------------------------------
     FM-Slot  FM-Unit   FM-HGLink   LC-Slot   LC-Unit   LC-HGLink 
    --------------------------------------------------------------
       24        0       HG09          4        0        HG06
       24        0       HG10          4        1        HG06
       24        0       HG11          4        2        HG06
       24        1       HG09          4        0        HG07
       24        1       HG10          4        1        HG07
       24        1       HG11          4        2        HG07
    
    
  • show system internal interface counters module module-number [nz] —Displays the rates for HG or fabric links on a module. The nz option displays only non-zero counters.

    switch# show system internal interface counters module 22 nz
    Internal Port Counters (150 secs rate) for Slot: 22
    ====================================================
    
    Interface ASIC ASIC  BCM  TxBitRate(BwUtil) TxPktRate RxBitRate(BwUtil) RxPktRate
              Port Inst Port     (bps)            (pps)       (bps)          (pps)
    ----------------------------------------------------------------------------------
    ii22/1/10  HG9    0   10     0(  0.00)         0       33064(  0.00)      17
    
    
    switch# show system internal interface counters module 22 
    Internal Port Counters (150 secs rate) for Slot: 22
    ====================================================
    
    Interface ASIC ASIC  BCM  TxBitRate(BwUtil) TxPktRate RxBitRate(BwUtil) RxPktRate
              Port Inst Port      (bps)           (pps)       (bps)           (pps)
    ----------------------------------------------------------------------------------
    ii22/1/1  HG0    0    1       0(  0.00)        0         0(  0.00)         0
    ii22/1/2  HG1    0    2       0(  0.00)        0         0(  0.00)         0
    ii22/1/3  HG2    0    3       0(  0.00)        0         0(  0.00)         0
    ii22/1/4  HG3    0    4       0(  0.00)        0         0(  0.00)         0
    ii22/1/5  HG4    0    5       0(  0.00)        0         0(  0.00)         0
    ii22/1/6  HG5    0    6       0(  0.00)        0         0(  0.00)         0
    ii22/1/7  HG6    0    7       0(  0.00)        0         0(  0.00)         0
    ii22/1/8  HG7    0    8       0(  0.00)        0         0(  0.00)         0
    ii22/1/9  HG8    0    9       0(  0.00)        0         0(  0.00)         0
    ii22/1/10  HG9    0   10      0(  0.00)        0     30888(  0.00)        12
    ii22/1/11 HG10    0   11      0(  0.00)        0         0(  0.00)         0
    ii22/1/12 HG11    0   12      0(  0.00)        0         0(  0.00)         0
    ii22/1/13 HG12    0   13      0(  0.00)        0         0(  0.00)         0
    ii22/1/14 HG13    0   14      0(  0.00)        0         0(  0.00)         0
    ii22/1/15 HG14    0   15      0(  0.00)        0         0(  0.00)         0
    ii22/1/16 HG15    0   16      0(  0.00)        0         0(  0.00)         0
    ii22/1/17 HG16    0   17      0(  0.00)        0         0(  0.00)         0
    ii22/1/18 HG17    0   18      0(  0.00)        0         0(  0.00)         0
    ii22/1/19 HG18    0   19      0(  0.00)        0         0(  0.00)         0
    ii22/1/20 HG19    0   20      0(  0.00)        0         0(  0.00)         0
    ii22/1/21 HG20    0   21      0(  0.00)        0         0(  0.00)         0
    ii22/1/22 HG21    0   22      0(  0.00)        0         0(  0.00)         0
    ii22/1/23 HG22    0   23      0(  0.00)        0         0(  0.00)         0
    ii22/1/24 HG23    0   24      0(  0.00)        0         0(  0.00)         0
    ii22/1/33  HG0    1    1      0(  0.00)        0         0(  0.00)         0
    ii22/1/34  HG1    1    2      0(  0.00)        0         0(  0.00)         0
    ii22/1/35  HG2    1    3      0(  0.00)        0         0(  0.00)         0
    ii22/1/36  HG3    1    4      0(  0.00)        0         0(  0.00)         0
    ii22/1/37  HG4    1    5      0(  0.00)        0         0(  0.00)         0
    ii22/1/38  HG5    1    6      0(  0.00)        0         0(  0.00)         0
    ii22/1/39  HG6    1    7      0(  0.00)        0         0(  0.00)         0
    ii22/1/40  HG7    1    8      0(  0.00)        0         0(  0.00)         0
    ii22/1/41  HG8    1    9      0(  0.00)        0         0(  0.00)         0
    ii22/1/42  HG9    1   10      0(  0.00)        0         0(  0.00)         0
    ii22/1/43 HG10    1   11      0(  0.00)        0         0(  0.00)         0
    ii22/1/44 HG11    1   12      0(  0.00)        0         0(  0.00)         0
    ii22/1/45 HG12    1   13      0(  0.00)        0         0(  0.00)         0
    ii22/1/46 HG13    1   14      0(  0.00)        0         0(  0.00)         0
    ii22/1/47 HG14    1   15      0(  0.00)        0         0(  0.00)         0
    ii22/1/48 HG15    1   16      0(  0.00)        0         0(  0.00)         0
    ii22/1/49 HG16    1   17      0(  0.00)        0         0(  0.00)         0
    ii22/1/50 HG17    1   18      0(  0.00)        0         0(  0.00)         0
    ii22/1/51 HG18    1   19      0(  0.00)        0         0(  0.00)         0
    ii22/1/52 HG19    1   20      0(  0.00)        0         0(  0.00)         0
    ii22/1/53 HG20    1   21      0(  0.00)        0         0(  0.00)         0
    ii22/1/54 HG21    1   22      0(  0.00)        0         0(  0.00)         0
    ii22/1/55 HG22    1   23      0(  0.00)        0         0(  0.00)         0
    ii22/1/56 HG23    1   24      0(  0.00)        0         0(  0.00)         0
    
    
  • show system internal interface counters detail module module-number —Displays detailed statistics for all HG or fabric links on a single module.

    show system internal interface counters detail module 4
    ................
    ............
    Interface: ii4/1/3  ASIC Inst# 0/Port# 3/Name HG2
    -------------------------------------------------------
    Last Cleared @ Thu Jan  1 00:00:00 2013
     (0)
    Tx/Rx Rates (per second):
              secs     tx bytes    tx packets  rx bytes    rx packets 
        [0] - 10       0           0           0           0          
        [1] - 150      9448        60          0           0          
        [2] - 300      9448        60          0           0          
    Mac Pktflow:
      Rx Counters:
        Ingress Packets  :       0x0000000000000000/0
        Unicast Packets  :       0x0000000000000000/0
        Multicast Packets:       0x0000000000000000/0
        Broadcast Packets:       0x0000000000000000/0
        Jumbo Packets    :       0x0000000000000000/0
        Total Bytes      :       0x0000000000000000/0
    
        Rx Bytes by Packet Size:
        64:             0x0000000000000000/0
        65   - 127:     0x0000000000000000/0
        128  - 255:     0x0000000000000000/0
        256  - 511:     0x0000000000000000/0
        512  - 1023:    0x0000000000000000/0
        1024 - 1518:    0x0000000000000000/0
        1519 - 1548:    0x0000000000000000/0
    
      Tx Counters:
        Egress Packets   :       0x0000000000001351/4945
        Unicast qackets:         0x0000000000001351/4945
        Multicast qackets:       0x0000000000000000/0
        Broadcast Packets:       0x0000000000000000/0
        Jumbo Packets    :       0x0000000000000000/0
        Undersize Packets:       0x0000000000000000/0
        Total Bytes      :       0x000000000008e756/583510
    
        Tx Bytes by Packet Size
        64:             0x0000000000000000/0
        65   - 127:     0x0000000000001351/4945
        128  - 255:     0x0000000000000000/0
        256  - 511:     0x0000000000000000/0
        512  - 1023:    0x0000000000000000/0
        1024 - 1518:    0x0000000000000000/0
        1519 - 1548:    0x0000000000000000/0
        trunk:          0x0000000000000000/0
    
    Mac Control:
        Rx Pause:       0x0000000000000000/0
        Tx Pause:       0x0000000000000000/0
        Reset:          0x0000000000000000/0
    Mac Errors:
        Undersize:      0x0000000000000000/0
        Runt:           0x0000000000000000/0
        Crc:            0x0000000000000000/0
        Input Errors:   0x0000000000000000/0
        In Discard:     0x0000000000000000/0
        Giants:         0x0000000000000000/0
        Output Errors:  0x0000000000000000/0
        Output Discard: 0x0000000000000000/0
        Bad Proto:      0x0000000000000000/0
        Collision:      0x0000000000000000/0
        Late Collision: 0x0000000000000000/0
        No Carrier:     0x0000000000000000/0