Troubleshoot Output from Catalyst 9000 Switches

Introduction

This document describes important checks and useful outputs from Catalyst 9000 Switches.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• General guidance  on  the software  selection for Catalyst 9000 switches
• In-Service Software Upgrade (ISSU) Compatibility Matrix
• Detailed ISSU Steps - Perform ISSU on Catalyst 3850 and Catalyst 9000 Series Switches

Components Used

The information in this document is based on the Catalyst 9000 Switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document is a guide on important checks and useful outputs from Catalyst 9000 Switches which can be collected when working with Cisco TAC. The information in this document is not meant to be exhaustive and is intended to serve as a general guide to customers especially when they are planning to open a service request with Cisco TAC.

Output to Collect

In case of upgrade related issues, session logs from the console terminal are required that provide the logs around the upgrade. In addition, these outputs are needed:

• show install summary
• show issu state detail

Basic Check

• Document exact and detailed problem description.
• When did the issue start? How often does the issue occur? Is the issue consistently observed?
• Was this working fine before? If it was working fine what changes occurred that could have triggered the issue? Have there been any recent hardware, software, or environmental changes?
• Is the problem reproducible? What are the steps to replicate the problem? Provide details about what is working and what is not working.
•  Collect output of show tech-support

Feature-specific show-tech Commands

In addition to regular show tech-support, there are different feature specific show tech-support packages that can be collected. These include important platform-independent and dependent output related to that feature.

Note: The show tech-support <feature> output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support <feature> | redirect flash:filename

*Per interface option is available on Cisco IOS XE Version 17.2.1 and higher

Binary Trace (Btrace) Collection

Binary trace (Btrace) is used to debug non-Cisco IOS processes more efficiently than regular debugs. Btrace contents are stored in the tracelogs folder in crashinfo directory.

Collecting Binary Trace

Commands for collecting binary trace:

• request platform software trace rotate all
• request platform software archive - This command creates an archive file on the flash memory of active switch/supervisor.
• request platform software trace slot RP standby archive -  Run this command in case of Catalyst 9400 high availability setup to collect Btrace from standby supervisor. This is a hidden command, configure service internal to run it. This command creates an archive file on the flash memory of the standby supervisor.

Specific Log Collection Scenarios

1. Power over Ethernet (PoE)

This section covers Power over Ethernet (PoE) not being provided to a powered device (PD).

Basic Checks

• Does the Powered Device (PD) in question not power up at all, or does it power up briefly and then power down?
• What type of device is being used (Cisco legacy, 802.3af, 803.3at, UPOE?), and does the Catalyst 9000 variant in question support this type?
• Is enough inline power available from the power supply on respective switch member/line card?
• Do all ports of one switch member/line card not provide PoE or only a few?
• Is more than one line card/switch member of a chassis/stack affected?
• Do only newly connected ports not provide PoE, and previously connected ports on the same switch member/line card work fine?
• If one of the connected ports with PoE status OK (on the same switch member/line card as the faulty port) is bounced (shut/no shut), does the PoE functionality break or continues to work fine?
• Is data connectivity affected or it is just the PoE functionality?
• Is the problem restricted to one type/model of PD?
• Any PoE syslog messages being seen?

Refer to Troubleshoot Power over Ethernet on Catalyst 9000 Switches, for detailed troubleshooting of PoE problems.

Output to Collect

These outputs are needed:

• show tech poe

Note:  The show tech-support poe output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support poe | redirect flash:filename

• request platform software archive - This command creates an archive file on the flash memory of active switch/supervisor.

2. Interface and Port Related Issues

This section covers layer 1 and layer 2 issues from the switch port perspective, like link errors, access/trunk issues, and link not coming up.

Basic Checks

• What is the make and model of the connected peer device?
• Does the same peer device come up fine on any other port on this switch?
• What is the SFP type and length/type of cable used? Is the SFP supported on this platform and code version?
• If there is a patch panel, can it be bypassed to get the peer device connected directly to the switch?
• Does hard coding speed/duplex make any difference?
• Does changing the port to default configuration help?
• If the peer is a Powered Device (PD), is PoE being provided on the port correctly? If not, collect the information from the PoE section as well.

Output to Collect

Use these commands to collect output:

• show tech-support port
• show tech-support port interface <interface_number>  (Cisco IOS XE Version 17.2 or higher)
• show tech-support port-channel

Note:  The show tech-support port output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support port | redirect flash:filename

• request platform software archive - This command can create an archive file on the flash memory of active switch/supervisor.

3. Smart Licensing

This section covers Smart Licensing issues that occur when the switch is not able to register the smart license to Cisco Smart Software Manager.
The most common types of smart licensing failures are described in Troubleshoot Smart Licensing on Catalyst Platforms.

Basic Checks

• Are any error messages being seen when trying to register the switch?
• What method is being used to register the switch (direct, proxy, on-prem)?
• Is there reachability to tools.cisco.com or the on-prem IP address from the switch in question?

Output to Collect

These outputs are needed:

• show tech-support license

Note:  show tech-support license output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support license | redirect flash:filename  

• request platform software archive - This command creates an archive file on the flash memory of active switch/supervisor.

4. Stacking

This section covers stacking related issues like errors on stack port and switches unable to join the stack.

Basic Checks

• Do all stacked switches run the same software/license level/FIPS mode?
• Are the stack cables and stack ports firmly seated?
• Has a reseat/replacement of stack cable or stack adapter been tried?

Output to Collect

These outputs are needed:

• show tech-support stack

Note: The show tech-support stack output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support stack | redirect flash:filename  

• Any system-report/archive file from flash/crashinfo directory created at the time of issue.
• request platform software archive - This command creates an archive file in the flash memory of active switch/supervisor.

5. StackWise Virtual

This section covers Stackwise Virtual (SVL) issues related to the configuration and communication between switches in SVL.

Basic Checks

• If day-1 issue, Is Stackwise Virtual feature supported on this platform and Cisco IOS XE version?
• Is the port / SFP / line card being used supported for SVL and Dual Active Detection (DAD) configuration?
• If the issue is related to SVL fail-over, reload/crash on SVL, collect the information from the Unexpected Reload and Crash section as well.

Output to Collect

These outputs are needed:

• show tech-support stackwise-virtual switch all

Note:  The show tech-support stackwise-virtual output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support stackwise-virtual | redirect flash:filename

• Any system-report/archive - file from flash/crashinfo directory created at the time of issue.
• request platform software archive -This command creates an archive file on the flash memory of active switch/supervisor.
• request platform software trace slot switch standby RP archive - Run this command in case of Catalyst 9400 SVL setup to collect Btrace from standby supervisor. This command can create an archive file on the flash memory of standby supervisor.

6. Unexpected Reloads and Crashes

This section covers unexpected reloads and crash-related issues.

Basic Checks

• When did the reload/crash happen?
• Did the switch recover automatically after the crash?

Output to Collect

These outputs are needed:

• show tech-support
  

Note:  The show tech-support output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support | redirect flash:filename  

• dir all
• Any system report/archive file from flash/crashinfo directory created at the time of reload/crash.
• request platform software archive - This command creates an archive file on the flash memory of active switch/supervisor.
• request platform software trace slot RP standby archive - Run this command in case of Catalyst 9400/Catalyst 9600 high availability setup to collect Btrace from standby supervisor. This is a hidden command, configure service internal to run it. This command can create an archive file on the flash memory of the standby supervisor.
• request platform software trace slot switch standby RP archive - Run this command in case of Catalyst 9400 SVL setup to collect Btrace from standby supervisor. This command can create an archive file on the flash memory of the standby supervisor.

7. Module Issues

This section covers module related issues on C9400/C9600 where one/more modules is not able to boot up, is not getting detected or has incorrect status reported on show module output.

Basic Checks

• If day-one issue, is the affected module supported on the chassis in question and Cisco IOS XE version?
• Has the affected module been reseated already?
• Has this module been tried on a different slot? And has a different working module been tried on this slot?

Output to Collect

These outputs are needed:

• show tech-support diagnostic

Note: The  show tech-support diagnostic output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support diagnostic | redirect flash:filename  

• show platform hardware authentication status
• show hw-module subslot <slot_number>/0 oir internal
• show idprom module <slot_number>  (In case of Catalyst 9600, run show idprom module <slot_number> eeprom)
• request platform software archive - This command can create an archive file on the flash memory of active switch/supervisor.

Catalyst 9400 specific commands:

• show platform software iomd <slot_number>/0 oir
• show platform software iomd redundancy
• request platform software trace slot RP standby archive. Run this command in case of Catalyst 9400 high availability setup to collect Btrace from standby supervisor. This is a hidden command, configure service internal to run it. This command can create an archive file on the flash memory of the standby supervisor.

8. Identity (dot1x/MAB)

This section covers dot1x/MAB client authentication on a port.

Basic checks

• Does the issue happen for all clients and ports, or is it specific to a particular type of client and/or particular port?

• Does the issue recover after shut/no shut of the port, disconnect/connect the client or after restarting authentication?

• What is the deployment scenario? Single host, the host behind a phone, multi-access, and so on?

Output to Collect

This information is needed:

• Configure access-session event-logging enable session-limit 200 event-limit 200 and recreate the problem again to generate the logs. Then collect the data below.
• show tech-support identity mac <mac address> interface <type_and_number>

Note: The show tech-support identity output can be large and it is recommended to redirect the output to a file on flash.
     Switch# show tech-support identity | redirect flash:filename  

• request platform software archive - This command creates an archive file on the flash memory of active switch/supervisor

Related Information

• Cisco Technical Support & Downloads