Field Notice: FN - 70016 - Nexus 3000/9000 Switch Might Reload Due to Software Issue - BIOS/Firmware Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	26-Sep-17	Initial Release
1.1	28-Sep-17	table change minor
1.15	11-Oct-17	small table add
1.2	17-Oct-17	add missing products Cisco Nexus 3164Q (N3K-C3164Q) – 7.0(3)I2(2b) and later Cisco Nexus 31128PQ-10GE (N3K-C31128PQ-10GE) – 7.0(3)I4(2) and later Cisco Nexus 3232C (N3K-C3232C) – 7.0(3)I4(1) and later Cisco Nexus 3264Q (N3K-C3264Q) – 7.0(3)I4(1) and later
1.3	26-Sep-18	Updated the Additional Information Section

Products Affected

Affected OS Type	Affected Release	Affected Release Number	Comments
NON-IOS	6	6.1(2)I3(4), 6.1(2)I3(4a), 6.1(2)I3(4e), 6.1(2)I3(5a), 6.1(2)I3(5b), 6.1(2)I3(4c), 6.1(2)I3(4b), 6.1(2)I3(3a)	Refer to the field notice tables for detailed software impact.
NON-IOS	7	7.0(3)I1(1a), 7.0(3)I1(3a), 7.0(3)I1(1b), 7.0(3)I1(1), 7.0(3)I1(3b), 7.0(3)I3(1)	Refer to the field notice tables for detailed software impact.
NON-IOS	7	7.0(3)I4(2), 7.0(3)I4(1), 7.0(3)I2(2b)	Refer to the field notice tables for detailed software impact.

Defect Information

Defect ID	Headline
CSCux61740	Watchdog Timeout During NMI Due to FPGA CRC Error

Problem Description

A Nexus 9000 switch might reload with “Supervisor Device Error” printed as the reset reason. Cisco bug ID CSCux61740 addresses an issue where transient field-programmable gate array (FPGA) Cyclic Redundancy Check (CRC) errors can lead to a crash with this reset reason.

Background

A known fix for Cisco bug ID CSCux61740 is available in the software releases identified in this software field notice. Cisco recommends that you upgrade to the fixed version or later.

Problem Symptom

In order to verify whether your switch reloads due to the transient FPGA CRC errors, enter the show logging nvram command and search for this message in the output:

NMI due to FPGA CRC error

If this message is not explicitly stated in the command output, then enter the show logging onboard stack command. The command output should indicate that a kernel panic event occurred and:

• One or more CPU call traces reference an idle handler.
  For example: (<EOI> [<ffffffff814d9a81>] ? cpuidle_enter_state+0x61/0xe0)

• CPU1 has a large number of interrupts due to "linux-kernel-bde" within the interrupt statistics section.

Workaround/Solution

In order to resolve the issue that is described in this field notice, Cisco recommends that you upgrade the erasable programmable logic device (EPLD) firmware. The new EPLD firmware contains enhancements that better handle the FPGA CRC events on the module in order to avoid overwhelming the main CPU with non-maskable hardware interrupts (NMIs), which causes a crash.

The Cisco Nexus Operating System (NX-OS) requires that the EPLD firmware version be equal to or later than the NX-OS version itself. Therefore, both the NX-OS and EPLD must be upgraded. Upgrades to the NX-OS version alone are not sufficient, so Cisco recommends that you upgrade the NX-OS code and then upgrade the EPLD version.

Refer to this table in order to determine the recommended upgrade versions:

Platform	Recommended NX-OS and EPLD Version(s)
N3K-C3164Q	7.0(3)I2(2b) or later
N3K-C31128PQ-10GE	7.0(3)I4(2) or later
N3K-C3232C	7.0(3)I4(1) or later
N3K-C3264Q	7.0(3)I4(1) or later
N9K-SUP-A	7.0(3)I2(2b) or later
N9K-SUP-B	7.0(3)I2(2b) or later
N9K-C93120TX	7.0(3)I2(2b) or later
N9K-C93128TX	7.0(3)I2(2b) or later
N9K-C9332PQ	7.0(3)I2(2b) or later
N9K-C9372PX	7.0(3)I2(2b) or later
N9K-C9372TX	7.0(3)I2(2b) or later
N9K-C9396PX	7.0(3)I2(2b) or later
N9K-C9396TX	7.0(3)I2(2b) or later
N9K-C9372PX-E	7.0(3)I4(1) or later
N9K-C92160YC-X	7.0(3)I4(1) or later

Notes:

• For Nexus 9000, see Recommended Cisco NX-OS Release for Cisco Nexus 9000 Series Switches for more information.

• For Nexus 3000, see Recommended Cisco NX-OS Release for Cisco Nexus 3000 Series Switches for more information.

How To Identify Affected Products

Enter this command in order to view the EPLD version:

N93120TX-MGMT# slot 1 show hardware internal version

--------------------------------------------------------
Name                InstanceNum     Version    Date
--------------------------------------------------------
Donner              0               0x02
Forwarding ASIC     0               0x03
IO FPGA             0               0x07       20150414
MI FPGA             0               0x09       20140109
MI FPGA 2           0               0x09       20140815
CPLD1               0               0x03
CPLD2               0               0x03

You can then compare it to the EPLD Release Notes for the Cisco Nexus 3000/9000 Series EPLD:

Component	EPLD Device	Release 7.0(3)I3(1)	Release 7.0(3)I4(1)	Release 7.0(3)I4(2)	Release 7.0(3)I4(3)
Cisco Nexus 93120TX (N9K-C93120TX)	IOFPGA	0x8 (0.008)	0x8 (0.008)	0x8 (0.008)	0x8 (0.008)
	MIFPGA1	0x10 (0.016)	0x10 (0.016)	0x10 (0.016)	0x10 (0.016)
	MIFPGA2	0x9 (0.009)	0x9 (0.009)	0x9 (0.009)	0x9 (0.009)

Additional Information

How to Upgrade Software

Fix for "NMI due to FPGA CRC Error"

This is available as an EPLD firmware upgrade. The new EPLD firmware contains enhancements that better handle FPGA CRC events on the module in order to avoid overwhelming the main CPU with NMIs, which causes a crash. Note that NX-OS requires that the EPLD firmware version is the same or earlier than the NX-OS version itself. Therefore, both NX-OS and EPLD need to be upgraded. An upgrade to the NX-OS alone is not sufficient. Cisco recommends for the customer to upgrade the NX-OS code and then upgrade the EPLD version.

Recommended Cisco NX-OS Release for Cisco Nexus 9000 Series Switches

EPLD Installation Procedure

For this list of platforms, both the NX-OS and EPLD on the module must be upgraded to 7.0(3)I2(2b) or later; earlier versions are vulnerable to this defect:

N3K-C3164Q, N9K-SUP-A, N9K-SUP-B, N9K-C93120TX, N9K-C93128TX, N9K-C9332PQ, N9K-C9372PX, N9K-C9372TX, N9K-C9396PX, N9K-C9396TX

For this list of platforms, both the NX-OS and EPLD must be upgraded to 7.0(3)I4(1) or later; earlier versions are vulnerable to this defect:

N3K-C3232C, N3K-C3264Q, N9K-C92160YC-X, N9K-C9372PX-E

For this platform, both the NX-OS and EPLD must be upgraded to 7.0(3)I4(2) or later; earlier versions are vulnerable to this defect. In addition to the documented EPLD installation procedure, you must specifically upgrade the backup EPLD with the install epld bootflash:<epld-file> module all golden command. Substitute in the same EPLD file name as appropriate:

N3K-C31128PQ-10GE