Field Notice: FN - 72080 - ISA 3000 Industrial Security Appliance - Some Appliances Might Fail to Pass Traffic After 3.2 Years Of Uptime - Power Cycle Required - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	26-Feb-21	Initial Release
1.1	18-Mar-21	Updated the Workaround/Solution Section
1.2	11-May-21	Updated the Products Affected Section
1.3	14-Jun-21	Updated the Products Affected Section, Defect Information, Workaround/Solution Sections, SNV link

Products Affected

Affected Product ID	Comments
ISA-3000-2C2F-K9=	Part Alternate
ISA-3000-2C2F-K9
ISA-3000-2C2F-FTD
ISA-3000-4C-K9=	Part Alternate
ISA-3000-4C-K9
ISA-3000-4C-FTD

Defect Information

Defect ID	Headline
CSCvx09123	M500IT Model Solid State Drives on ISA3000 may go unresponsive after 3.2 Years in service

Problem Description

Due to a flaw in Solid State Drive (SSD) firmware, the SSD internal to the ISA 3000 security appliance will no longer respond after approximately 3.2 years of cumulative operation. After the first unresponsive event is experienced, every subsequent ISA 3000 power-cycle will allow the SSD to operate for approximately six weeks of cumulative operation before the SSD will no longer respond again.

Background

After 28,224 hours (approximately 3.2 years) of accumulated Power On Hours (POH), a memory buffer overrun condition occurs which triggers the firmware event in the SSD. This causes the drive to become unresponsive until it is power-cycled. No data loss will occur when the memory buffer overrun firmware event occurs. A power-cycle of the ISA 3000 security appliance restores normal operation of the drive. The drive continues to operate normally for 1008 additional accumulated POH (six weeks), at which time the drive will become unresponsive again. Power-cycling the ISA 3000 security appliance again will re-initiate the 1008 hour window.

The ISA 3000 SSD is used for data storage when running ASA with Firepower Services or Firepower Threat Defense (FTD) software.

Problem Symptom

The ISA 3000 Industrial Security Appliance no longer passes network traffic. The management console will be responsive, but users with valid credentials might not be able to log in. Previously logged in sessions might continue with reduced functionality.

Workaround/Solution

Workaround

A power-cycle of the ISA 3000 security appliance is required in order to temporarily recover from this issue. However, this failure will reappear after 1008 hours of operation.

Solution

In order to prevent occurrence of this issue and disruption to the network and operations, Cisco recommends to upgrade the firmware of the SSD proactively before the accumulated uptime reaches 28,224 hours. Refer to the How To Identify Affected Products section and follow the firmware upgrade procedure accordingly.

If the system is already impacted, the SSD firmware upgrade will permanently resolve this defect.

A product return and replacement (RMA) is not recommended as the firmware upgrade process will resolve the issue.

Upgrade the ASA or Firepower software to update the SSD firmware and resolve the issue. A service contract is not required to download these software images.

Software Upgrade for ASA-Only or ASA with Firepower Services

For ASA-based platforms, upgrade to one of these ASA software versions to update the SSD firmware. This software is available from the Cisco Software Download Center.

• 9.8.4.33 or later
• 9.9.2.83 or later
• 9.12.4.13 or later
• 9.13.1.19 or later
• 9.14.2.8 or later
• 9.15.1.7 or later

Refer to the Cisco ASA Upgrade Guide for instructions on how to upgrade your ASA software.

Additional ASA software upgrade guidance is available in the ASA 9.x : Upgrade a Software Image using ASDM or CLI Configuration Example.

Software Upgrade for FTD

For ISA 3000 appliances that run FTD, complete one of the options in this section.

Option 1: Upgrade to one of these FTD Hotfix versions in order to update the SSD firmware for the required software release.

• All 6.2.3.x Releases use: Cisco_FTD_Hotfix_EH-6.2.3.999-6.sh.REL.tar
• All 6.3.0.x Releases use: Cisco_FTD_Hotfix_AZ-6.3.0.999-1.sh.REL.tar
• All 6.4.0.x Releases use: Cisco_FTD_Hotfix_DV-6.4.0.999-1.sh.REL.tar
• All 6.5.0.x Releases use: Cisco_FTD_Hotfix_T-6.5.0.999-1.sh.REL.tar
• All 6.6.0.x Releases use: Cisco_FTD_Hotfix_N-6.6.0.999-1.sh.REL.tar
• All 6.6.1.x Releases use: Cisco_FTD_Hotfix_AB-6.6.1.999-1.sh.REL.tar
• All 6.7.0.x Releases use: Cisco_FTD_Hotfix_C-6.7.0.999-2.sh.REL.tar

See the Cisco Firepower Hotfix Release Notes for instructions on how to install the Firepower Hotfix.

The Firepower Hotfix can be applied with Firepower Management Center (FMC) or with Firepower Device Manager (FDM).

Note: You cannot roll back to a previous ASA software version, uninstall the Firepower Hotfix, or reimage the ISA 3000 security appliance in order to downgrade the SSD firmware after it has been updated.

Option 2: Upgrade to the latest Software Release 7.0: Cisco_FTD_Upgrade-7.0.0-94.sh.REL.tar

See the Cisco Firepower Threat Defense Configuration Guide for instructions on how to upgrade to Release 7.0 with FDM.

Note: Refer to Upgrading Firepower Threat Defense Software before you upgrade the software.

See the Firepower Management Center Configuration Guide for instructions on how to upgrade to Release 7.0 with FMC.

Note: Refer to Upgrade System Software before you upgrade the software.

How To Identify Affected Products

The SSD installed in the ISA 3000 and its firmware version can be determined with one of these procedures.

ASA-Only or ASA with Firepower Services

Cisco ISA 3000 Series security appliances that run Cisco ASA FirePOWER Module might be affected and Cisco recommends to upgrade the ASA software.

Cisco ISA 3000 Series security appliances that run ASA-only software are not affected since the SSD is not used to store information for this application. However, Cisco recommends to upgrade the ASA software in case the ISA 3000 is used to provide Firepower Services or converted to FTD software at a future date.

There are no previously defined CLI commands available in ASA software to display the SSD information.

However, the ASA software versions that fix this issue (see the Workaround/Solution section) provide additional debug commands to display system log files and show if the SSD firmware upgrade was performed. The new debug commands are:

debug menu file-system 9
debug menu file-system 10 (provides additional details)

A sample output of the debug menu file-system 9 command is shown here.

firepower# debug menu file-system
WARNING: These utilities are intended for troubleshooting purposes only.
Use of these utilities can severely impact performance and the proper
operation of the device.
File-system commands are:
*** Debug menu displayed ***
firepower# debug menu file-system 9
-------------------------------------------------------------------------------
micron-ssd-firmware.sh: Called at (Tue Jan 26 17:57:34 UTC 2021)
-------------------------------------------------------------------------------
*** Output continues ***

Completed the firmware upgrade, verifying...
Successfully upgraded the SSD firmware on Micron_M500IT_MTFDDAT064MBD to CT02
firepower#

Firepower Software

For ISA 3000 security appliances that run FTD, the SSD model and its firmware revision can be determined as follows:

1. Connect to the ISA 3000 and log in as an admin user via either a SSH or serial console connection.
2. Enter the expert command to switch to expert mode.
3. Enter this command to display the SSD model and current firmware revision:

   sudo hdparm -I /dev/sda | egrep 'Model Number | Firmware Revision'

A sample output of this command is shown here:

> expert
$ sudo hdparm -I /dev/sda | egrep 'Model Number|Firmware Revision'
Model Number:       M500IT_MTFDDAT050MBD
Firmware Revision:  CT01.00

Cisco ISA 3000 Security Appliances

If the SSD Model Number is Micron_M500IT_* and the Firmware Revision is CT01.00, then a Firepower software upgrade is required to prevent the issue.

Serial Number Validation

This field notice provides the ability to determine if the serial number(s) of a device is impacted by this issue. In order to verify your serial number(s), enter it in the Serial Number Validation tool.

Please note: For security reasons, you must click on the preceeding SNV link. Use of the link external to this field notice will fail.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.