The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes the ISSU upgrade pre-checklist and how to troubleshoot potential issues on the Cisco Catalyst 9800 Series Wireless Controller.
Cisco recommends that you have knowledge of these topics:
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
To perform ISSU, there is a series of background information to follow:
Note: For more information about ISSU Support between Releases visit In-Service Software Upgrade (ISSU).
Ensure that the system is adequately prepared to upgrade the current software to a later release while the network continues to forward packets using the In-Service Software Upgrade option on the 9800 Controllers.
WLC#show version | i Installation mode
Installation mode is INSTALL
WLC#show boot
BOOT variable = bootflash:packages.conf,12;
CONFIG_FILE variable =
BOOTLDR variable does not exist
Configuration register is 0x2102
Standby BOOT variable = bootflash:packages.conf,12;
Standby CONFIG_FILE variable =
Standby BOOTLDR variable does not exist
Standby Configuration register is 0x2102
Note: ISSU is not supported in BUNDLE mode. Refer the Change Installation Modes on Catalyst 9800 Wireless Controller technote on how to do the conversion.
WLC#show rom-monitor chassis active r0
==========================================================
System Bootstrap, Version 16.12(3r), RELEASE SOFTWARE
Copyright (c) 1994-2019 by cisco Systems, Inc.
WLC#show rom-monitor chassis standby r0
==========================================================
System Bootstrap, Version 16.12(3r), RELEASE SOFTWARE
Copyright (c) 1994-2019 by cisco Systems, Inc.
Note: Refer Upgrading Field Programmable Hardware Devices for Cisco Catalyst 9800 to check latest rommon information and upgrade procedure.
WLC#show chassis rmi
Chassis/Stack Mac Address : f4bd.9e57.9d80 - Local Mac Address
Mac persistency wait time: Indefinite
Local Redundancy Port Type: Twisted Pair
H/W Current
Chassis# Role Mac Address Priority Version State IP RMI-IP
--------------------------------------------------------------------------------------------------------
*1 Active f4bd.9e57.9d80 2 V02 Ready 169.254.10.2 NA <--Both Chassis in Ready state
2 Standby f4bd.9e57.8b20 1 V02 Ready 169.254.10.4 NA
Always check redundancy information, ensure peer controller is in "Standby Hot" and Communications "Up". Additionally, always go into the Standby peer controller command line and ensure it is not in Recovery state.
WLC#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 8 hours, 38 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE
Uptime in current state = 8 hours, 38 minutes
Image Version = Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.4a, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Fri 20-Oct-23 10:28 by mcpre
BOOT = bootflash:packages.conf,12;
CONFIG_FILE =
Configuration register = 0x2102
Recovery mode = Not Applicable
Fast Switchover = Enabled
Initial Garp = Enabled
Peer Processor Information :
----------------------------
Standby Location = slot 2
Current Software state = STANDBY HOT
Uptime in current state = 8 hours, 35 minutes
Image Version = Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.4a, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Fri 20-Oct-23 10:28 by mcpre
BOOT = bootflash:packages.conf,12;
CONFIG_FILE =
Configuration register = 0x2102
WLC_CL#show romvar
ROMMON variables:
SWITCH_NUMBER = 1
CHASSIS_HA_IFNAME = GigabitEthernet3 <<< Same HA interface
CHASSIS_HA_IFMAC = XX:XX:XX:XX:XX:XX
SWITCH_PRIORITY = 2
WLC_CL-stby#show romvar
ROMMON variables:
SWITCH_NUMBER = 2
CHASSIS_HA_IFNAME = GigabitEthernet3 <<< Same HA interface
CHASSIS_HA_IFMAC = YY:YY:YY:YY:YY:YY
SWITCH_PRIORITY = 1
If HA Interface numbers are different do not proceed with ISSU. Refer the Potential Problems section in this document.
Tip: For more details regarding HA SSO setup and verification review the High Availability SSO on Catalyst 9800 Quick Start Guide.
Ensure sufficient bootflash memory available to store the image in both active and standby controllers, make sure there is space for at least three times the size of new code.
WLC#dir bootflash: | in free
26458804224 bytes total (21142224896 bytes free)
WLC#dir stby-bootflash: | in free
26458804224 bytes total (21293092864 bytes free)
In the case of not enough space in bootflash consider cleaning up the old installation files using the install remove inactive command.
WLC#install remove inactive
install_remove: START Mon Feb 12 18:40:00 UTC 2024
install_remove: Removing IMG
Cleaning up unnecessary package files
...
The following files will be deleted: <-- Review files to be deleted
[R0]: /bootflash/C9800-CL-rpboot.17.09.04.SPA.pkg
[R1]: /bootflash/C9800-CL-rpboot.17.09.04.SPA.pkg
[R0]: /bootflash/C9800-CL-universalk9.17.12.01.SPA.bin
[R1]: /bootflash/C9800-CL-universalk9.17.12.01.SPA.bin
[R0]: /bootflash/C9800-CL-mono-universalk9.17.09.04.SPA.pkg
[R1]: /bootflash/C9800-CL-mono-universalk9.17.09.04.SPA.pkg
Do you want to remove the above files? [y/n] y
WLC#show install summary
[ Chassis 1/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG C 17.09.04a.0.6 <-- Only one version can be activated and committed.
--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------
WLC#show issu state detail
Current ISSU Status: Enabled
Previous ISSU Operation: N/A
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
Operational Mode HA-REMOTE
=======================================================
No ISSU operation is in progress <-- If different consider abort any previous ISSU process.
Caution: Schedule the upgrade when your network is stable and steady and avoid any power interruption during the ISSU upgrade process.
Note: For ISSU upgrade refer instructions in Chapter: In-Service Software Upgrade from Cisco Catalyst 9800 Series Wireless Controller Software Configuration Guide. Optionally, after ISSU upgrade you can Fallback to Primary controller using the redundancy force-switchover command.
If TFTP server became unreachable, then connection time out can happen.
install_add: START Tue Feb 6 18:12:02 Pacific 2024 Downloading file tftp://10.31.104.72//ayhusain/C9800-L-universalk9_wlc.17.09.05.SPA.bin FAILED: Failed to download file tftp://10.31.104.72//ayhusain/C9800-L-universalk9_wlc.17.09.05.SPA.bin: Timed out
Solution: Ensure the TFTP server is up and reachable or change to a different transport type to start over the file transfer process.
If upgrade is interrupted due to corrupt file or invalid file.
install_add: START Tue Feb 6 18:54:46 Pacific 2024 FAILED: install_add : bootflash:C9800-L-universalk9_wlc.corrupt_17.09.05.SPA.bin is not valid file or cannot be handled by install CLI.
Solution: Ensure the image MD5 checksum matches using the verify /md5 command. If the code does not match download the image again from the official Cisco Software Download site and upload it in the controller. If using tftp server ensure files are sent in binary mode to avoid file modification.
WLC#verify /md5 bootflash:C9800-L-universalk9_wlc.17.09.05.SPA.bin
....................................................................Done!
verify /md5 (bootflash:C9800-L-universalk9_wlc.17.09.05.SPA.bin) = 07ff2f59787530d2814874ea39416b46
Tip: To prevent connectivity problems and save time during the upgrade maintenance window, transfer the bin file to the C9800 flash beforehand.
This scenario is applicable when install activate issu command is issued before completing AP pre-download.
install_activate: START Tue Feb 6 19:09:10 Pacific 2024
System configuration has been modified.
Press Yes(y) to save the configuration and proceed.
Press No(n) for proceeding without saving the configuration.
Press Quit(q) to exit, you may save configuration and re-enter the command. [y/n/q]
y
Building configuration...
[OK]Modified configuration has been saved
install_activate: Activating ISSU
NOTE: Going to start Activate ISSU install process
STAGE 0: System Level Sanity Check
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
--- Verifying Platform specific ISSU admission criteria ---
CONSOLE: FAILED: Install operation is not allowed.
Reason -> AP pre-image download is mandatory for hitless software upgrade.
Action -> Trigger AP pre-image download.
FAILED: Platform specific ISSU admission criteria
ERROR: install_activate exit(2 ) Tue Feb 6 19:09:43 Pacific 2024
Solution: In such instances, run ap image pre-download command and then proceed with the activation only after the AP predownload is completed. If AP download is slow verify the link between AP and the controller.
WLC#terminal monitor
WLC#ap image predownload
*Feb 12 13:26:21.829: %UPGRADE-5-AP_SW_UPDATE_LOG_MSG: Chassis 1 R0/0: wncmgrd: AP SW update PREPARE: predownload is in progress
WLC#show ap image
Total number of APs : 9
Number of APs
Initiated : 0
Downloading : 0
Predownloading : 0
Completed downloading : 1
Completed predownloading : 1
Not Supported : 0
Failed to Predownload : 0
Predownload in progress : Yes
AP Name Primary Image Backup Image Predownload Status Predownload Version Next Retry Time Retry Count Method
------------------------------------------------------------------------------------------------------------------------------------------------------------------
AP00f6.63c8.6a14 17.9.4.27 17.9.5.47 Complete 17.9.5.47 0 0 CAPWAP
AP38ED.18C6.0C60 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
AP70D3.79D6.0A00 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
AP7488.BB66.1CF2 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
AP4C71.0DC8.9D3A 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
AP6C71.0DF4.29CC 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
APA4B2.3986.C900 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
APC828.E56F.6190 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
APA453.0E5B.3E30 17.9.4.27 17.9.5.47 None 0.0.0.0 N/A 0 N/A
WLC#
*Feb 12 13:36:32.832: %UPGRADE-5-AP_SW_UPDATE_LOG_MSG: Chassis 1 R0/0: wncmgrd: AP SW update Predownload is successful. <--AP predownload completed
WLC#install activate issu <-- Then activate ISSU upgrade
install_activate: START Mon Feb 12 13:26:50 Pacific 2024
install_activate: Activating ISSU
*Feb 12 13:26:50.895: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R0/0: install_engine: Started install activate ISSU
NOTE: Going to start Activate ISSU install process
A common event during ISSU upgrade is that the system apparently rollback on its own to previous version hours after the upgrade finished. When the system reloads after activating a new image, the auto-abort-timer is triggered. If the timer expires before committing the new image, then the installation process is aborted; the device reloads again and boots up with the previous version of the software image.
Solution: To avoid this issue make sure to issue the 'install commit' command within 6 hours (default time) of the 'install activate issu'command, otherwise the system can revert to the original committed position.
WLC#show issu state detail
Current ISSU Status: In Progress
Previous ISSU Operation: N/A
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
Operational Mode HA-REMOTE
=======================================================
Added Image:
Name Compatible
-------------------------------------------------------
17.09.05.0.6450 Yes
Operation type: Step-by-step ISSU
Install type : Image installation using ISSU
Current state : Activated state
Last operation: Switchover
Completed operations:
Operation Start time
-------------------------------------------------------
Activate location standby chassis 2/R0 2024-02-12:13:27:12
Activate location active chassis 1/R0 2024-02-12:13:38:43
Switchover 2024-02-12:13:43:10
State transition: Added -> Standby activated -> Active switched-over
Auto abort timer: automatic, remaining time before rollback: 04:25:37<-- Remaining time before rollback
Abort Reason: N/A
Running image: bootflash:packages.conf
Operating mode: sso, terminal state reachedWLC#
show install summary
[ Chassis 1/R0 2/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG U 17.09.05.0.6450<-- System is activated but uncommited
--------------------------------------------------------------------------------
Auto abort timer: active , time before rollback - 04:25:52<-- Remaining time before rollback
--------------------------------------------------------------------------------
WLC#install commit <-- Commit the new code
install_commit: START Mon Feb 12 15:20:35 Pacific 2024
install_commit: Committing ISSU
*Feb 12 15:20:36.362: %INSTALL-5-INSTALL_START_INFO: Chassis 2 R0/0: install_engine: Started install commit
NOTE: Going to start Commit ISSU install process
STAGE 0: System Level Sanity Check
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
--- Verifying Platform specific ISSU admission criteria ---
Finished Initial System Level Sanity Check
STAGE 1: Dispatching the commit command to remote
===================================================
--- Starting install_commit_remote ---
Performing install_commit_remote on Chassis remote
[1] install_commit_remote package(s) on chassis 1/R0
[1] Finished install_commit_remote on chassis 1/R0
install_commit_remote: Passed on [1/R0]
Finished install_commit_remote
SUCCESS: install_commit Mon Feb 12 15:21:12 Pacific 2024
WLC#
*Feb 12 15:21:12.367: %INSTALL-5-INSTALL_COMPLETED_INFO: Chassis 2 R0/0: install_engine: Completed install commit ISSU
WLC#show issu state detail <-- Verify ISSU state is successful
Current ISSU Status: Enabled
Previous ISSU Operation: Successful
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
Operational Mode HA-REMOTE
=======================================================
No ISSU operation is in progress
WLC#show install summary <-- Verify new code is Activated & Committed
[ Chassis 1/R0 2/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG C 17.09.05.0.6450
--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------
(Optional) To stop the auto-abort-timer use the install auto-abort-timer stop command. This stops the termination timer so that the upgrade process is not terminated after the default auto-abort time.
WLC#install auto-abort-timer stop
install_auto_abort_timer: START Mon Feb 12 15:13:59 Pacific 2024
Abort timer cancelled, auto-rollback will not take place
Please use 'install abort issu' to rollback the ISSU upgrade
SUCCESS: install_auto_abort_timer Mon Feb 12 15:14:00 Pacific 2024
WLC#show install summary
[ Chassis 1/R0 2/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG U 17.09.05.0.6450<-- You still need to commit new image to persist
--------------------------------------------------------------------------------
Auto abort timer: inactive<-- Remaining time before rollback. Commit still needed for new image to persist.
--------------------------------------------------------------------------------
(Optional) To modify the auto abort timer use the activate issu auto-abort-timer command. This can be done before activating the image.
WLC#install activate issu auto-abort-timer ?
<30-1200> Number of minutes the abort-timer is set for
Certain configurations cannot be compatible with the new IOS-XE version and can cause issues during the upgrade. Configuration synchronization failures cause the standby controller reload and rollback to original software image.
To resolve configuration issues remove the offending configuration. Provided a list of the most common configuration compatibility issues seen during ISSU upgrades.
Tip: Always review the Before You Upgrade section in the target image Release Notes for important details on software changes and configuration adjustments to consider prior to the upgrade.
A common error in ISSU upgrade is when the C9800-CL WLCs have defined different High Availability Interfaces causing the Mismatched Command List (MCL) to terminate the ISSU Operation on the active controller.
Solution: If your controllers have different HA interfaces then ISSU is not possible. Redeploy your controllers and ensure the HA interface mapping in the virtual environment is the same for both controllers.
Note: Refer the 9800-CL network interface mappings section in the Cisco Catalyst 9800-CL Wireless Controller for Cloud Deployment Guide.
WLC#show redundancy config-sync failures historic mcl
Mismatched Command List
-----------------------
-interface GigabitEthernet3 <-- Gi3 is detected as MCL leading to config sync failure.
! <submode> "interface"
negotiation auto
no mop enabled
no mop sysid
! </submode> "interface"
WLC#show romvar
ROMMON variables:
SWITCH_NUMBER = 1
CHASSIS_HA_IFNAME = GigabitEthernet3 <-- HA Interface differs
CHASSIS_HA_IFMAC = XX:XX:XX:XX:XX:XX
SWITCH_PRIORITY = 2
WLC-stby#show romvar
ROMMON variables:
SWITCH_NUMBER = 2
CHASSIS_HA_IFNAME = GigabitEthernet1 <-- HA Interface differs
CHASSIS_HA_IFMAC = YY:YY:YY:YY:YY:YY
SWITCH_PRIORITY = 1
Controller upgrade from Cisco IOS 17.3.x to any release using ISSU can fail if the domain command is configured.
Solution: Run the no domain command before starting an ISSU upgrade because the domain command has been removed from Cisco IOS 17.6.x.
WLC#show redundancy config-sync failures mcl
Mismatched Command List
-----------------------
-domain example.local. <-- Run "no domain" from configuration
! <submode> "cent_domain"
! </submode> "cent_domain"
Controller upgrade from Cisco IOS 17.3.x to any release using ISSU can fail if the snmp-server enable traps hsrp command is configured.
Solution: Remove the snmp-server enable traps hsrp command from the configuration before starting an ISSU upgrade because that command has been removed from Cisco IOS 17.4.x.
WLC#show redundancy config-sync failures mcl Mismatched Command List ----------------------- -snmp-server enable traps hsrp
<-- Remove hsrp trap
Controller upgrade to Cisco IOS 17.9.x and onwards using ISSU can fail if there is a WLAN that is misconfigured with WPA3 without Authentication Key Management (AKM). Starting Cisco IOS 17.9.x release it is mandatory to have a valid combination, requiring the enforcement of the AKM parameter for WPA3. During the ISSU process, if the WLC detects an invalid WPA3 configuration, it leads to a configuration synchronization failure.
Solution: Avoid this condition by configuring the appropriate AKM for WPA3 before starting ISSU upgrade.
WLC#show redundancy config-sync failures prc
PRC Failed Command List
-----------------------
wlan test 1 test
! <submode> "wlan"
- no shutdown <-- The wlan shuts down due to the invalid WPA3 configuration
! </submode> "wlan"
Note: To configure WPA3, see the Chapter: Wi-Fi Protected Access 3 of the Cisco Catalyst 9800 Series Wireless Controller Software Configuration Guide
You can manually abort the activation of a software image using the install abort issucommand to cancel current process and return the system to the version that was running before installing the new software. Use this command before issuing the install commitcommand.
WLC#terminal monitor
WLC#install abort issu
install_abort: START Fri Jan XX 16:50:00 EDT 2024
<5>%INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install abort ISSU
Once finished verify ISSU status.
WLC#show issu state detail
Current ISSU Status: Enabled
Previous ISSU Operation: Abort Successful
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
=======================================================
No ISSU operation is in progress
If there is an unwanted ISSU operaton that could not be cleared issuing the abort command you can manually clean up the internal installation states.
Caution: Use this procedure with caution as reload can be required for clear install state process.
1. Enable Service Internal in the global configuration mode.
WLC# conf t
WLC(config)# service internal
WLC(config)# end
2. Clean ISSU state using the clear install state command.
WLC#clear install state
3. Verify the install state using the show install summary and show issu state detail command.
4. Disable Service Internal in the global configuration mode.
WLC# conf t
WLC(config)# no service internal
WLC(config)# end
To verify the upgrade process and to check the mismatched command list.
#show install summary
#show issu state detail
#show install log
#show chassis rmi
#show redundancy
#show ap image
#show redundancy config-sync failures mcl
#show redundancy config-sync failures historic mcl
#show redundancy config-sync failures bem
#show redundancy config-sync failures prc
Revision | Publish Date | Comments |
---|---|---|
1.0 |
16-Feb-2024 |
Initial Release |