Troubleshooting Smart Licensing Using Policy

This section provides the list of Smart Licensing Using Policy-related system messages you may encounter, possible reasons for failure, and recommended action.

System Message Overview

The system software sends system messages to the console (and, optionally, to a logging server on another system). Not all system messages mean problems with your system. Some messages are informational, and others can help diagnose problems with communications lines, internal hardware, or the system software.

How to Read System Messages

System log messages can contain up to 80 characters. Each system message begins with a percent sign (%) and is structured as follows:

%FACILITY

Two or more uppercase letters that show the facility to which the message refers. A facility can be a hardware device, a protocol, or a module of the system software

SEVERITY

A single-digit code from 0 to 7 that reflects the severity of the condition. The lower the number, the more serious the situation.

Table 1. Message Severity Levels

Severity Level

Description

0 - emergency

System is unusable.

1 - alert

Immediate action required.

2 - critical

Critical condition.

3 - error

Error condition.

4 - warning

Warning condition.

5 - notification

Normal but significant condition.

6 - informational

Informational message only.

7 - debugging

Message that appears during debugging only.

MNEMONIC

A code that uniquely identifies the message.

Message-text

Message-text is a text string describing the condition. This portion of the message sometimes contains detailed information about the event, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memory address space. Because the information in these variable fields changes from message to message, it is represented here by short strings enclosed in square brackets ([ ]). A decimal number, for example, is represented as [dec].

Table 2. Variable Fields in Messages

Severity Level

Description

[char]

Single character

[chars]

Character string

[dec]

Decimal number

[enet]

Ethernet address (for example, 0000.FEED.00C0)

[hex]

Hexadecimal number

[inet]

Internet address (for example, 10.0.2.16)

[int]

Integer

[node]

Address or node name

[t-line]

Terminal line number in octal (or in decimal if the decimal-TTY service is enabled)

[clock]

Clock (for example, 01:20:08 UTC Tue Mar 2 1993

System Messages

This section provides the list of Smart Licensing Using Policy-related system messages you may encounter, possible reasons for failure (incase it is a failure message), and recommended action (if action is required).

For all error messages, if you are not able to solve the problem, contact your Cisco technical support representative with the following information:

  • The message, exactly as it appears on the console or in the system log.

  • The output from the show license tech support , show license history message , and the show platform software sl-infra privileged EXEC commands.

Smart Licensing Using Policy-related system messages:

Error Message %SMART_LIC-3-POLICY_INSTALL_FAILED: The installation of a new 
licensing policy has failed: [chars].

Explanation: A policy was installed, but an error was detected while parsing the policy code, and installation failed. [chars] is the error string with details of the failure.

Possible reasons for failure include:

  • A signature mismatch: This means that the system clock is not accurate.

  • A timestamp mismatch: This means the system clock on the product instance is not synchronized with CSSM.

Recommended Action:

For both possible failure reasons, ensure that the system clock is accurate and synchronized with CSSM. Configure the ntp server command in global configuration mode. For example:
Device(config)# ntp server 198.51.100.100 version 2 prefer

If the above does not work and policy installation still fails, contact your Cisco technical support representative.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-3-AUTHORIZATION_INSTALL_FAILED: The install of a new 
licensing authorization code has failed on [chars]: [chars].

Explanation: Authorization code installation was attempted, but installation failed. The first [chars] is the UDI for which the authorization code installation failed, and the second [chars] is the error string with details of the failure.

Possible reasons for failure include:

  • Not enough licenses with authorization for currently configured features: This means that you have not provided the requisite number of authorization codes.

  • UDI mismatch: One or more UDIs in the authorization code file do not match with the product instance where you are installing the authorization code file. If you have generated authorization codes for multiple UDIs, for a High Availability or stacking set-up, all the UDIs listed in the authorization code file must match with all the UDIs in the High Availability or stacking set-up. If this is not the case, installation fails.

    Cross-check all UDIs in the authorizatin code file against the UDIs of the product instance (standalone or High Availability).
    Excerpt of UDI information in a SLAC file: 
    <smartLicenseAuthorization>
    <udi>P:C9300X-24HX,SN:FOC2519L8R7</udi>
    
    <output truncated>
    </smartLicenseAuthorization>
    
    Sample output of UDI information on a product instance:
    Device# show license udi
    UDI: PID:C9300X-24HX,SN:FOC2519L8R7
  • A signature mismatch: This means that the system clock is not accurate. If the clock is not synchronized, your attempts at requesting SLAC are not reflected in the show license tech output.
    Authorization Confirmation:
      Attempts: Total=0, Success=0, Fail=0  Ongoing Failure: Overall=0 Communication=0

Recommended Action

  • In the output of the show license tech support command, check the Failure Reason: field to understand what may have gone wrong.
    Device# show license tech support
    <output truncated>
    
    Communication Statistics:
    =======================
    Authorization Confirmation:
      Attempts: Total=2, Success=2, Fail=0  Ongoing Failure: Overall=0 Communication=0
      Last Response: OK on Sep 23 17:51:52 2020 UTC
        Failure Reason: <none>
      Last Success Time: Sep 23 17:51:52 2020 UTC
      Last Failure Time: <none>
  • Not enough licenses in authorization for currently configured features and UDI mismatch:

  • Use the show license udi command to verify that you have the correct and complete list of UDIs. This command displays all product instances in case of High Availability and stacking set-up. Then install SLAC again.

  • Signature mismatch: Ensure that the system clock is accurate and synchronized with CSSM. To do this, configure the ntp server command in global configuration mode. For example:
    Device(config)# ntp server 198.51.100.100 version 2 prefer

    After you complete this configuration, again use the show license tech to verify if the clock has actually synchronized. If successfully synchronized, the Clock sync-ed with NTP field is set to True. If not synchronized, this field is set to False.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-3-COMM_FAILED: Communications failure with the [chars] :
[chars]

Explanation: Smart Licensing communication either with CSSM, CSLU, or SSM On-Prem failed. The first [chars] is the currently configured transport type, and the second [chars] is the error string with details of the failure. This message appears for every communication attempt that fails.

Possible reasons for failure include:

  • CSSM, CSLU, SSM On-Prem is not reachable: This means that there is a network reachability problem.

  • 404 host not found: This means the CSSM server is down.

  • A TLS or SSL handshake failure caused by a missing client certificate. The certificate is required for TLS authentication of the two communicating sides. A recent server upgrade may have cause the certificate to be removed. This reason applies only to a topology where the product instance is directly connected to CSSM.


    Note


    If the error message is displayed for this reason, there is no actual configuration error or disruption in the communication with CSSM.


For topologies where the product instance initiates the sending of RUM reports (Connected to CSSM Through CSLU: Product Instance-Initiated Communication, Connected Directly to CSSM, CSLU Disconnected from CSSM: Product Instance-Initiated Communication, and SSM On-Prem Deployment: Product Instance-Initiated Communication) if this communication failure message coincides with scheduled reporting (license smart usage interval interval_in_days global configuration command), the product instance attempts to send out the RUM report for up to four hours after the scheduled time has expired. If it is still unable to send out the report (because the communication failure persists), the system resets the interval to 15 minutes. Once the communication failure is resolved, the system reverts the reporting interval to last configured value.

Recommended Action:

Troubleshooting steps are provided for when CSSM is not reachable or there is a missing client certificate, when CSLU is not reachable, and when SSM On-Prem is not reachable.

  • If a client certificate is missing and there is no actual configuration error or disruption in the communication with CSSM:

    To resolve the error, configure the ip http client secure-trustpoint trustpoint-name command in global configuration mode. For trustpoint-name, enter only SLA-TrustPoint. This command specifies that the secure HTTP client should use the certificate associated with the trustpoint indicated by the trustpoint-name argument.

  • If CSSM is not reachable and the configured transport type is smart:

    1. Check if the smart URL is configured correctly. Use the show license status command in privileged EXEC mode, to check if the URL is exactly as follows: https://smartreceiver.cisco.com/licservice/license. If it is not, reconfigure the license smart url smart smar_URL command in global configuration mode.

    2. Check DNS resolution. Verify that the product instance can ping smartreceiver.cisco.com or the nslookup translated IP. The following example shows how to ping the translated IP
      Device# ping 171.70.168.183
      Type escape sequence to abort.
      Sending 5, 100-byte ICMP Echos to 171.70.168.183, timeout is 2 seconds:
      !!!!!
      Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
  • If CSSM is not reachable and the configured transport type is callhome:

    1. Check if the URL is entered correctly. Use the show license status command in privileged EXEC mode, to check if the URL is exactly as follows: https://tools.cisco.com/its/service/oddce/services/DDCEService.

    2. Check if Call Home profile CiscoTAC-1 is active and destination URL is correct. Use the show call-home profile all command in privileged EXEC mode:
      Current smart-licensing transport settings:
       Smart-license messages: enabled
       Profile: CiscoTAC-1 (status: ACTIVE)
       Destination  URL(s):  https://tools.cisco.com/its/service/oddce/services/DDCEService
      
    3. Check DNS Resolution. Verify that the product instance can ping tools.cisco.com, or the nslookup translated IP.
      Device# ping tools.cisco.com
      Type escape sequence to abort.
      Sending 5, 100-byte ICMP Echos to 173.37.145.8, timeout is 2 seconds:
      !!!!!
      Success rate is 100 percent (5/5), round-trip min/avg/max = 41/41/42 ms
      

      If the above does not work check the following: if the product instance is set, if the product instance IP network is up. To ensure that the network is up, configure the no shutdown command in interface configuration mode.

      Check if the device is subnet masked with a subnet IP, and if the DNS IP is confgured.

    4. Verify that the HTTPs client source interface is correct.

      Use the show ip http client command in privileged EXEC mode to display current configuration. Use ip http client source-interface command in global configuration mode to reconfigure it.

      In case the above does not work, double-check your routing rules, and firewall settings.

  • If CSLU is not reachable:

    1. Check if CSLU discovery works.

      • Zero-touch DNS discovery of cslu-local or DNS discovery of your domain..

        In the show license all command output, check if the Last ACK received: field. If this has a recent timestamp it means that the product instance has connectivity with CSLU. If it is not, proceed with the following checks:

        Check if the product instance is able to ping cslu-local. A successful ping confirms that the product instance is reachable.

        If the above does not work, configure the name server with an entry where hostname cslu-local is mapped to the CSLU IP address (the windows host where you installed CSLU). Configure the ip domain name domain-name and ip name-server server-address commands in global configuration mode. Here the CSLU IP is 192.168.0.1 and name-server creates entry cslu-local.example.com:
        Device(config)# ip domain name example.com
        Device(config)# ip name-server 192.168.0.1
      • CSLU URL is configured.

        In the show license all command output, under the Transport: header check the following: The Type: must be csluand Cslu address: must have the hostname or the IP address of the windows host where you have installed CSLU. Check if the rest of the address is configured as shown below and check if the port number is 8182.
        Transport:
          Type: cslu
          Cslu address: http://192.168.0.1:8182/cslu/v1/pi
        If it is not, configure the license smart transport cslu and license smart url cslu http://<cslu_ip_or_host>:8182/cslu/v1/pi commands in global configuration mode
    2. For CSLU-initiated communication, in addition to the CSLU discovery checks listed above, check the following:

      Verify HTTP connectivity. Use the show ip http server session-module command in privileged EXEC mode. In the output, under header HTTP server current connections:, check that SL_HTTP is active. If it is not re-configure the ip http commands as mentioned in Ensuring Network Reachability for CSLU-Initiated Communication.

      From a Web browser on the device where CSLU is installed, verify https://<product-instance-ip>/. This ensures that the REST API from CSLU to the product instance works as expected.

  • If SSM On-Prem is not reachable:

    1. For product instance-initiated communication, check if the SSM On-Prem transport type and URL are configured correctly.

      In the show license all command output, under the Transport: header check the following: The Type: must be csluand Cslu address: must have the hostname or the IP address of the server where you have installed SSM On-Prem and <tenantID> of the default local virtual account. See the example below:
      Transport:
        Type: cslu
        Cslu address: https://192.168.0.1/cslu/v1/pi/on-prem-default
      
      Check if you have the correct URL from SSM On-Prem (See Retrieving the Transport URL (SSM On-Prem UI)) and then configure license smart transport cslu and license smart url cslu http://<ip>/cslu/v1/pi/<tenant ID> commands in global configuration mode.

      Check that you have configured any other required commands for your network, as mentioned in Ensuring Network Reachability for Product Instance-Initiated Communication

    2. For SSM On-Prem-initiated communication, check HTTPs connectivity.

      Use the show ip http server session-module command in privileged EXEC mode. In the output, under header HTTP server current connections:, check that SL_HTTP is active. If it is not re-configure the ip http commands as mentioned in Ensuring Network Reachability for SSM On-Prem-Initiated Communication.

    3. Check trustpoint and that certificates are accepted.

      For both forms of communication in an SSM On-Prem Deployment, ensure that the correct trustpoint is used and that the necessary certificates are accepted:
      Device(config)# crypto pki trustpoint SLA-TrustPoint 
      Device(ca-trustpoint)#
      Device(ca-trustpoint)# enrollment terminal
      Device(ca-trustpoint)# revocation-check none
      Device(ca-trustpoint)# end
      Device# copy running-config startup-config

If the above does not work and the communication failure persists, contact your Cisco technical support representative.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message  %SMART_LIC-3-COMM_RESTORED: Communications with the [chars] restored.
[chars] - depends on the transport type
        - Cisco Smart Software Manager (CSSM)
        - Cisco Smart License utility (CSLU)
Smart Agent communication with either the Cisco Smart Software Manager (CSSM) or the Cisco Smart License 
utility (CSLU) has been restored. No action required.
 

Explanation: Product instance communication with either the CSSM, CSLU, or SSM On-Prem is restored.

Recommended Action: No action required.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-3-POLICY_REMOVED: The licensing policy has been removed.

Explanation: A previously installed custom licensing policy has been removed. The Cisco default policy is then automatically effective. This may cause a change in the behavior of smart licensing.

Possible reasons for failure include:

If you have entered the license smart factory reset command in privileged EXEC mode all licensing information including the policy is removed.

Recommended Action:

If the policy was removed intentionally, then no further action is required.

If the policy was removed inadvertantly, you can reapply the policy. Depending on the topology you have implemented, follow the corresponding method to retrieve the policy:


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-3-TRUST_CODE_INSTALL_FAILED: The install of a new licensing 
trust code has failed on [chars]: [chars].

Explanation: Trust code installation has failed. The first [chars] is the UDI where trust code installation was attempted. The second [chars] is the error string with details of the failure.

Possible reasons for failure include:

  • A trust code is already installed: Trust codes are node-locked to the UDI of the product instance. If the UDI is already registered, and you try to install another one, installation fails.

  • Smart Account-Virtual Account mismatch: This means the Smart Account or Virtual Account (for which the token ID was generated) does not include the product instance on which you installed the trust code. The token generated in CSSM, applies at the Smart Account or Virtual Account level and applies only to all product instances in that account.

  • A signature mismatch: This means that the system clock is not accurate.

  • Timestamp mismatch: This means the product instance time is not synchronized with CSSM, and can cause installation to fail.

Recommended Action:

  • A trust code is already installed: If you want to install a trust code inspite of an existing trust code on the product instance, re-configure the license smart trust idtoken id_token_value{ local| all} [ force] command in privileged EXEC mode, and be sure to include the force keyword this time. Entering the force keyword sets a force flag in the message sent to CSSM to create a new trust code even if one already exists.

  • Smart Account-Virtual Account mismatch:

    Log in to the CSSM Web UI at https://software.cisco.com. Under Smart Software Licensing, click the Manage licenseslink. Click the Inventory tab. From the Virtual Account drop-down list, choose the required virtual account. Click the Product Instances tab.

    Check if the product instance on which you want to generate the token is listed in the selected Virtual Account. If it is, proceed to the next step: Generating a New Token for a Trust Code from CSSM and Establishing Trust with an ID Token. If not, check and select the correct Smart Account and Virtual Account. Then complete the next steps.

  • Timestamp mismatch and signature mismatch: Configure the ntp server command in global configuration mode. For example:
    Device(config)# ntp server 198.51.100.100 version 2 prefer

----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message    %SMART_LIC-4-REPORTING_NOT_SUPPORTED: The CSSM OnPrem that this 
product instance is connected to is down rev and does not support the enhanced policy and usage 
reporting mode.
 

Explanation: Cisco Smart Software Manager On-Prem (formerly known as Cisco Smart Software Manager satellite) is supported in the Smart Licensing Using Policy environment starting with Cisco IOS XE Amsterdam 17.3.3 only (See SSM On-Prem Deployment). In unsupported releases, the product instance will behave as follows:

  • Stop sending registration renewals and authorization renewals.

  • Start recording usage and saving RUM reports locally.

Recommended Action:

You have the following options:

  • Refer to and implement one of the supported topologies instead. See: Connecting to Cisco SSM.

  • Upgrade to a release where SSM On-Prem is supported with Smart Licensing Using Policy. <<Migrating to a Version of SSM On-Prem That Supports Smart Licensing Using Policy>>


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-6-POLICY_INSTALL_SUCCESS: A new licensing policy 
was successfully installed.

Explanation: A policy was installed in one of the following ways:

  • Using Cisco IOS commands.

  • CSLU-initiated communication.

  • As part of an ACK response.

Recommended Action: No action is required. If you want to know which policy is applied (the policy in-use) and its reporting requirements, enter the show license all command in privileged EXEC mode.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-6-AUTHORIZATION_INSTALL_SUCCESS: A new licensing 
authorization code was successfully installed on: [chars].

Explanation: [chars] is the UDI where the authorization code was installed successfully.

Recommended Action: No action is required. If you want to know the details of the authorization code that was installed, enter the show license authorization command in privileged EXEC mode.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-6-AUTHORIZATION_REMOVED: A licensing authorization code has
 been removed from [chars]

Explanation: [chars] is the UDI where the authorization code was installed. The authorization code has been removed. This removes the licenses from the product instance and may cause a change in the behavior of smart licensing and the features using licenses.

Recommended Action: No action is required. If you want to see the current state of the license, enter the show license all command in privileged EXEC mode.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-6-REPORTING_REQUIRED: A Usage report acknowledgement 
will be required in [dec] days.

Explanation: This is an alert which means that RUM reporting to Cisco is required. [dec] is the amount of time (in days) left to meet this reporting requirements.

Recommended Action: Ensure that RUM reports are sent within the requested time. The topology you have implemented determines the reporting method.

  • Connected to CSSM Through CSLU

    • For product instance-initiatied communication: Enter the license smart sync command in privileged EXEC mode. If CSLU is currently logged into CSSM the reports will be automatically sent to the associated Smart Account and Virtual Account in CSSM.

    • For CSLU-initiated communication, complete this task: Collecting Usage Reports: CSLU Initiated (CSLU Interface).

  • Connected Directly to CSSM: Enter the license smart sync command in privileged EXEC mode.

  • Connected to CSSM Through a Controller: If the product instance is managed by a controller, the controller will send the RUM report at the scheduled time.

    If you are using Cisco DNA Center as the controller, you have the option of ad-hoc reporting. See the Cisco DNA Center Administrator Guide of the required release (Release 2.2.2 onwards) > Manage Licenses > Upload Resource Utilization Details to CSSM.

  • CSLU Disconnected from CSSM: If the product instance is connected to CSLU, synchronize with the product instance as shown for "Connected to CSSM Through CSLU"above, then complete these tasks: Export to Cisco SSM (CSLU Interface) > Uploading Data or Requests to Cisco SSM and Downloading a File > Import from Cisco SSM (CSLU Interface).

  • No Connectivity to CSSM and No CSLU: Enter the license smart save usage command in privileged EXEC mode, to save the required usage information in a file. Then, from a workstation where you have connectivity to CSSM, complete these tasks: Uploading Data or Requests to Cisco SSM and Downloading a File > Installing a File on the Product Instance.

  • SSM On-Prem Deployment:

    Synchronize the product instance with SSM On-Prem:

    • For product instance-initiatied communication: Enter the license smart sync command in privileged EXEC mode. If CSLU is currently logged into CSSM the reports will be automatically sent to the associated Smart Account and Virtual Account in CSSM.

    • For SSM On-Prem-initiated communication, complete this task: In the SSM On-Prem UI, navigate to Reports > Synchronisation pull schedule with the devices > Synchronise now with the device.

    Synchronize usage information with CSSM (choose one)


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Error Message %SMART_LIC-6-TRUST_CODE_INSTALL_SUCCESS: A new licensing trust code
 was successfully installed on [chars].

Explanation:[chars] is the UDI where the trust code was successfully installed.

Recommended Action: No action is required. If you want to verify that the trust code is installed, enter the show license status command in privileged EXEC mode. Look for the updated timestamp under header Trust Code Installed: in the output.


----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------