Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Terminology

ASIC Version Information (UADP 2.0 versus 3.0)

General Hardware Validation Commands

Cisco IOS XE 17.x General Hardware Validation Commands

Cisco IOS XE 16.x General Hardware Validation Commands

Per-Feature Hardware Validation Commands

Scenario: IPv4 Prefixes

IPv4 Syslogs

Scenario: ACL

ACL Syslogs

Scenario: NAT

NAT Syslogs

Scenario: MPLS

MPLS Syslogs

Scenario: QoS

QoS Syslogs

Related Information

Cisco Bug IDs

Introduction

This document describes how to understand and troubleshoot hardware resources on Catalyst 9000 series switches.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

- Cisco Catalyst 9200, 9300, 9400, 9500 non-HP series switches on Cisco IOS® XE 16.x & 17.x software

- Cisco Catalyst 9500HP, 9600 series switches on Cisco IOS® XE 16.x & 17.x software

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

- Various features on Catalyst 9000 Series Switches consume limited hardware resources. These resources exist to accelerate the performance of those features, and to deliver the expected high forward rates expected from a switch.

- The size of these tables can vary from switch model to switch model, but the basic troubleshooting methodology remains the same.

- Commonly, the primary limited hardware resource in LAN switching is referred to as TCAM  (TCAM is a memory technology especially suited to store Longest Prefix Match (LPM) information for fast lookup or other types of OR logic lookups).

- In Catalyst 9000 Series switches, multiple memory types are used beyond just TCAM, suited to specific needs of a given feature (HASH is another type of simplified memory. The MAC address table is an example of this memory type).

When you troubleshoot a feature that does not operate as expected, a good starting point is to confirm that hardware is not beyond the scale of the switch in question. While switches can vary in the size of these tables, the verification and troubleshoot methodology remains mostly the same.

Note: This page is also a reference page where you can find information on various features, and how to check their hardware scale.

Note: Per-platform, the CLI sometimes includes the term switch and sometimes does not. (show platform hardware fed <number|active|standby> fwd-asic resource tcam utilization versus show platform hardware fed <active> fwd-asic resource tcam utilization

Terminology

EM

Exact Match

An entry in Hash memory that is a 1:1 match (host route, Directly Connected host)

LPM

Longest Prefix Match

Any route that is /31 or shorter (/32 routes are EM type)

TCAM

Ternary Content-Addressable Memory

A type of memory that stores and queries entries with three different inputs: 0, 1 and X. This type of memory must be used in cases where there can be multiple matches to the same entry, and the resulting Hash for each would not be unique. This table includes a mask or X value that allows it to know if it matches or does not match this entry.

CAM

Content-Addressable Memory

General term for hardware memory (Hash/TCAM)

RIB

Routing Information Base

The routing table seen in show ip route.

FIB

Forwarding Information Base

Simplified table with prefixes added by the RIB and ARP tables with a pointer to the ADJ table.

Directly Connected

Directly Connected Route

A locally connected host prefix (ARP adjacent).

Indirectly Connected

Indirectly Connected Route

A route that is via a remote next hop to reach.

ADJ

Adjacency (table)

Stores next hop information used for packet rewrite.

EM

Exact Match

Connected hosts, indirect /32 host prefixes

TCAM

Ternary Content-Addressable Memory

Indirect prefixes /31 or shorter

FED

Forward Engine Driver

The ASIC (hardware) layer

FMAN-FP

Forward Manager- Forwarding Plane

FMAN-FP manages software objects that add, delete, or modify FED information.

SI

Station Index

Station Index =  packet rewrite information (RI = Rewrite Index) and outbound interface information (DI = Destination Index)

RI

Rewrite Index

MAC address rewrite information for layer 3 forwarding to the next hop adjacency.

DI

Destination Index

Index that points to the outbound interface.

UADP

Cisco Unified Access™ Data Plane

The ASIC architecture used in the switch.

ASIC Version Information (UADP 2.0 versus 3.0)

The key difference between the 2.0 and 3.0 versions of the Catalyst 9000 series ASICs is how FIB hardware is populated or used.

In UADP 3.0 memory called EM/LPM is used:

- host routes (/32 mask length) and directly connected (ARP adjacent)

- /31 or shorter prefixes (Where a mask comparison is required to make a forward decision.)

In UADP 3.0 TCAM, it still exists for FIB, but is used only for special cases or exceptions where EM/LPM cannot be used.

- An example of this would be if the IP address space is not contiguous or multiple address spaces used, and merge into EM/LPM is not possible.

In UADP 2.0 memory is split into two sections EM & TCAM:

- EM is used for /32 host routes and directly connected (ARP adjacent) hosts.

- TCAM is used for /31 or shorter prefixes where a prefix mask comparison is required.

Compare these outputs between the two ASIC types:

In this example, the 9500-12Q has significantly more TCAM space. But, the 9500-48Y4C (9500H) has an even greater scale of EM/LPM.

- LPM - the same logic applies to the TCAM of the 9500-12Q, but, its not specifically called out.

- The EM/LPM on 9500H indicates that this shared memory space is used for both Exact Match (EM) and LPM (prefix-based) entries. The system uses an optimized memory system to achieve both scale, performance, and flexibility.

- The significantly reduced TCAM on 9500H exists to store special entries, particularly Hash Collisions (when a unique hash cannot be generated for a particular entry).

9500-48Y4C (9500H / High Performance - UADP 3.0 based switch)

Switch#show platform hardware fed active fwd-asic resource tcam utilization

Codes: EM - Exact_Match, I - Input, O - Output, IO - Input & Output, NA - Not Applicable

CAM Utilization for ASIC [0]

Table                  Subtype      Dir       Max     Used    %Used       V4       V6    MPLS    Other

------------------------------------------------------------------------------------------------------

IP Route Table         EM/LPM       I      212992        3    0.01%        2        0       1        0  <-- LPM matches now stored here

IP Route Table         TCAM         I        1536       15    0.02%        6        6       2        1  <-- Used for exception cases

9500-12Q (UADP 2.0 based switch)

Switch#show platform hardware fed active fwd-asic resource tcam utilization

Codes: EM - Exact_Match, I - Input, O - Output, IO - Input & Output, NA - Not Applicable

CAM Utilization for ASIC [0]

Table                  Subtype      Dir       Max     Used    %Used       V4       V6    MPLS    Other

------------------------------------------------------------------------------------------------------

IP Route Table         EM           I        49152        3    0.01%        2        0       1        0

IP Route Table         TCAM         I        65536       15    0.02%        6        6       2        1  <-- LPM matches are stored here in 2.0

Note: For more information on the UADP Architecture see Cisco Catalyst 9500 Architecture White Paper

General Hardware Validation Commands

These commands show high level usage statistics for Hash, TCAM, Interface, Rewrite resources used.

- These resources are related, and exhaustion of a dependent resource can affect the ability to fully use other available resources.

- Changes to the outputs from these commands in 17.x train makes the ability to read hardware, and diagnose specific issues much easier.

Example: A switch can have available Hash / TCAM, but run out of Adjacencys.

- Packet ability to forward can be impacted to some destination prefix not because hardware cannot program FIB but because it cannot program a new rewrite entry.

show platform hardware fed <switch> active fwd-asic resource tcam utilization    <-- Hash & TCAM

show platform hardware fed <switch> active fwd-asic resource utilization         <-- SI/RI/DI/etc (other related resources)

show platform hardware fed <switch> active fwd-asic resource rewrite utilization <-- IP Adjacency. LISP adjacency, Tunnel Adjacency, etc

### 17.x train CLI displays multiple resources in one place (these are not available in 16.x) ###

New CLI combines aspects of all 3 commands into one table for easier diagnosis of all resources related to IPv4

show platform hardware fed active fwd-asic resource features ip-adjacency utilization

Cisco IOS XE 17.x General Hardware Validation Commands

show platform hardware fed active fwd-asic resource tcam utilization command is the first place you want to look to evaluate if you have a hardware scale issue. (It displays information on a per-ASIC basis).

Codes:

- EM - Exact_Match                                     <-- Consult Terminology table for definition

- I - Input, O - Output, IO - Input & Output,  <-- If resource is directional it is noted

- NA - Not Applicable                                  <-- If direction is not applicable

Switch#show platform hardware fed active fwd-asic resource tcam utilization

Codes: EM - Exact_Match, I - Input, O - Output, IO - Input & Output, NA - Not Applicable                 <-- Key for table abbreviations

CAM Utilization for ASIC [0]                                                                             <-- Content Addressable Memory for ASIC 0

Table                  Subtype      Dir       Max     Used    %Used       V4       V6    MPLS    Other   <-- CAM usage broken down per resource & memory type (EM versus TCAM)

------------------------------------------------------------------------------------------------------

Mac Address Table      EM           I        65536       18    0.03%        0        0       0       18

Mac Address Table      TCAM         I         1024       21    2.05%        0        0       0       21

L3 Multicast           EM           I        16384        0    0.00%        0        0       0        0

L3 Multicast           TCAM         I         1024        9    0.88%        3        6       0        0

L2 Multicast           EM           I        16384        0    0.00%        0        0       0        0

L2 Multicast           TCAM         I         1024       11    1.07%        3        8       0        0

IP Route Table         EM           I        49152        3    0.01%        2        0       1        0  <-- Data from RIB/FIB populated here

IP Route Table         TCAM         I        65536       15    0.02%        6        6       2        1  <-- Data from RIB/FIB populated here

QOS ACL                TCAM         IO       18432       85    0.46%       28       38       0       19

Security ACL           TCAM         IO       18432      129    0.70%       26       58       0       45

Netflow ACL            TCAM         I         1024        6    0.59%        2        2       0        2

PBR ACL                TCAM         I         2048       22    1.07%       16        6       0        0  <-- Data for PBR & NAT populated here

Netflow ACL            TCAM         O         2048        6    0.29%        2        2       0        2

Flow SPAN ACL          TCAM         IO        1024       13    1.27%        3        6       0        4

Control Plane          TCAM         I          512      276   53.91%      126      106       0       44

Tunnel Termination     TCAM         I         1024       18    1.76%        8       10       0        0

Lisp Inst Mapping      TCAM         I         2048        1    0.05%        0        0       0        1

Security Association   TCAM         I          512        4    0.78%        2        2       0        0

CTS Cell Matrix/VPN

Label                  EM           O         8192        0    0.00%        0        0       0        0  <-- Outbound resource used to reach remote VPNv4 prefixes

CTS Cell Matrix/VPN

Label                  TCAM         O          512        1    0.20%        0        0       0        1

Client Table           EM           I         4096        0    0.00%        0        0       0        0

Client Table           TCAM         I          256        0    0.00%        0        0       0        0

Input Group            LE TCAM      I         1024        0    0.00%        0        0       0        0

Output Group           LE TCAM      O         1024        0    0.00%        0        0       0        0

Macsec SPD             TCAM         I         1024        2    0.20%        0        0       0        2

CAM Utilization for ASIC [1]

<...snip...>

If hardware scale from command show platform hardware fed active fwd-asic resource tcam utilization looks okay, check other dependent resources

Note: There are many shared resources. This is just a few that are commonly used. (This table appearance does not change between 16.x & 17.x)

Switch#show platform hardware fed active fwd-asic resource utilization

Resource Info for ASIC Instance: 0

Resource Name           Allocated     Free  <-- Number available. If this is at max (or very close) possible issues can occur

------------------------------------------

RSC_DI                      61       41805  <-- DI = Destination Index

RSC_RI                       3       57317  <-- RI = Rewrite Index

RSC_RI_REP                  10       49143  <-- RI_REP = Multicast Rewrite/Replication Index

RSC_SI                     519       64849  <-- SI = Station Index

<...snip...>

Switch#show platform hardware fed switch active fwd-asic resource rewrite utilization

Resource Info for ASIC Instance: 0

Rewrite Data                        Allocated     Free    <-- Rewrite specific hardware resources

-------------------------------------------------------

PHF_EGRESS_destMacAddress                 0       32000   <-- Destination MAC (Layer 3 next hop MAC rewrite)

IPV4_TUNNEL_SRC_IP_ADDR                   0          16   <-- IPv4 Tunnel Source IP

IPV4_TUNNEL_DEST_IP_ADDR                  0         256   <-- IPv4 Tunnel Destination IP

IPV4_GRE_TUNNEL_DEST_IP_ADDR              0        1024   <-- GRE specific tunnel Destination IP

GRE_HEADER                                0         684

GRE_KEY                                   0         684   <-- GRE keys

NAT_L3_DEST_IPV4                          0        7168   <-- NAT Layer 3 IPv4 Destination

NAT_DST_PORT_UNICAST                      0        8192   <-- NAT Destination Ports

NAT_L3_SRC_IPV4                           0        8192   <-- NAT Layer 3 IPv4 Source

NAT_SRC_PORT_UNICAST                      0        8192   <-- NAT Source Ports

<...snip...>

Switch#show platform hardware fed active fwd-asic resource features ip-adjacency utilization

IPv4 unicast adjacency resource info

Resource Info for ASIC Instance: 0 [A:0, C:0]                                          <-- Per-ASIC & Core [Asic 0, Core 0]

Shared Resource Name                     Allocated        Free              Usage%     <-- Shared resources

----------------------------------------------------------------------------------

RSC_RI                                   3                57317               0.01     <-- RI = Rewrite Index

RSC_SI                                   519              64849               0.79     <-- SI = Station Index

<-- These are tables that maintain port map info, and other necessary details to send packets

<-- These resources are shared, and used by many features

Rewrite Data                             Allocated        Free              Usage%     <-- Rewrite resources (Dest MAC)

-----------------------------------------------------------------------------------

PHF_EGRESS_destMacAddress                0                32000               0.00     <-- Destination MAC usage

<-- When a packet is sent to a next hop, it must be written with a destination MAC address

CAM Table Utilization Info               Allocated        Free              Usage%     <-- EM (Hash) & TCAM resources

------------------------------------------------------------------------------------

IP Route table Host/Network                   0/ 0            0/32768    0.00/ 0.00

<-- Resource that programs prefixes, either local/host routes (EM/Hash) or Shorter /31 or less prefixes (TCAM)

Note: 9500H & 9600 ASIC have the ability to store shorter prefix mask in Hash memory (called EM/LPM) versus TCAM. See the IPv4 specific scenario for more details

Cisco IOS XE 16.x General Hardware Validation Commands

show platform hardware fed active fwd-asic resource tcam utilization command is the first place you want to look to evaluate if you have a hardware scale issue. (It displays information on a per-ASIC basis). You can see that in 16.x train the output is less granular, and some of the descriptions vary.

In most cases, the Table list is clear with a couple exceptions:

- Directly or indirectly connected routes. This needed improvement, as it was not clear that directly means both ARP adjacent routes AND /32 host routes. Indirectly means any route /31 or shorter

- Policy Based Routing ACEs include NAT related configuration. Keep this in mind when NAT is the feature of concern.

Switch#show platform hardware fed switch active fwd-asic resource tcam utilization

CAM Utilization for ASIC  [0]

Table                                              Max Values        Used Values

--------------------------------------------------------------------------------

Unicast MAC addresses                               32768/1024        19/21

L3 Multicast entries                                 8192/512          0/9

L2 Multicast entries                                 8192/512          0/11

Directly or indirectly connected routes             24576/8192         3/19 <-- First value 24576 = EM / Second value 8192 = TCAM

QoS Access Control Entries                           5120             85

Security Access Control Entries                      5120            126

Ingress Netflow ACEs                                  256              8

Policy Based Routing ACEs                            1024             22

Egress Netflow ACEs                                   768              8

Flow SPAN ACEs                                       1024             13

Control Plane Entries                                 512            255

Tunnels                                               512             17

Lisp Instance Mapping Entries                        2048              3

Input Security Associations                           256              4

SGT_DGT                                              8192/512          0/1

CLIENT_LE                                            4096/256          0/0

INPUT_GROUP_LE                                       1024              0

OUTPUT_GROUP_LE                                      1024              0

Macsec SPD                                            256              2

Note:The commands listed here did not have a CLI change between 16 and 17 code trains, and they are only described once in the 17.x section of this document.

show platform hardware fed <switch> active fwd-asic resource utilization <-- SI/RI/DI/etc (other related resources)

show platform hardware fed <switch> active fwd-asic resource rewrite utilization <-- IP Adjacency. LISP adjacency, Tunnel Adjacency, etc

Per-Feature Hardware Validation Commands

Scenario: IPv4 Prefixes

IPv4 hardware validation can be found on this page Understand IPv4 Hardware Resources on Catalyst 9000 Switches

Symptom the resource is beyond scale.

1. Device or prefix reachability issues. While routes that exist or devices can remain reachable, any new or updated prefixes are not reachable.

2. Log messages indicate the hardware is not able to take new object updates.

3. Object layer, which programs software into hardware become congested.

4. Absent entries at the impacted hardware layer (in this case the FIB is the impacted layer).

IPv4 Syslogs

If you run out of a particular IPv4 FIB or Adjacency resource, SYSLOG message are generated by the system

IPv4 FIB Log Message

Definition

Recovery Action

%FED_L3_ERRMSG-3-RSRC_ERR: Switch 1 R0/0: fed: "Failed to allocate hardware resource for fib entry due to hardware resource exhaustion" error.

Hardware reserved for IPv4 FIB entries has run out of space (EM or TCAM).

Summarize routes or take some other action to reduce the scale of FIB entries (this can be exact match or TCAM, whichever one is exhausted).

%FED_L3_ERRMSG-3-RSRC_ERR: R0/0: fed: "Failed to allocate hardware resource for adj entry - rc:1"error.

The Adjacency table is exhausted. This is the table in hardware where next hop destination MAC addresses are stored.

Reduce the scale number of directly connected (ARP adjacent) hosts

Scenario: ACL

ACL hardware validation can be found on this page Validate Security ACLs on Catalyst 9000 Switches

ACL Syslogs

If you run out of a particular Security ACL resource, SYSLOG message is generated by the system (interface, Vlan, label, and so on values can differ).

ACL Log message

Definition

Recovery Action

%ACL_ERRMSG-4-UNLOADED: Switch 1 fed: Input <ACL> on interface <interface> could not be programmed in hardware and traffic can be dropped.

ACL is Unloaded (held in software).

Investigate TCAM scale. If beyond scale, redesign ACLs.

%ACL_ERRMSG-6-REMOVED: 1 fed:  The unloaded configuration for Input <ACL> on interface <interface> has been removed for label <label>asic<number>

Unloaded ACL configuration is removed from interface.

ACL is already removed, no action to take.

%ACL_ERRMSG-6-RELOADED: 1 fed:  Input <ACL> on interface <interface> has now been loaded into the hardware for label <label> on asic<number>

ACL is now installed in Hardware.

Issue with ACL is now in hardware resolved, no action to take.

%ACL_ERRMSG-3-ERROR: 1 fed:  Input <ACL> IP ACL <NAME> configuration could not be applied on <interface> at bindorder <number>

Other type ACL error (such as dot1x ACL install failure)

Confirm ACL configuration is supported, and TCAM is not beyond scale

%ACL_ERRMSG-6-GACL_INFO: Switch 1 R0/0: fed: Logging is not supported for GACL

GACL has log option configured.

GACL do not support log. Remove log statements from GACL.

%ACL_ERRMSG-6-PACL_INFO: Switch 1 R0/0: fed: Logging is not supported for PACL

PACL has log option configured.

PACL do not support log. Remove log statements from PACL.

%ACL_ERRMSG-3-ERROR: Switch 1 R0/0: fed: Input IPv4 Group ACL implicit_deny:<name>: configuration could not be applied on Client MAC 0000.0000.0000

(dot1x) ACL fails to apply on target port.

Confirm ACL configuration is supported, and TCAM is not beyond scale.

Scenario: NAT

NAT hardware validation can be found on this page Configure and Verify NAT on Catalyst 9000 Switches

NAT Syslogs

NAT Feature does not have any syslog that prints when hardware resources are out of scale. Cisco bug ID CSCvz46804 was filed as an enhancement to add these logs.

If you experience NAT issues and want to verify hardware resource usage, check show platform hardware fed switch active fwd-asic resource tcam utilization (the PBR ACL region is highly utilized when NAT TCAM is depleted).

Also verify you have configured NAT in accordance with the limitations noted here: Limitations of NAT

Scenario: MPLS

MPLS hardware validation can be found on this page Configure and Verify MPLS on Catalyst 9000 Switches

MPLS Syslogs

If you run out of a particular resource, such as MPLS labels, SYSLOG message are generated by the system.

Key points to remember:

- MPLS LABEL is used for label disposition. (This resource is consumed when prefixes are learned from a local CE)

- LSPA is used for label imposition. (This resource is consumed when prefixes are learned from a remote PE)

MPLS Log Message

Definition

Recovery Action

%FED_L3_ERRMSG-3-RSRC_ERR: Switch 1 R0/0: fed: "Failed to allocate hardware resource for fib entry" error due to hardware resource exhaustion.

Hardware reserved for IP prefixes has run out of space (EM or TCAM).

Take one of these actions to reduce the number of prefixes learned by the local or remote PE:

1. Summarize prefixes at CE.

2. Change label allocation mode from per-prefix to per-vrf.

%FED_L3_ERRMSG-3-mpls_out_of_resource: Switch 1 R0/0: fed: "Out of resource for MPLS LABEL ENTRY" error. Failed to program local label:8205 (8192/8192) in hardware.

Local label Allocation: Hardware reserved for MPLS local labels has run out of space (EM or TCAM).

Take one of these actions to reduce number of labels used on local PE:

1. Summarize prefixes at local CE or local PE.

2. Change label allocation mode from per-prefix to per-vrf on the local PE.

%FED_L3_ERRMSG-3-MPLS_LENTRY_PAUSE: Switch 1 R0/0: fed: Critical limit reached for MPLS LABEL ENTRY resource. Lentry Create PAUSED.

Local label Allocation: Hardware reserved for MPLS local labels has run out of space (EM or TCAM).

Take one of these actions to reduce number of labels used on local PE:

1. Summarize prefixes at local CE or local PE.

2. Change label allocation mode from per-prefix to per-vrf on the local PE.

%FED_L3_ERRMSG-3-mpls_out_of_resource: Switch 1 R0/0: fed: "Out of resource for MPLS LSPA. Failed to program in hardware" error.

Remote label allocation:  Hardware reserved for LSPA remote labels has run out of space.

Take one of these actions to reduce number of labels used on remote PE:

1. Summarize prefixes at remote CE or remote PE.

2. Change label allocation mode from per-prefix to per-vrf on the remote PE.

Scenario: QoS

QoS hardware validation can be found on this page Understand QoS Hardware Resources on Catalyst 9000 Switches

QoS Syslogs

If you run out of QoS related resources, SYSLOG messages are generated by the system:

QoS related Syslog Message

Definition

Recovery Actions

%FED_QOS_ERRMSG-4-TCAM_OVERFLOW: Switch 1 R0/0: fed: "Failed to program"error TCAM for policy-map ingress_pmap2 on GigabitEthernet1/0/10.

Hardware (TCAM) reserved for QoS entires has run out of space.

1. Ensure you have a valid / supported configuration.

2. Review the remainder of this document to validate the current scale utilization of your switch and possible steps to reduce if it is overutilized.

%FED_QOS_ERRMSG-3-QUEUE_SCHEDULER_HW_ERROR: Switch 1 R0/0: fed: "Failed to configure queue scheduler"error for GigabitEthernet1/0/27.

Installation to hardware of QoS queue scheduler has failed.

1. Verify your configuration is supported

2. Review the QoS configuration guide for your specific platform and version of software.

For 9200LONLY: Review Cisco bug IDCSCvz54607and Cisco bug IDCSCvz76172

FED_QOS_ERRMSG-3-QUEUE_BUFFER_HW_ERROR: R0/0: fed: "Failed to configure default queue buffer" error.

Installation to hardware of QoS queue buffers has failed.

1. Verify your configuration is supported.

2. Review the QoS configuration guide for your specific platform and version of software.

3. Review Cisco bug IDCSCvs49401

Related Information

Technical Support & Documentation - Cisco Systems

Cisco Catalyst 9200 Series Switches Data Sheet

Cisco Catalyst 9300 Series Switches Data Sheet

Cisco Catalyst 9400 Series Switches Data Sheets

Cisco Catalyst 9500 Series Switches Data Sheets

Cisco Catalyst 9600 Series Switches Data Sheet

Cisco Catalyst 9500 Architecture White Paper

Cisco Bug IDs

Cisco bug ID CSCvg60292  (When maximum routes in TCAM are hit, no routes are able to be installed in Hash table.)

Cisco bug ID CSCvx57822  (Hardware Tables need 90% utilization watermark.)

Cisco bug ID CSCvs49401

Cisco bug ID CSCvz54607

Cisco bug ID CSCvz76172