Configuring the Embedded Event Manager

This chapter contains the following sections:

About Embedded Event Manager

The ability to detect and handle critical events in the Cisco NX-OS system is important for high availability. The Embedded Event Manager (EEM) provides a central, policy-driven framework to detect and handle events in the system by monitoring events that occur on your device and taking action to recover or troubleshoot these events, based on your configuration..

EEM consists of three major components:

Event statements

Events to monitor from another Cisco NX-OS component that may require some action, workaround, or notification.

Action statements

An action that EEM can take, such as sending an e-mail or disabling an interface, to recover from an event.

Policies

An event paired with one or more actions to troubleshoot or recover from the event.

Without EEM, each individual component is responsible for detecting and handling its own events. For example, if a port flaps frequently, the policy of "putting it into errDisable state" is built into ETHPM.

Embedded Event Manager Policies

An EEM policy consists of an event statement and one or more action statements. The event statement defines the event to look for as well as the filtering characteristics for the event. The action statement defines the action EEM takes when the event occurs.

For example, you can configure an EEM policy to identify when a card is removed from the device and log the details related to the card removal. By setting up an event statement that tells the system to look for all instances of card removal and an then with an action statement that tells the system to log the details.

You can configure EEM policies using the command line interface (CLI) or a VSH script.

EEM gives you a device-wide view of policy management. Once EEM policies are configured, the corresponding actions are triggered. All actions (system or user-configured) for triggered events are tracked and maintained by the system.

Preconfigured System Policies

Cisco NX-OS has a number of preconfigured system policies. These system policies define many common events and actions for the device. System policy names begin with two underscore characters (__).

Some system policies can be overridden. In these cases, you can configure overrides for either the event or the action. The overrides that you configure take the place of the system policy.


Note

Override policies must include an event statement. Override policies without event statements override all possible events for the system policy.

To view the preconfigured system polices and determine which polices you can override, use the show event manager system-policy command.

User-Created Policies

User-created policies allow you to customize EEM policies for your network. If a user policy is created for an event, actions in the policy are triggered only after EEM triggers the system policy actions related to the same event.

Log Files

The log file that contains data that is related to EEM policy matches is maintained in the event_archive_1 log file located in the /log/event_archive_1 directory.

Event Statements

Any device activity for which some action, such as a workaround or notification, is taken is considered an event by EEM. In many cases, events are related to faults in the device, such as when an interface or a fan malfunctions.

Event statements specify which event or events triggers a policy to run.


Tip

You can configure EEM to trigger an EEM policy that is based on a combination of events by creating and differentiating multiple EEM events in the policy and then defining a combination of events to trigger a custom action.

EEM defines event filters so that only critical events or multiple occurrences of an event within a specified time period trigger an associated action.

Some commands or internal events trigger other commands internally. These commands are not visible, but will still match the event specification that triggers an action. You cannot prevent these commands from triggering an action, but you can check which event triggered an action.

Supported Events

EEM supports the following events in event statements:

  • Counter events

  • Fan absent events

  • Fan bad events

  • Memory thresholds events

  • Events being used in overridden system policies.

  • SNMP notification events

  • Syslog events

  • System manager events

  • Temperature events

  • Track events

Action Statements

Action statements describe the action that is triggered by a policy when an event occurs. Each policy can have multiple action statements. If no action is associated with a policy, EEM still observes events but takes no actions.

In order for triggered events to process default actions, you must configure the EEM policy to allow the default action. For example, if you match a CLI command in a match statement, you must add the event-default action statement to the EEM policy or EEM does not allow the command to execute.


Note

When configuring action statements within your user policy or overriding policy, it is important that you confirm that action statements do not negate each other or adversely affect the associated system policy.

Supported Actions

EEM supports the following actions in action statements:

  • Execute any CLI commands

  • Update a counter

  • Reload the device

  • Generate a syslog message

  • Generate an SNMP notification

  • Use the default action for the system policy

VSH Script Policies

You can write policies in a VSH script, by using a text editor. Policies that are written using a VSH script have an event statement and action statement(s) just as other policies, and these policies can either augment or override system policies.

After you define your VSH script policy, copy it to the device and activate it.

Licensing Requirements for Embedded Event Manager

This feature does not require a license. Any feature not included in a license package is bundled with the Cisco NX-OS system images and is provided at no extra charge to you. For a complete explanation of the Cisco NX-OS licensing scheme, see the Cisco NX-OS Licensing Guide.

Prerequisites for Embedded Event Manager

You must have network-admin privileges to configure EEM.

Guidelines and Limitations for Embedded Event Manager

When you plan your EEM configuration, consider the following:

  • The maximum number of configurable EEM policies is 500.

  • Action statements within your user policy or overriding policy should not negate each other or adversely affect the associated system policy.

  • To allow a triggered event to process any default actions, you must configure the EEM policy to allow the default action. For example, if you match a command in a match statement, you must add the event-default action statement to the EEM policy or EEM does not allow the command to execute.

  • The following guidelines apply to Event Log Auto-Collection and Backup:

    • By default, enabled log collection on a switch provides between 15 minutes to several hours of event logs depending on size, scale and component activity.

    • To be able to collect relevant logs that span a longer period, only enable event log retention for the specific services/features you need. See "Enabling Extended Log File Retention For a Single Service". You can also export the internal event logs. See "External Log File Storage".

    • When troubleshooting, it is good practice to manually collect a snapshot of internal event logs in real time. See "Generating a Local Copy of Recent Log Files".

  • An override policy that consists of an event statement and no action statement triggers no action and no notification of failures.

  • An override policy without an event statement overrides all possible events in the system policy.

  • In regular command expressions: all keywords must be expanded, and only the asterisk (*) symbol can be used for replace the arguments.

  • EEM event correlation supports up to four event statements in a single policy. The event types can be the same or different, but only these event types are supported: cli, counter, snmp, syslog, and track.

  • When more than one event statement is included in an EEM policy, each event statement must have a tag keyword with a unique tag argument.

  • EEM event correlation does not override the system default policies.

  • Default action execution is not supported for policies that are configured with tagged events.

  • If your event specification matches a CLI pattern, you can use SSH-style wild card characters.

    For example, if you want to match all show commands, enter the show * command. Entering the show . * command does not work.

  • If your event specification is a regular expression for a matching syslog message, you can use a proper regular expression.

    For example, if you want to detect ADMIN_DOWN events on any port where a syslog is generated, use .ADMIN_DOWN. . Entering the ADMIN_DOWN command does not work.

  • In the event specification for a syslog, the regex does not match any syslog message that is generated as an action of an EEM policy.

  • If an EEM event matches a show command in the CLI and you want the output for that show command to display on the screen (and to not be blocked by the EEM policy), you must specify the event-default command for the first action for the EEM policy.

Default Settings for Embedded Event Manager

Table 1. Default EEM Parameters
Parameters Default
System Policies Active

Configuring Embedded Event Manager

Defining an Environment Variable

Defining an environment variable is an optional step but is useful for configuring common values for repeated use in multiple policies.

SUMMARY STEPS

  1. configure terminal
  2. event manager environment variable-name variable-value
  3. (Optional) show event manager environment {variable-name | all}
  4. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

event manager environment variable-name variable-value

Example:

switch(config) # event manager
environment emailto "admin@anyplace.com"

Creates an environment variable for EEM.

The variable-name can be any case-sensitive, alphanumeric string up to 29 characters.

The variable-value can be any quoted case-sensitive, alphanumeric string up to 39 characters.

Step 3

(Optional) show event manager environment {variable-name | all}

Example:

switch(config) # show event manager
environment all
 (Optional)

Displays information about the configured environment variables.

Step 4

(Optional) copy running-config startup-config

Example:

switch(config)# copy running-config startup-config
(Optional)

Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

Defining a User Policy Using the CLI

SUMMARY STEPS

  1. configure terminal
  2. event manager applet applet-name
  3. (Optional) description policy-description
  4. event event-statement
  5. (Optional) tag tag {and | andnot | or} tag [and | andnot | or {tag}] {happens occurs in seconds}
  6. action number[.number2] action-statement
  7. (Optional) show event manager policy-state name [module module-id]
  8. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

event manager applet applet-name

Example:

switch(config)# event manager applet monitorShutdown
switch(config-applet)#

Registers the applet with EEM and enters applet configuration mode.

The applet-name can be any case-sensitive, alphanumeric string up to 29 characters.

Step 3

(Optional) description policy-description

Example:

switch(config-applet)# description "Monitors interface shutdown."
 (Optional)

Configures a descriptive string for the policy.

The string can be any alphanumeric string up to 80 characters. Enclose the string in quotation marks.

Step 4

event event-statement

Example:

switch(config-applet)# event cli match "shutdown"

Configures the event statement for the policy.

Step 5

(Optional) tag tag {and | andnot | or} tag [and | andnot | or {tag}] {happens occurs in seconds}

Example:

switch(config-applet)# tag one or two happens 1 in 10000
 (Optional)

Correlates multiple events in the policy.

The range for the occurs argument is from 1 to 4294967295.

The range for the seconds argument is from 0 to 4294967295 seconds.

Step 6

action number[.number2] action-statement

Example:

switch(config-applet)# action 1.0 cli show interface e 3/1

Configures an action statement for the policy. Repeat this step for multiple action statements.

Step 7

(Optional) show event manager policy-state name [module module-id]

Example:

switch(config-applet)# show event manager policy-state monitorShutdown
 (Optional)

Displays information about the status of the configured policy.

Step 8

(Optional) copy running-config startup-config

Example:

switch(config)# copy running-config startup-config
(Optional)

Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

Configuring Event Statements

Use one of the following commands in EEM configuration mode (config-applet) to configure an event statement:

Before you begin

Define a user policy.

SUMMARY STEPS

  1. event cli [tag tag] match expression [count repeats | time seconds
  2. event counter [tag tag] name counter entry-val entry entry-op {eq | ge | gt | le | lt | ne} {exit-val exit exit-op {eq | ge | gt | le | lt | ne}
  3. event fanabsent [fan number] time seconds
  4. event fanbad [fan number] time seconds
  5. event memory {critical | minor | severe}
  6. event policy-default count repeats [time seconds]
  7. event snmp [tag tag] oid oid get-type {exact | next} entry-op {eq | ge | gt | le | lt | ne} entry-val entry [exit-comb {and | or}]exit-op {eq | ge | gt | le | lt | ne} exit-val exit exit-time time polling-interval interval
  8. event sysmgr memory [module module-num] major major-percent minor minor-percent clear clear-percent
  9. event temperature [module slot] [sensor number] threshold {any | down | up}
  10. event track [tag tag] object-number state {any | down | up

DETAILED STEPS

  Command or Action Purpose

Step 1

event cli [tag tag] match expression [count repeats | time seconds

Example:

switch(config-applet) # event cli match "shutdown"

Triggers an event if you enter a command that matches the regular expression.

The tag tag keyword-argument pair identifies this specific event when multiple events are included in the policy.

The repeats range is from 1 to 65000.

The time range is from 0 to 4294967295, where 0 indicates no time limit.

Step 2

event counter [tag tag] name counter entry-val entry entry-op {eq | ge | gt | le | lt | ne} {exit-val exit exit-op {eq | ge | gt | le | lt | ne}

Example:

switch(config-applet) # event counter name mycounter entry-val 20 gt

Triggers an event if the counter crosses the entry threshold based on the entry operation. The event resets immediately. Optionally, you can configure the event to reset after the counter passes the exit threshold.

The tag tag keyword-argument pair identifies this specific event when multiple events are included in the policy.

The counter name can be any case-sensitive, alphanumeric string up to 28 characters.

The entry and exit value ranges are from 0 to 2147483647.

Step 3

event fanabsent [fan number] time seconds

Example:

switch(config-applet) # event fanabsent time 300

Triggers an event if a fan is removed from the device for more than the configured time, in seconds.

The number range is is from 1 to 1 and is module-dependent.

The seconds range is from 10 to 64000.

Step 4

event fanbad [fan number] time seconds

Example:

switch(config-applet) # event fanbad time 3000

Triggers an event if a fan fails for more than the configured time, in seconds.

The number range is module-dependent.

The seconds range is from 10 to 64000.

Step 5

event memory {critical | minor | severe}

Example:

switch(config-applet) # event memory critical

Triggers an event if a memory threshold is crossed.

Step 6

event policy-default count repeats [time seconds]

Example:

switch(config-applet) # event policy-default count 3

Uses the event configured in the system policy. Use this option for overriding policies.

The repeats range is from 1 to 65000.

The seconds range is from 0 to 4294967295, where 0 indicates no time limit.

Step 7

event snmp [tag tag] oid oid get-type {exact | next} entry-op {eq | ge | gt | le | lt | ne} entry-val entry [exit-comb {and | or}]exit-op {eq | ge | gt | le | lt | ne} exit-val exit exit-time time polling-interval interval

Example:

switch(config-applet) # event snmp oid 
1.3.6.1.2.1.31.1.1.1.6 get-type next 
entry-op lt 300 entry-val 0 exit-op eq 400 
exit-time 30 polling-interval 300

Triggers an event if the SNMP OID crosses the entry threshold based on the entry operation. The event resets immediately, or optionally you can configure the event to reset after the counter passes the exit threshold. The OID is in dotted decimal notation.

The tag tag keyword-argument pair identifies this specific event when multiple events are included in the policy.

The entry and exit value ranges are from 0 to 18446744073709551615.

The time, in seconds, is from 0 to 2147483647.

The interval, in seconds, is from 0 to 2147483647.

Step 8

event sysmgr memory [module module-num] major major-percent minor minor-percent clear clear-percent

Example:

switch(config-applet) # event sysmgr memory minor 80

Triggers an event if the specified system manager memory threshold is exceeded.

The percent range is from 1 to 99.

Step 9

event temperature [module slot] [sensor number] threshold {any | down | up}

Example:

switch(config-applet) # event temperature module 2 threshold any

Triggers an event if the temperature sensor exceeds the configured threshold.

The sensor range is from 1 to 18.

Step 10

event track [tag tag] object-number state {any | down | up

Example:

switch(config-applet) # event track 1 state down

Triggers an event if the tracked object is in the configured state.

The tag tag keyword-argument pair identifies this specific event when multiple events are included in the policy.

The object-number range is from 1 to 500.

What to do next

Configure action statements.

If you have already configured action statements or choose not to, complete any of the optional tasks:

  • Define a policy using a VSH script. Then, register and activate a VSH script policy.

  • Configure memory thresholds

  • Configure the syslog as an EEM publisher.

  • Verify your EEM configuration.

Configuring Action Statements

You can configure an action by using one of the following commands in EEM configuration mode (config-applet):


Note

If you want to allow a triggered event to process any default actions, you must configure the EEM policy to allow the default action.

For example, if you match a command in a match statement, you must add the event-default action statement to the EEM policy or EEM does not allow the command to execute. You can use the terminal event-manager bypass command to allow all EEM policies with matches to execute the command.

Before you begin

Define a user policy.

SUMMARY STEPS

  1. action number[.number2] cli command1[command2.] [local]
  2. action number[.number2] counter name counter value val op {dec | inc | nop | set}
  3. action number[.number2] event-default
  4. action number[.number2] policy-default
  5. action number[.number2] reload [module slot [- slot]]
  6. action number[.number2] snmp-trap [intdata1 integer-data1] [intdata2 integer-data2] [strdata string-data]
  7. action number[.number2] syslog [priority prio-val] msg error-message

DETAILED STEPS

  Command or Action Purpose

Step 1

action number[.number2] cli command1[command2.] [local]

Example:

switch(config-applet) # action 1.0 cli "show interface e 3/1"

Runs the configured commands. You can optionally run the commands on the module where the event occurred.

The action label is in the format number1.number2.

The number can be any number from 1 to 16 digits.

The range for number2 is from 0 to 9.

Step 2

action number[.number2] counter name counter value val op {dec | inc | nop | set}

Example:

switch(config-applet) # action 2.0 counter name mycounter value 20 op inc

Modifies the counter by the configured value and operation.

The action label is in the format number1.number2.

The number can be any number from 1 to 16 digits.

The range for number2 is from 0 to 9.

The counter can be any case-sensitive, alphanumeric string up to 28 characters.

The val can be an integer from 0 to 2147483647 or a substituted parameter.

Step 3

action number[.number2] event-default

Example:

switch(config-applet) # action 1.0 event-default

Completes the default action for the associated event.

The action label is in the format number1.number2.

The number can be any number from 1 to 16 digits.

The range for number2 is from 0 to 9.

Step 4

action number[.number2] policy-default

Example:

switch(config-applet) # action 1.0 policy-default

Completes the default action for the policy that you are overriding.

The action label is in the format number1.number2.

The number can be any number from 1 to 16 digits.

The range for number2 is from 0 to 9.

Step 5

action number[.number2] reload [module slot [- slot]]

Example:

switch(config-applet) # action 1.0 reload module 3-5

Forces one or more modules to the entire system to reload.

The action label is in the format number1.number2.

The number can be any number from 1 to 16 digits.

The range for number2 is from 0 to 9.

Step 6

action number[.number2] snmp-trap [intdata1 integer-data1] [intdata2 integer-data2] [strdata string-data]

Example:

switch(config-applet) # action 1.0 snmp-trap strdata "temperature problem"

Sends an SNMP trap with the configured data. The action label is in the format number1.number2.

The number can be any number from 1 to 16 digits.

The range for number2 is from 0 to 9.

The data elements can be any number up to 80 digits.

The string can be any alphanumeric string up to 80 characters.

Step 7

action number[.number2] syslog [priority prio-val] msg error-message

Example:

switch(config-applet) # action 1.0 syslog priority notifications msg "cpu high"

Sends a customized syslog message at the configured priority.

The action label is in the format number1.number2.

The number can be any number from 1 to 16 digits.

The range for number2 is from 0 to 9.

The error-message can be any quoted alphanumeric string up to 80 characters.

What to do next

Configure event statements.

If you have already configured event statements or choose not to, complete any of the optional tasks:

  • Define a policy using a VSH script. Then, register and activate a VSH script policy.

  • Configure memory thresholds

  • Configure the syslog as an EEM publisher.

  • Verify your EEM configuration.

Defining a Policy Using a VSH Script

This is an optional task. Complete the following steps if you are using a VSH script to write EEM policies:

SUMMARY STEPS

  1. In a text editor, list the commands that define the policy.
  2. Name the text file and save it.
  3. Copy the file to the following system directory: bootflash://eem/user_script_policies

DETAILED STEPS

Step 1

In a text editor, list the commands that define the policy.

Step 2

Name the text file and save it.

Step 3

Copy the file to the following system directory: bootflash://eem/user_script_policies

What to do next

Register and activate a VSH script policy.

Registering and Activating a VSH Script Policy

This is an optional task. Complete the following steps if you are using a VSH script to write EEM policies.

Before you begin

Define a policy using a VSH script and copy the file to the system directory.

SUMMARY STEPS

  1. configure terminal
  2. event manager policy policy-script
  3. (Optional) event manager policy internal name
  4. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

event manager policy policy-script

Example:

switch(config)# event manager policy moduleScript

Registers and activates an EEM script policy.

The policy-script can be any case-sensitive, alphanumeric string up to 29 characters.

Step 3

(Optional) event manager policy internal name

Example:

switch(config)# event manager policy internal moduleScript
 (Optional)

Registers and activates an EEM script policy.

The policy-script can be any case-sensitive alphanumeric string up to 29 characters.

Step 4

(Optional) copy running-config startup-config

Example:

switch(config)# copy running-config startup-config
(Optional)

Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

What to do next

Complete any of the following, depending on your system requirements:

  • Configure memory thresholds.

  • Configure the syslog as an EEM publisher.

  • Verify your EEM configuration.

Overriding a System Policy

SUMMARY STEPS

  1. configure terminal
  2. (Optional) show event manager policy-state system-policy
  3. event manager applet applet-name override system-policy
  4. description policy-description
  5. event event-statement
  6. section number action-statement
  7. (Optional) show event manager policy-state name
  8. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

(Optional) show event manager policy-state system-policy

Example:

switch(config-applet)# show event
manager policy-state __ethpm_link_flap
Policy __ethpm_link_flap
   Cfg count : 5
   Cfg time interval : 10.000000
(seconds)
   Hash default, Count 0
 (Optional)

Displays information about the system policy that you want to override, including thresholds. Use the show event manager system-policy command to find the system policy names.

Step 3

event manager applet applet-name override system-policy

Example:

switch(config-applet)# event manager applet
ethport override __ethpm_link_flap
switch(config-applet)#

Overrides a system policy and enters applet configuration mode.

The applet-name can be any case-sensitive, alphanumeric string up to 80 characters.

The system-policy must be one of the system policies.

Step 4

description policy-description

Example:

switch(config-applet)# description
"Overrides link flap policy"

Configures a descriptive string for the policy.

The policy-description can be any case-sensitive, alphanumeric string up to 80 characters, but it must be enclosed in quotation marks.

Step 5

event event-statement

Example:

switch(config-applet)# event
policy-default count 2 time 1000

Configures the event statement for the policy.

Step 6

section number action-statement

Example:

switch(config-applet)# action 1.0 syslog
priority warnings msg "Link is
flapping."

Configures an action statement for the policy. For multiple action statements, repeat this step.

Step 7

(Optional) show event manager policy-state name

Example:

switch(config-applet)# show event
manager policy-state ethport
 (Optional)

Displays information about the configured policy.

Step 8

(Optional) copy running-config startup-config

Example:

switch(config)# copy running-config startup-config
(Optional)

Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

Configuring Syslog as an EEM Publisher

Configuring syslog as an EEM publisher allows you to monitor syslog messages from the switch.


Note

The maximum number of searchable strings to monitor syslog messages is 10.

Before you begin

  • Confirm that EEM is available for registration by the syslog.

  • Confirm that the syslog daemon is configured and executed.

SUMMARY STEPS

  1. configure terminal
  2. event manager applet applet-name
  3. event syslog [tag tag] {occurs number | period seconds | pattern msg-text | priority priority}
  4. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

event manager applet applet-name

Example:

switch(config)# event manager applet abc
switch (config-appliet)#

Registers an applet with EEM and enters applet configuration mode.

Step 3

event syslog [tag tag] {occurs number | period seconds | pattern msg-text | priority priority}

Example:

switch(config-applet)# event syslog occurs 10

Registers an applet with EEM and enters applet configuration mode.

Step 4

(Optional) copy running-config startup-config

Example:

switch(config)# copy running-config startup-config
(Optional)

Saves the change persistently through reboots and restarts by copying the running configuration to the startup configuration.

What to do next

Verify your EEM configuration.

Verifying the Embedded Event Manager Configuration

Use one of the following commands to verify the configuration:

Command Purpose

show event manager environment [variable-name | all]

Displays information about the event manager environment variables.

show event manager event-types [event | all | module slot]

Displays information about the event manager event types.

show event manager history events [detail] [maximum num-events] [severity {catastrophic | minor | moderate | severe}]

Displays the history of events for all policies.

show event manager policy-state policy-name

Displays information about the policy state, including thresholds.

show event manager script system [policy-name | all]

Displays information about the script policies.

show event manager system-policy [all]

Displays information about the predefined system policies.

show running-config eem

Displays information about the running configuration for EEM.

show startup-config eem

Displays information about the startup configuration for EEM.

Event Log Auto-Collection and Backup

Automatically collected event logs are stored locally on switch memory. Event log file storage is a temporary buffer that stores files for a fixed amount of time. Once the time period has elapsed, a roll-over of the buffer makes room for the next files. The roll-over uses a first-in-first-out method.

Beginning with Cisco NX-OS Release 9.3(3), EEM uses the following methods of collection and backup:

  • Extended Log File Retention

  • Trigger-Based Event Log Auto-Collection

Extended Log File Retention

Beginning with Cisco NX-OS release 9.3(3), all Cisco Nexus platform switches, with at least 8Gb of system memory, support the extended retention of event logging files. Storing the log files locally on the switch or remotely through an external container, reduces the loss of event logs due to rollover.

Enabling Extended Log File Retention For All Services

Extended Log File Retention is enabled by default for all services running on a switch. If the switch doesn't have the log file retention feature enabled (no bloggerd log-dump is configured), use the following procedure to enable it.

SUMMARY STEPS

  1. configure terminal
  2. bloggerd log-dump all

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:
switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

bloggerd log-dump all

Example:
switch(config)# bloggerd log-dump all
switch(config)#

Enables the log file retention feature for all services.
Example
switch# configure terminal
switch(config)# bloggerd log-dump all
Sending Enable Request to Bloggerd
Bloggerd Log Dump Successfully enabled
switch(config)#

Disabling Extended Log File Retention For All Services

Extended Log File Retention is disabled by default for all services on the switch. If the switch has the log file retention feature enabled for all services and you want to disable it, use the following procedure.

SUMMARY STEPS

  1. configure terminal
  2. no bloggerd log-dump all

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:
switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

no bloggerd log-dump all

Example:
switch(config)# no bloggerd log-dump all
switch(config)#

Disables the log file retention feature for all services on the switch.
Example
switch# configure terminal
switch(config)# no bloggerd log-dump all
Sending Disable Request to Bloggerd
Bloggerd Log Dump Successfully disabled
switch(config)#

Enabling Extended Log File Retention For a Single Service

Extended Log File Retention is enabled by default for all services running on a switch. If the switch doesn't have the log file retention feature enabled (no bloggerd log-dump is configured), use the following procedure to enable it for a single service.

SUMMARY STEPS

  1. show system internal sysmgr service name service-type
  2. configure terminal
  3. bloggerd log-dump sap number
  4. show system internal bloggerd info log-dump-info

DETAILED STEPS

  Command or Action Purpose

Step 1

show system internal sysmgr service name service-type

Example:
switch# show system internal sysmgr service name aclmgr

Displays information about the ACL Manager including the service SAP number.

Step 2

configure terminal

Example:
switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 3

bloggerd log-dump sap number

Example:
switch(config)# bloggerd log-dump sap 351

Enables the log file retention feature for the ACL Manager service.

Step 4

show system internal bloggerd info log-dump-info

Example:

switch(config)# show system internal bloggerd info log-dump-info

Displays information about the log file retention feature on the switch.
Example
switch# show system internal sysmgr service name aclmgr
Service "aclmgr" ("aclmgr", 80):
        UUID = 0x182, PID = 653, SAP = 351
        State: SRV_STATE_HANDSHAKED (entered at time Mon Nov  4 11:10:41 2019).
        Restart count: 1
        Time of last restart: Mon Nov  4 11:10:39 2019.
        The service never crashed since the last reboot.
        Tag = N/A
        Plugin ID: 0
switch(config)# configure terminal
switch(config)# bloggerd log-dump sap 351
Sending Enable Request to Bloggerd
Bloggerd Log Dump Successfully enabled
switch(config)# show system internal bloggerd info log-dump-info 
-------------------------------------------------------------
Log Dump config is READY
Log Dump is DISABLED for ALL application services in the switch
Exceptions to the above rule (if any) are as follows: 
-------------------------------------------------------------
Module     | VDC        | SAP                       | Enabled?  
-------------------------------------------------------------
1          | 1          | 351 (MTS_SAP_ACLMGR     ) | Enabled  
-------------------------------------------------------------
-------------------------------------------------------------
Log Dump Throttle Switch-Wide Config:
-------------------------------------
Log Dump Throttle                                      : ENABLED
Minimum buffer rollover count (before throttling)      : 5
Maximum allowed rollover count per minute              : 1
-------------------------------------------------------------

switch(config)#

Displaying Extended Log Files

Use this task to display the event log files currently stored on the switch.

SUMMARY STEPS

  1. dir debug:log-dump/

DETAILED STEPS

Command or Action Purpose

dir debug:log-dump/

Example:
switch# dir debug:log-dump/

Displays the event log files currently stored on the switch.
Example
switch# dir debug:log-dump/

3676160 Dec 05 02:43:01 2019 20191205023755_evtlog_archive.tar
3553280 Dec 05 06:05:06 2019 20191205060005_evtlog_archive.tar

Usage for debug://sup-local
913408 bytes used
4329472 bytes free
5242880 bytes total

Disabling Extended Log File Retention For a Single Service

Extended Log File Retention is enabled by default for all services on the switch. If the switch has the log file retention feature enabled for a single service or all services (by default in Cisco NX-OS Release 9.3(5)), and you want to disable a specific service or services, use the following procedure.

SUMMARY STEPS

  1. show system internal sysmgr service name service-type
  2. configure terminal
  3. no bloggerd log-dump sap number
  4. show system internal bloggerd info log-dump-info

DETAILED STEPS

  Command or Action Purpose

Step 1

show system internal sysmgr service name service-type

Example:
switch# show system internal sysmgr service name aclmgr

Displays information about the ACL Manager including the service SAP number.

Step 2

configure terminal

Example:
switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 3

no bloggerd log-dump sap number

Example:
switch(config)# no bloggerd log-dump sap 351

Disables the log file retention feature for the ACL Manager service.

Step 4

show system internal bloggerd info log-dump-info

Example:

switch(config)# show system internal bloggerd info log-dump-info

Displays information about the log file retention feature on the switch.
Example

The following example shows how to disable extended log file retention for a service named "aclmgr":

switch# show system internal sysmgr service name aclmgr
Service "aclmgr" ("aclmgr", 80):
        UUID = 0x182, PID = 653, SAP = 351
        State: SRV_STATE_HANDSHAKED (entered at time Mon Nov  4 11:10:41 2019).
        Restart count: 1
        Time of last restart: Mon Nov  4 11:10:39 2019.
        The service never crashed since the last reboot.
        Tag = N/A
        Plugin ID: 0
switch(config)# configure terminal
switch(config)# no bloggerd log-dump sap 351
Sending Disable Request to Bloggerd
Bloggerd Log Dump Successfully disabled
switch(config)# show system internal bloggerd info log-dump-info 
-------------------------------------------------------------
Log Dump config is READY
Log Dump is DISABLED for ALL application services in the switch
Exceptions to the above rule (if any) are as follows: 
-------------------------------------------------------------
Module     | VDC        | SAP                       | Enabled?  
-------------------------------------------------------------
1          | 1          | 351 (MTS_SAP_ACLMGR     ) | Disabled  
-------------------------------------------------------------
-------------------------------------------------------------
Log Dump Throttle Switch-Wide Config:
-------------------------------------
Log Dump Throttle                                      : ENABLED
Minimum buffer rollover count (before throttling)      : 5
Maximum allowed rollover count per minute              : 1
-------------------------------------------------------------

switch(config)#

Trigger-Based Event Log Auto-Collection

Trigger-based log collection capabilities:

  • Automatically collect relevant data when issues occur.

  • No impact on control plane

  • Customizable configuration:

    • Defaults populated by Cisco

    • Selectively override what-to-collect by network administrator or by Cisco TAC.

    • Automatically update new triggers on image upgrades.

  • Store logs locally on the switch or remotely on an external server.

  • Supports severity 0, 1, and 2 syslogs:

  • Custom syslogs for ad-hoc events (auto-collection commands attached to the syslogs)

Enabling Trigger-Based Log File Auto-Collection

To enable trigger-based automatic creation of log files, you must create an override policy for the __syslog_trigger_default system policy with a custom YAML file and define the specific logs for which information will be collected.

For more information on creating a custom YAML file to enable log file auto-collection, see Configuring the Auto-Collection YAML File.

Auto-Collection YAML File

The Auto-Collection YAML file that is specified in the action command in the EEM function, defines actions for different system or feature components. This file is located in the switch directory: /bootflash/scripts. In addition to the default YAML file, you can create component-specific YAML files and place them in the same directory. The naming convention for component-specific YAML files is component-name.yaml. If a component-specific file is present in the same directory, it takes precedence over the file that is specified in the action command. For example, if the action file, bootflash/scripts/platform.yaml is in the /bootflash/scripts directory with the default action file, bootflash/scripts/test.yaml, then the instructions defined in platform.yaml file take precedence over the instructions for the platform component present in the default test.yaml file.

Examples of components are, ARP, BGP, IS-IS, and so on. If you are not familiar with all the component names, contact Cisco Customer Support for assistance in defining the YAML file for component-specific actions (and for the default test.yaml file as well).

Example:

event manager applet test_1 override __syslog_trigger_default
  action 1.0 collect test.yaml $_syslog_msg
Configuring the Auto-Collection YAML File

A contents of a YAML file determines the data collected during trigger-based auto-collection. There must be only one YAML file on the switch but it can contain auto-collection meta-data for any number of switch components and messages.

Locate the YAML file in the following directory on the switch:

/bootflash/scripts

Invoke the YAML file for trigger-based collection by using the following example. The example shows the minimum required configuration for trigger-based collection to work with a user-defined YAML file. 

switch# show running-config eem
!Command: show running-config eem
!Running configuration last done at: Mon Sep 30 19:34:54 2019
!Time: Mon Sep 30 22:24:55 2019
version 9.3(3) Bios:version 07.59
event manager applet test_1 override __syslog_trigger_default
  action 1.0 collect test.yaml $_syslog_msg

In the preceding example, "test_1" is the name of the applet and "test.yaml" is the name of the user-configured YAML file present in the /bootflash/scripts directory.

Example YAML File

The following is an example of a basic YAML file supporting the trigger-based event log auto-collection feature. The definitions for the keys/values in the file are in the table that follows.


Note

Make sure that the YMAL file has proper indentation. As a best practice, run it through any "online YAML validator" before using it on a switch. 

bash-4.3$ cat /bootflash/scripts/test.yaml
version: 1
components:
   securityd:
        default:
            tech-sup: port
            commands: show module
   platform:
        default:
            tech-sup: port
            commands: show module

Key: Value

Description

version: 1

Set to 1. Any other number creates an incompatibility for the auto collect script.

components:

Keyword specifying that what follows are switch components.

securityd:

Name of the syslog component (securityd is a facility name in syslog).

default:

Identifies all messages belonging to the component.

tech-sup: port

Collect tech support of the port module for the securityd syslog component.

commands: show module

Collect show module command output for the securityd syslog component.

platform:

Name of the syslog component (platform is a facility name in syslog).

tech-sup: port

Collect tech support of the port module for the platform syslog component.

commands: show module

Collect show module command output for the platform syslog component.

Use the following example to associate auto-collect metadata only for a specific log. For example, SECURITYD-2-FEATURE_ENABLE_DISABLE

securityd:
                feature_enable_disable:
                        tech-sup: security
                        commands: show module

Key: Value

Description

securityd:

Name of the syslog component (securityd is a facility name in syslog).

feature_enable_disable:

Message ID of the syslog message.

tech-sup: security

Collect tech support of the security module for the securityd syslog component.

commands: show module

Collect show module command output for the security syslog component.

Example syslog output for the above YAML entry:

2019 Dec 4 12:41:01 n9k-c93108tc-fx %SECURITYD-2-FEATURE_ENABLE_DISABLE: User 
has enabled the feature bash-shell

Use the following example to specify multiple values.

version: 1
components:
   securityd:
        default:
            commands: show module;show version;show module
            tech-sup: port;lldp

Note

Use semicolons to separate multiple show commands and tech support key values (see the preceding example).

Beginning with Release 10.1(1), test.yaml can be replaced with a folder inside which more than one YAML files can be present. All the YAML files in the folder must follow the ComponentName.yaml naming convention.

In the following example, test.yaml is replaced with test_folder: 


test.yaml:
event manager applet logging2 override __syslog_trigger_default
   action 1.0 collect test.yaml rate-limt 30 $_syslog_msg

test_folder:
event manager applet logging2 override __syslog_trigger_default
   action 1.0 collect test_folder rate-limt 30 $_syslog_msg

The following example shows the path and component(s) for test_folder: 


ls /bootflash/scripts/test_folder
bgp.yaml ppm.yaml

Limiting the Amount of Auto-Collections Per Component

For auto-collection, the limit of the number of bundles per component event is set to three (3) by default. If more than three events occur for a component, then the events are dropped with the status message EVENTLOGLIMITREACHED. The auto-collection of the component event restarts when the event log has rolled over.

Example:

switch# show system internal event-logs auto-collect history 
DateTime              Snapshot ID  Syslog                   Status/Secs/Logsize(Bytes)
2020-Jun-27 07:20:03  1140276903   ACLMGR-0-TEST_SYSLOG     EVENTLOGLIMITREACHED
2020-Jun-27 07:15:14  1026359228   ACLMGR-0-TEST_SYSLOG     RATELIMITED
2020-Jun-27 07:15:09  384952880    ACLMGR-0-TEST_SYSLOG     RATELIMITED
2020-Jun-27 07:13:55  1679333688   ACLMGR-0-TEST_SYSLOG     PROCESSED:2:9332278
2020-Jun-27 07:13:52  1679333688   ACLMGR-0-TEST_SYSLOG     PROCESSING
2020-Jun-27 07:12:55  502545693    ACLMGR-0-TEST_SYSLOG     RATELIMITED
2020-Jun-27 07:12:25  1718497217   ACLMGR-0-TEST_SYSLOG     RATELIMITED
2020-Jun-27 07:08:25  1432687513   ACLMGR-0-TEST_SYSLOG     PROCESSED:2:10453823
2020-Jun-27 07:08:22  1432687513   ACLMGR-0-TEST_SYSLOG     PROCESSING
2020-Jun-27 07:06:16  90042807     ACLMGR-0-TEST_SYSLOG     RATELIMITED
2020-Jun-27 07:03:26  1737578642   ACLMGR-0-TEST_SYSLOG     RATELIMITED
2020-Jun-27 07:02:56  40101277     ACLMGR-0-TEST_SYSLOG     PROCESSED:3:10542045
2020-Jun-27 07:02:52  40101277     ACLMGR-0-TEST_SYSLOG     PROCESSING

Auto-Collection Log Files

About Auto-Collection Log Files

The configuration in a YAML file determines the contents of an auto-collected log file. You can't configure the amount of memory used for collected log files. You can configure the frequency of when the stored files get purged.

Autocollected log files get saved in the following directory:

switch# dir bootflash:eem_snapshots
   44205843    Sep 25 11:08:04 2019  1480625546_SECURITYD_2_FEATURE_ENABLE_DISABLE_eem_snapshot.tar.gz
  Usage for bootflash://sup-local
 6940545024 bytes used
44829761536 bytes free
51770306560 bytes total
Accessing the Log Files

Locate the logs by using the command keyword "debug": 

switch# dir debug:///
...
       26    Oct 22 10:46:31 2019  log-dump
       24    Oct 22 10:46:31 2019  log-snapshot-auto
       26    Oct 22 10:46:31 2019  log-snapshot-user

The following table describes the log locations and the log types stored.

Location

Description

log-dump

This folder stores Event logs on log rollover.

log-snapshot-auto

This folder contains the auto-collected logs for syslog events 0, 1, 2.

log-snapshot-user

This folder stores the collected logs when you run the bloggerd log-snapshot <> command.

Use the following example to view the log files generated on log rollover:

switch# dir debug:log-dump/
debug:log-dump/20191022104656_evtlog_archive.tar
debug:log-dump/20191022111241_evtlog_archive.tar
debug:log-dump/20191022111841_evtlog_archive.tar
debug:log-dump/20191022112431_evtlog_archive.tar
debug:log-dump/20191022113042_evtlog_archive.tar
debug:log-dump/20191022113603_evtlog_archive.tar
Parsing the Log tar Files

Use the following example to parse the logs in the tar files:

switch# show system internal event-logs parse debug:log-dump/20191022104656_evtlog_archive.tar
--------LOGS:/tmp/BLOGGERD0.991453012199/tmp/1-191022104658-191022110741-device_test-M27-V1-I1:0-P884.gz--------
2019 Oct 22 11:07:41.597864 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):Data Space Limits(bytes): Soft: -1  Ha rd: -1
2019 Oct 22 11:07:41.597857 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):Stack Space Limits(bytes): Soft: 500000  Hard: 500000
2019 Oct 22 11:07:41.597850 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):AS: 1005952076 -1
2019 Oct 22 11:07:41.597406 E_DEBUG Oct 22 11:07:41 2019(device_test_process_events):Sdwrap msg unknown
2019 Oct 22 11:07:41.597398 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):Going back to select
2019 Oct 22 11:07:41.597395 E_DEBUG Oct 22 11:07:41 2019(nvram_test):TestNvram examine 27 blocks
2019 Oct 22 11:07:41.597371 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):Parent: Thread created test index:4 thread_id:-707265728
2019 Oct 22 11:07:41.597333 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):Node inserted
2019 Oct 22 11:07:41.597328 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):The test index in diag is 4
2019 Oct 22 11:07:41.597322 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):result severity level
2019 Oct 22 11:07:41.597316 E_DEBUG Oct 22 11:07:41 2019(diag_test_start):callhome alert level

The following table describes the additional keywords available for parsing the specific tar file:

Keyword

Description

component

Decode logs belonging to the component identified by process name.

from-datetime

Decode logs from a specific date and time in yy[mm[dd[HH[MM[SS]]]]] format.

instance

List of SDWRAP buffer instances to be decoded (comma separated).

module

Decode logs from modules such as SUP and LC (using module IDs).

to-datetime

Decode logs up to a specific date and time in yy[mm[dd[HH[MM[SS]]]]] format.
Copying Logs to a Different Location

Use the following example to copy logs to a different location such as a remote server:

switch# copy debug:log-dump/20191022104656_evtlog_archive.tar scp://<ip-adress>/nobackup/<user> vrf management  use-kstack 
Enter username: user@<ip-address>'s password: 
20191022104656_evtlog_archive.tar                                             100%  130KB 130.0KB/s   00:00    
Copy complete, now saving to disk (please wait)...
Copy complete.
Purging Auto-Collection Log Files

There are two types of generated trigger-based auto-collection logs: EventHistory and EventBundle.

Purge Logic for EventHistory Logs

For event history, purging occurs in the /var/sysmgr/srv_logs/xport folder. 250MB of partitioned RAM is mounted at /var/sysmgr/srv_logs directory.

If the /var/sysmgr/srv_logs memory usage is under 65% of the 250MB allocated, no files get purged. When the memory utilization reaches the 65% limit level, the oldest files get purged until there's enough memory available to continue saving new logs.

Purge Logic for EventBundle Logs

For event bundles, the purge logic occurs in the /bootflash/eem_snapshots folder. For storing the auto-collected snapshots, the EEM auto-collect script allocates 5% of the bootflash storage. The logs get purged once the 5% bootflash capacity is used.

When a new auto-collected log is available but there's no space to save it in bootflash (already at 5% capacity), the system checks the following:

  1. If there are existing auto-collected files that are more than 12 hours old, the system deletes the files and the new logs get copied.

  2. If the existing auto collected files are less than 12 hours old, the system discards the newly collected logs without saving them.

You can modify the 12-hour default purge time by using the following commands. The time specified in the command is in minutes.

switch(config)# event manager applet test override __syslog_trigger_default
switch(config-applet)# action 1.0 collect test.yaml purge-time 300 $_syslog_msg

event manager command: test is an example name for the policy. __syslog_trigger_default is the name of the system policy that you want to override. This name must begin with a double underscore ( __ ).

action command: 1.0 is an example number for the order in which the action is executed. collect indicates that data is collected using the YAML file. test.yaml is an example name of the YAML file. $_syslog_msg is the name of the component.


Note

At any given time, there can be only one trigger-based auto-collection event in progress. If another new log event is attempting to be stored when auto-collection is already occurring, the new log event is discarded.

By default, there's only one trigger-based bundle collected every five minutes (300 sec). This rate limiting is also configurable by the following commands. The time specified in the command is in seconds. 

switch(config)# event manager applet test override __syslog_trigger_default
switch(config-applet)# action 1.0 collect test.yaml rate-limit 600 $_syslog_msg

event manager command: test is an example name for the policy. __syslog_trigger_default is an example name of the system policy to override. This name must begin with a double underscore ( __ ).

action command: 1.0 is an example number for the order in which the action is executed. collect indicates that data is collected using the YAML file. test.yaml is an example name of the YAML file. $_syslog_msg is the name of the component.

Beginning with Release 10.1(1), the rate of collection can also be regulated using a maximum number of triggers option, ensuring that only those many number of triggers are honored. After the max-triggers value is reached, no more bundles will be collected on the syslog occurrence. 


event manager applet test_1 override __syslog_trigger_default
  action 1.0 collect test.yaml rate-limt 30 max-triggers 5 $_syslog_msg

Note

If you delete auto collected bundles manually from debug:log-snapshot-auto/, then it will restart the collection based on the configured number of max-triggers when the next event occurs.

Auto-Collection Statistics and History

The following example shows trigger-based collection statistics:

switch# show system internal event-logs auto-collect statistics
---------------------EEM Auto Collection Statistics--------------------
Syslog Parse Successful :88 Syslog Parse Failure :0
Syslog Ratelimited :0 Rate Limit Check Failed :0
Syslog Dropped(Last Action In Prog) :53 Storage Limit Reached :0
User Yaml Action File Unavailable :0 User Yaml Parse Successful :35
User Yaml Parse Error :0 Sys Yaml Action File Unavailable :11
Sys Yaml Parse Successful :3 Sys Yaml Parse Error :0
Yaml Action Not Defined :0 Syslog Processing Initiated :24
Log Collection Failed :0 Tar Creation Error :0
Signal Interrupt :0 Script Exception :0
Syslog Processed Successfully :24 Logfiles Purged :0

The following example shows trigger-based collection history (the processed syslogs, process time, size of the data collected) obtained using a CLI command: 

switch# show system internal event-logs auto-collect history
DateTime Snapshot ID Syslog Status/Secs/Logsize(Bytes)
2019-Dec-04 05:30:32 1310232084 VPC-0-TEST_SYSLOG PROCESSED:9:22312929
2019-Dec-04 05:30:22 1310232084 VPC-0-TEST_SYSLOG PROCESSING
2019-Dec-04 04:30:13 1618762270 ACLMGR-0-TEST_SYSLOG PROCESSED:173:33194665
2019-Dec-04 04:28:47 897805674 SYSLOG-1-SYSTEM_MSG DROPPED-LASTACTIONINPROG
2019-Dec-04 04:28:47 947981421 SYSLOG-1-SYSTEM_MSG DROPPED-LASTACTIONINPROG
2019-Dec-04 04:27:19 1618762270 ACLMGR-0-TEST_SYSLOG PROCESSING
2019-Dec-04 02:17:16 1957148102 CARDCLIENT-2-FPGA_BOOT_GOLDEN NOYAMLFILEFOUND

Verifying Trigger-Based Log Collection

Verify that the trigger-based log collection feature is enabled by entering the show event manager system-policy | i trigger command as in this example: 

switch# show event manager system-policy | i trigger n 2
           Name : __syslog_trigger_default
    Description : Default policy for trigger based logging
    Overridable : Yes
     Event type : 0x2101

Checking Trigger-Based Log File Generation

You can check to see if the trigger-based auto-collection feature has generated any event log files. Enter one of the commands in the following examples: 

switch# dir bootflash:eem_snapshots
9162547 Nov 12 22:33:15 2019 1006309316_SECURITYD_2_FEATURE_ENABLE_DISABLE_eem_snapshot.tar.gz

Usage for bootflash://sup-local
8911929344 bytes used
3555950592 bytes free
12467879936 bytes total

switch# dir debug:log-snapshot-auto/
63435992 Dec 03 06:28:52 2019 20191203062841_1394408030_PLATFORM_2_MOD_PWRDN_eem_snapshot.tar.gz

Usage for debug://sup-local
544768 bytes used
4698112 bytes free
5242880 bytes total

Local Log File Storage

Local log file storage capabilities:

  • Amount of local data storage time depends on the scale, and type, of deployment. For both modular and nonmodular switches, the storage time is from 15 minutes to several hours of data. To be able to collect relevant logs that span a longer period:

  • Compressed logs are stored in RAM.

  • 250MB memory is reserved for log file storage.

  • Log files are optimized in tar format (one file for every five minutes or 10MB, whichever occurs first).

  • Allow snap-shot collection.

Generating a Local Copy of Recent Log Files

Extended Log File Retention is enabled by default for all services running on a switch. For local storage, the log files are stored on flash memory. Use the following procedure to generate a copy of up to ten of the most recent event log files.

SUMMARY STEPS

  1. bloggerd log-snapshot [ file-name ] [ bootflash: file-path | logflash: file-path | usb1: ] [ size file-size ] [ time minutes ]

DETAILED STEPS

Command or Action Purpose

bloggerd log-snapshot [ file-name ] [ bootflash: file-path | logflash: file-path | usb1: ] [ size file-size ] [ time minutes ]

Example:
switch# bloggerd log-snapshot snapshot1

Creates a snapshot bundle file of the last ten event logs stored on the switch. Default storage for this operation is logflash .

file-name : The filename of the generated snapshot log file bundle. Use a maximum of 64 characters for file-name .

Note

 

This variable is optional. If it is not configured, the system applies a timestamp and "_snapshot_bundle.tar" as the filename. Example: 

20200605161704_snapshot_bundle.tar

bootflash: file-path : The file path where the snapshot log file bundle is being stored on the bootflash. Choose one of the following initial paths:

  • bootflash:///

  • bootflash://module-1/

  • bootflash://sup-1/

  • bootflash://sup-active/

  • bootflash://sup-local/

logflash: file-path : The file path where the snapshot log file bundle is being stored on the logflash. Choose one of the following initial paths:

  • logflash:///

  • logflash://module-1/

  • logflash://sup-1/

  • logflash://sup-active/

  • logflash://sup-local/

usb1: : The file path where the snapshot log file bundle is being stored on the USB device.

size file-size : The snapshot log file bundle based on size in megabytes (MB). Range is from 5MB through 250MB.

time minutes : The snapshot log file bundle based on the last x amount of time (minutes). Range is from 1 minute through 30 minutes.

Example
switch# bloggerd log-snapshot snapshot1
Snapshot generated at logflash:evt_log_snapshot/snapshot1_snapshot_bundle.tar Please cleanup once done.
switch#
switch# dir logflash:evt_log_snapshot
159098880 Dec 05 06:40:24 2019 snapshot1_snapshot_bundle.tar
159354880 Dec 05 06:40:40 2019 snapshot2_snapshot_bundle.tar

Usage for logflash://sup-local
759865344 bytes used
5697142784 bytes free
6457008128 bytes total

Display the same files using the command in this example:

switch# dir debug:log-snapshot-user/
159098880 Dec 05 06:40:24 2019 snapshot1_snapshot_bundle.tar
159354880 Dec 05 06:40:40 2019 snapshot2_snapshot_bundle.tar

Usage for debug://sup-local
929792 bytes used
4313088 bytes free
5242880 bytes total

Note

The file name is identified at the end of the example. Each individual log file is also identified by the date and time it was generated.

Beginning with Release 10.1(1), the LC core file includes the log-snapshot bundle. The log-snapshot bundle filename is tac_snapshot_bundle.tar.gz. An example is shown below: 


bash-4.2$ tar -tvf 1610003655_0x102_aclqos_log.17194.tar.gz
drwxrwxrwx root/root 0 2021-01-07 12:44 pss/
-rw-rw-rw- root/root 107 2021-01-07 12:44 pss/dev_shm_aclqos_runtime_info_lc.gz
-rw-rw-rw- root/root 107 2021-01-07 12:44 pss/dev_shm_aclqos_runtime_cfg_lc.gz
-rw-rw-rw- root/root 107 2021-01-07 12:44 pss/dev_shm_aclqos_debug.gz
-rw-rw-rw- root/root 129583 2021-01-07 12:44 pss/clqosdb_ver1_0_user.gz
-rw-rw-rw- root/root 20291 2021-01-07 12:44 pss/clqosdb_ver1_0_node.gz
-rw-rw-rw- root/root 444 2021-01-07 12:44 pss/clqosdb_ver1_0_ctrl.gz
drwxrwxrwx root/root 0 2021-01-07 12:44 proc/
-rw-rw-rw- root/root 15159 2021-01-07 12:44 0x102_aclqos_compress.17194.log.25162
-rw-rw-rw- root/root 9172392 2021-01-07 12:43 0x102_aclqos_core.17194.gz
-rw-rw-rw- root/root 43878 2021-01-07 12:44 0x102_aclqos_df_dmesg.17194.log.gz
-rw-rw-rw- root/root 93 2021-01-07 12:44 0x102_aclqos_log.17194
-rw-rw-rw- root/root 158 2021-01-07 12:44 0x102_aclqos_mcore.17194.log.gz
drwxrwxrwx root/root 0 2021-01-07 12:44 usd17194/
-rw-rw-rw- root/root 11374171 2021-01-07 12:44 tac_snapshot_bundle.tar.gz

External Log File Storage

An external server solution provides the capability to store logs off-switch in a secure manner.


Note

To create the external storage capability, contact Cisco Technical Assistance Center(TAC) to help deploy the external server solution.

Te following are external log file storage capabilities:

  • Enabled on-demand

  • HTTPS-based transport

  • Storage requirements:

    • Nonmodular switches: 300MB

    • Modular switches: 12GB (per day, per switch)

  • An external server generally stores logs for 10 switches. However, there's no firm limit to the number of switches supported by an external server.

The external server solution has the following characteristics:

  • Controller-less environment

  • Manual management of security certificates

  • Three supported use-cases:

    • Continuous collection of logs from selected switches

    • TAC-assisted effort to deploy and upload logs to Cisco servers.

    • Limited on-premise processing


Note

Contact Cisco TAC for information regarding the setup and collection of log files in an external server.

Verifying the Embedded Event Manager Configuration

Use one of the following commands to verify the configuration:

Command Purpose

show event manager environment [variable-name | all]

Displays information about the event manager environment variables.

show event manager event-types [event | all | module slot]

Displays information about the event manager event types.

show event manager history events [detail] [maximum num-events] [severity {catastrophic | minor | moderate | severe}]

Displays the history of events for all policies.

show event manager policy-state policy-name

Displays information about the policy state, including thresholds.

show event manager script system [policy-name | all]

Displays information about the script policies.

show event manager system-policy [all]

Displays information about the predefined system policies.

show running-config eem

Displays information about the running configuration for EEM.

show startup-config eem

Displays information about the startup configuration for EEM.

Configuration Examples for Embedded Event Manager

The following example shows how to override the __lcm_module_failure system policy by changing the threshold for only module 3 hitless upgrade failures. It also sends a syslog message. The settings in the system policy, __lcm_module_failure, apply in all other cases. 

event manager applet example2 override __lcm_module_failure
event module-failure type hitless-upgrade-failure module 3 count 2
   action 1 syslog priority errors msg module 3 "upgrade is not a hitless upgrade!"
   action 2 policy-default

The following example shows how to override the __ethpm_link_flap system policy and shut down the interface: 

event manager applet ethport override __ethpm_link_flap
   event policy-default count 2 time 1000
   action 1 cli conf t
   action 2 cli int et1/1
   action 3 cli no shut

The following example shows how to create an EEM policy that allows the command to execute but triggers an SNMP notification when a user enters configuration mode on the device: 

event manager applet TEST
   event cli match "conf t"
   action 1.0 snmp-trap strdata "Configuration change"
   action 2.0 event-default

Note

You must add the event-default action statement to the EEM policy or EEM does not allow the command to execute.

The following example shows how to correlate multiple events in an EEM policy and execute the policy based on a combination of the event triggers. In this example, the EEM policy is triggered if one of the specified syslog patterns occurs within 120 seconds. 

event manager applet eem-correlate
   event syslog tag one pattern "copy bootflash:.* running-config.*"
   event syslog tag two pattern "copy run start"
   event syslog tag three pattern "hello" 
   tag one or two or three happens 1 in 120
   action 1.0 reload module 1

Additional References

Related Documents

Related Topic Document Title

EEM commands

Cisco Nexus 3600 NX-OS Command Reference

Standards

There are no new or modified standards supported by this feature, and support for existing standards has not been modified by this feature.