You can use graceful insertion and removal to gracefully eject a switch and isolate it from the network in order to perform debugging or upgrade operations. The switch is removed from the regular forwarding path with minimal traffic disruption. When you are finished performing debugging or upgrade operations, you can use graceful insertion to return the switch to its fully operational (normal) mode.

When you place the switch in maintenance mode, all configured Layer 3 control-plane protocols are isolated from the network. Directly connected routes are not withdrawn or modified during this state. When normal mode is restored, the advertisement of all routes is restored.

In graceful removal, all protocols and vPC domains are gracefully brought down and the switch is isolated from the network. In graceful insertion, all protocols and vPC domains are restored.

The following protocols are supported (for both IPv4 and IPv6 address families):

• Border Gateway Protocol (BGP)

• Enhanced Interior Gateway Routing Protocol (EIGRP)

• Intermediate System-to-Intermediate System (ISIS)

• Open Shortest Path First (OSPF)

• Protocol Independent Multicast (PIM)

• Routing Information Protocol (RIP)

Note	For graceful insertion and removal, the PIM protocol is applicable only to vPC environments. During graceful removal, the vPC forwarding role is transferred to the vPC peer for all northbound sources of multicast traffic.

Graceful Insertion and Replacement have the following configuration guidelines and limitations:

• Beginning with Cisco NX-OS Release 9.2(1), L2 Graceful Insertion and Replacement is supported. When moving from normal to maintenance mode, MCT goes down resulting in north to south traffic convergence. Zero packet loss is not supported. The following table provides an example of traffic convergence of 10 vPCs with 2 port member on each VPC port and 60k mac scale.

  Table 1.

  Trigger	Role	North to South Traffic	South to North Traffic
  Normal to maintenance mode	Primary	760 ms	1320 ms
  Maintenance mode to normal	Primary	13155 ms	27980 ms
  Normal to maintenance mode	Secondary	300 ms	1375 ms
  Maintenance mode to normal	Secondary	15905 ms	23350 ms

• Beginning with Cisco NX-OS Release 9.2(1), if you configure the isolate option for OSPF, direct routes and stub routes are advertised as max-metric routes. As a result, north-to-south traffic to the SVI hosts goes through the vPC peer when only one vPC switch is isolated.

• Remove all existing custom profiles before creating new custom profiles for normal-mode and maintenance-mode.

• Beginning with Cisco NX-OS Release 9.3(5), the include-local option is added to the existing isolate command. However, the include-local option applies only to router bgp.

• Beginning with Cisco NX-OS Release 10.3(1)F, during a binary reload scenario, when the system is switched to maintenance mode, the interface is shut until the system boots up completely and declares system readiness.

Follow these steps to complete the graceful insertion and removal (GIR) workflow:

1. (Optional) Create the maintenance-mode profile. (See Configuring the Maintenance-Mode Profile.)

2. (Optional) Create the normal-mode profile. (See Configuring the Normal-Mode Profile.)

3. Take a snapshot before triggering graceful removal. (See Creating a Snapshot.)

4. Trigger graceful removal to put the switch in maintenance mode. (See Triggering Graceful Removal.)

5. Trigger graceful insertion to return the switch to normal mode. (See Triggering Graceful Insertion.)

6. Take a snapshot after triggering graceful insertion. (See Creating a Snapshot.)

7. Use the show snapshots compare command to compare the operational data before and after the graceful removal and insertion of the switch to make sure that everything is running as expected. (See Verifying the GIR Configuration.)

Starting with Release 7.0(3)I5(1), the following maintenance mode enhancements have been added to Cisco Nexus 9000 Series switches:

• In the system maintenance shutdown mode, the following message is added:

  NOTE: The command system interface shutdown will shutdown all interfaces excluding mgmt 0.

• Entering the CLI command, system mode maintenance checks and sends alerts for the orphan ports.

• In isolate mode, when the vPC is configured, the following message is added:

  NOTE: If you have vPC orphan interfaces, please ensure vpc orphan-port suspend is configured under them, before proceeding further.

• Custom Profile Configuration: A new CLI command, system mode maintenance always-use-custom-profile is added for custom profile configuration. A new CLI command, system mode maintenance non-interactive is added for Cisco Nexus 9000 Series switches only. It provides a way to facilitate the transition to maintenance mode or normal mode without confirmation being done or each step being printed on the CLI session.

  When a loopback interface is configured with an IP address on a device, and this device is advertised to a peer device, then the device (with the loopback interface) moves to maintenance mode. In such a case, use the custom maintenance profile when system interface shutdown is configured on the device.

  When you create a custom profile (in maintenance or normal mode), it displays the following message:

  Please use the command system mode maintenance always-use-custom-profile  if you want to always use the custom profile.

• A delay has been added before the after_maintenance snapshot is taken. The no system mode maintenance command exits once all the configuration for the normal mode has been applied, the mode has been changed to normal mode, and a timer has been started to take the after_maintenance snapshot. Once the timer expires, the after_maintenance snapshot is taken in the background and a new warning syslog, MODE_SNAPSHOT_DONE is sent once the snapshot is complete.

  The final output of the CLI command no system mode maintenance indicates when the after_maintenance snapshot is generated:

  The after_maintenance snapshot will be generated in <delay> seconds. After that time, please use show snapshots compare before_maintenance after_maintenance to check the health of the system. The timer delay for the after_maintenance snapshot is defaulted to 120 seconds but it can be changed by a new configuration command.

  The new configuration command to change the timer delay for the after_maintenance snapshot is system mode maintenance snapshot-delay <seconds> . This configuration overrides the default setting of 120 seconds to any value between 0 and 65535 and it is displayed in the ASCII configuration.

  A new show command, show maintenance snapshot-delay has also been added to display the current snapshot-delay value. This new show command supports the XML output.

• A visible CLI indicator has been added to display when the system is in the maintenance mode, for example, switch(maint-mode)#.

• Support for the SNMP traps has been added when the device moves from the maintenance mode to the normal mode and vice-versa through CLI reload, or system reset. The snmp-server enable traps mmode cseMaintModeChangeNotify trap is added to enable changing to the maintenance mode trap notification. The snmp-server enable traps mmode cseNormalModeChangeNotify is added to enable changing to the normal mode trap notification. Both the traps are disabled by default.

• During maintenance mode, you can remove router from forward traffic with BGP active. You can advertise route installed locally to its iBGP peers with local setting in the absence of graceful shutdown. From Cisco NX-OS Release 10.4(1)F, you can configure custom isolation on route-map. In this mode, you can configure a route-map to choose the routes for BGP to be advertise.

To display the GIR configuration, perform one of the following tasks:

The redistribute direct configuration under Border Gateway Protocol (BGP) will attract traffic as the BGP isolate mode does not withdraw direct routes. This example shows how to use the route-map command to enable BGP to withdraw direct routes in isolate mode.

Policy Configuration

From Cisco NX-OS Release 10.4(1)F, you can match locally originated, redistributed direct routes.Configure route maps using below command:

Configuring redistribute for route-map

switch(config)# route-map <route map name>
switch(config-routemap)# match route-type redistributed-direct

Use the route-map my-rmap-deny command in maintenance mode to exclude SVIs with a tag 200 configuration.

switch(config)# route-map my-rmap-deny deny 10
switch(config-route-map)# match tag 200
switch(config-route-map)# exit
switch(config)# route-map my-rmap-deny permit 20

Use the route-map my-rmap-permit command in normal mode to include SVIs with a tag 200 configuration.

switch(config)# route-map my-rmap-permit permit 10
switch(config-route-map)# match tag 200
switch(config-route-map)# exit
switch(config)# route-map my-rmap-permit permit 20

Virtual IP (vIP)/Switch Virtual Interface (SVI) Configuration

switch(config)# interface loopback 200
switch(config-if)# ip address 192.0.2.100/8 tag 200
switch(config)# interface vlan 2
switch(config-if)# ip address 192.0.2.108/8 tag 200
....
switch(config)# interface vlan 3
switch(config-if)# ip address 192.0.2.102/8 tag 200

BGP Configuration

switch(config)# feature bgp
switch(config)# router bgp 100
switch(config-router)# neighbor 192.0.2.100
....

Maintenance Mode Profile

switch# configure maintenance profile maintenance-mode
switch(config-mm-profile)# router bgp 200
switch(config-mm-profile-router)# address-family ipv4 unicast
switch(config-mm-profile-router-af)# redistribute direct route-map my-rmap-deny
switch(config-mm-profile-router-af)# exit
switch(config-mm-profile)# sleep instance 1 10

Normal Mode Profile

switch# configure maintenance profile normal-mode
switch(config-mm-profile)# router bgp 100
switch(config-mm-profile-router)# address-family ipv4 unicast
switch(config-mm-profile-router-af)# redistribute direct route-map my-rmap-permit
switch(config-mm-profile-router-af)# exit
switch(config-mm-profile)# sleep instance 1 20

Custom Isolation Mode Profile

switch# configure maintenance profile maintenance-mode 
switch(config-mm-profile)# router bgp 100
switch(config-mm-profile-router)# isolate route-map <route map name>