SRP Hardware Troubleshooting Guide

Introduction

This document provides tips to troubleshoot spatial reuse protocol (SRP) links between Cisco routers. This document also provides examples of SRP troubleshooting at layers 1 and 2, and explains SRP concepts and describes how to use Cisco IOS^® commands to verify SRP connectivity.

Figure 1 shows the setup that this document uses.

Figure 1 – Topology

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Overview of OC-12c DPT

• Configuring the OC-12c DPT Port Adapter

Components Used

This document is not restricted to specific software and hardware versions.

Related Products

The hardware in this list currently supports SRP/ Dynamic Packet Transport (DPT) links between Cisco routers:

• 12xxx at Optical Carrier OC12/STM4 and OC48/STM16 and OC192/STM64

• Cisco 10720 router at OC48

• 1519x at OC12 and OC48

• 720x / 720xVXR at OC12

• uBR720x / uBR720xVXR at OC12

• 75xx at OC12

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Background Information

Here are the main factors in the installation of SRP/DPT links between routers:

• Side A must always connect to Side B.

• Transmit (Tx) must always connect to Receive (Rx).

• The power levels coming into the card must be within specification.

• The distance limitations must be within specification.

• Clocking must be set up correctly.

• Framing must be set up correctly.

Note: The link can come up and run for a while even if the power level is not within the specification. However, unexpected problems appear later if the power is not within the specification.

SRP Overview

This section provides an overview of the major components in SRP links between Cisco routers.

Fiber Type

There are two types of fiber for the OC12 SRP card:

• Multimode (MM)

• Singlemode (SM)

In general, there is one type of MM card and up to three different types of SM cards. The only difference between the SM cards are the power levels, which translates into the maximum distance the link can be between two nodes. The difference between the MM and SM cards is that the MM cards use a LED as the light source while the SM cards use a laser. The OC48 SRP cards come in SM only.

There is only one line card used for the12xxx (GSR) family, called 1-Port OC-192c/STM-64c DPT, that is available with very-short-reach (VSR), short-reach (SR), and intermediate-reach (IR) optics to meet your specific distance needs. Although the SR and IR models use the SC connectors and SM fiber, the VSR model uses a special connector called Multiple Terminations Push-pull (MTP) latch, which bundles 12x 62.5 micron MM fibers, and can operate for short distances up to 400 meters with lower costs. The VRS optics are connected with special MTP cables. Therefore VRS optics can interconnect only compatible devices, usually similar line cards in same room or building.

Fiber Topology

You can get fiber runs between SRP nodes in two ways:

• One is a Telco-provided circuit with Telco Synchronous Optical Network (SONET) equipment in-between the two SRP nodes (equipment like a multiplexer (MUX), fiber regenerator, or cross-connect). This is when you use the hard loopback test to demonstrate to the Telco that the SRP node (the Cisco router) is not at fault for any errors that occur.

• The other fiber set up is the use of dark fiber, which is sometimes called direct to fiber. Dark fiber is any run of fiber where the only equipment that provide power (light) are the end devices of the circuit. The Telco can provide this type of fiber, but the Telco does not have any equipment attached to the fiber; it is just fiber in the ground. Another example of dark fiber is where both nodes are in the same room, and a fiber run is installed between them.

Clocking and power level are the important factors of dark fiber. See the Clocking and Power Level sections of this document for details.

Clocking

SRP runs over a SONET link. Therefore, SRP interfaces have the same clocking rules as Packet-over-SONET (POS) interfaces. Like POS interfaces, you can set SRP interfaces to:

• Internal, which provides clock for the link

  OR

• Line, which receives clock from the link

Use the srp clock-source [type] [side] command under interface configuration mode to set each side (A and B) with its own clocking configuration.

Clocking is different for Telco networks and dark fiber networks. For Telco networks, you must set up the interface in the same way as the Telco, where usually everything is set to line clocking.

For dark fiber networks, the ideal clocking scheme is to set all the A sides to internal, and all the B sides to line. All sides set to internal also works, but BIP(Bx) errors show up when the clock starts to slip. You cannot set both sides to line clocking, because this is not supported.

Framing

There are two types of framing:

1. SONET

   SONET is the North American standard.

2. SDH

   SDH is the European standard.

Like clocking, framing can be side-independent if you use the srp framing [type] [side] command. The default framing is SONET.

Troubleshoot at Layer 1

SRP runs over SONET. Troubleshooting SRP physical layer problems is the same as troubleshooting a High-Level Data (HDLC) or Point to Point Protocol (PPP) Packet Over SONET (POS) link. Most problems with SRP links are due to improper physical configuration or power levels out of the specification.

Troubleshoot the Physical Configuration

The physical configuration of the fibers used for the SRP links is important for the ring to work correctly. Verify whether:

• Transmit (Tx) ports are connected to receive (Rx) ports

• Side A is connected to the correct neighbour side B

Figure 2 shows the configuration used in this lab setup.

Figure 2 – Configuration

Two possible physical setup errors can occur on an SRP ring:

• Transmit (Tx) is not connected to a Receive (Rx) port. This is the easiest scenario to troubleshoot as the SRP interface does not activate when incorrectly configured.

• Side B is not connected to Side A of the neighbor (Side B is connected to Side B). This scenario requires you to troubleshoot the incorrectly configured nodes.

Issue the show controllers srp command to check whether the physical setup is wrong.

In this example, the Rx ports have been switched on hswan-12410-3a. The path trace buffer is wrong for the links that are crossed. Remember, Tx is in fact connected to Rx, so the link comes up. However, here side B is connected to side B, which is an invalid configuration.

Figure 3 – Example of an Invaid Configuration

hswan-12410-3a#show controllers srp
SRP0/0 - Side A (Outer Rx, Inner Tx)
SECTION
  LOF = 1          LOS    = 1                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 16         BIP(B3) = 21
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-10720-3a
  Remote interface: SRP1/1
  Remote IP addr  : 100.1.1.4
  Remote side id  : A  

!--- The remote interface is also Side A. !--- This must be Side B. This is a physical cabling error.

          
BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

SRP0/0 - Side B (Inner Rx, Outer Tx)
SECTION
  LOF = 1          LOS    = 1                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 16         BIP(B3) = 18
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-12016-2a
  Remote interface: SRP12/0
  Remote IP addr  : 100.1.1.5
  Remote side id  : B 

!--- The remote interface is also Side B. !--- This must be Side A. This is a physical cabling error.


BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

In this case, hswan-12410-3a sees the below errors in the log. The other two nodes connected to hswan-12410-3a do not show these errors.

hswan-12410-3a#
%SRP-3-RING_ID_ERROR: SRP0/0 : Rx side A, Tx side of fibeA
%SRP-3-RING_ID_ERROR: SRP0/0 : Rx side B, Tx side of fibeB

If you put the Rx ports back to a proper configuration and switch the Tx ports on hswan-12410-3a, you get these errors on the nodes connected to hswan-12410-3a, but not on that node. That is why you must have a physical diagram of how the ring must be set up.

Figure 4 – How to Set Up the Ring

hswan-12016-2a#
%SRP-3-RING_ID_ERROR: SRP12/0 : Rx side B, Tx side of fibeB

hswan-10720-3a#
%SRP-3-RING_ID_ERROR: SRP1/1 : Rx side A, Tx side of fiber originates on A

!--- Note that the error syntax is different !--- on the Cisco 10720 router.

hswan-12016-2a#show controllers srp
SRP12/0 - Side A (Outer Rx, Inner Tx)
SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-12008-2b
  Remote interface: SRP6/0
  Remote IP addr  : 100.1.1.2
  Remote side id  : B
          
BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

SRP12/0 - Side B (Inner Rx, Outer Tx)
SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-12410-3a
  Remote interface: SRP0/0
  Remote IP addr  : 100.1.1.1
  Remote side id  : B 

!--- The remote interface is also Side B. !--- This must be Side A. This is a physical cabling error.


BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

hswan-12410-3a#show controllers srp
SRP0/0 - Side A (Outer Rx, Inner Tx)
SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-12016-2a
  Remote interface: SRP12/0
  Remote IP addr  : 100.1.1.5
  Remote side id  : B
          
BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

SRP0/0 - Side B (Inner Rx, Outer Tx)
SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-10720-3a
  Remote interface: SRP1/1
  Remote IP addr  : 100.1.1.4
  Remote side id  : A      

BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

hswan-10720-3a#show controllers srp
Interface SRP1/1
Hardware is OC48 SRP

SRP1/1 - Side A (Outer Rx, Inner Tx)

OPTICS
Rx readout values: -6 dBm    - Within specifications

SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-12410-3a
  Remote interface: SRP0/0
  Remote IP addr  : 100.1.1.1
  Remote side id  : A 

!--- The remote interface is also Side A. !--- This must be Side B. This is a physical cabling error.


BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

SRP1/1 - Side B (Inner Rx, Outer Tx)

OPTICS
Rx readout values: -5 dBm    - Within specifications

SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-12008-2b
  Remote interface: SRP6/0
  Remote IP addr  : 100.1.1.2
  Remote side id  : A

BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

Troubleshoot the Power Level

With the exception of the Cisco 10720 router, the correct way to check the power levels (sometimes referred to as the light level) is with a third party light tester. The Cisco 10720 router has a built-in power tester. You can see the output in the show controllers srp command.

To test the power level, take the power reading at the Rx end of the link. Disconnect the Rx fiber from the port and connect the Rx fiber to the light tester. This actually tests the Tx power from the other end of the link. The output of the test must fall within the power specifications of the card. Each type of card can have a different power range. Check the specifications for the card used.

The power level must be in the negative dBm range. If more power is added to the link, dBm is closer to zero. If there is too much power (a link that is too fast), you can add attenuation to the link with inline attenuators. These external attenuators usually run in 5dB increments. Add attenuation until the link is back within the specification. A fast link is usually just a power level problem and normally does not indicate a problem with the fiber or interface.

If the power level is too low (sometimes called a "cold" link), there can be a problem with:

• The fiber, for example, a fiber cut

• The distance of the link

• The interface to which fiber is connected

First, clean all optical connections and ensure that there are no problems with the fiber. For example, ensure that there are no kinks, breaks and tight bends. If the power level does not increase, try to reduce the number of fiber connections and splices, for example, patch panel connections. If the problem persists and the link has previously worked, there can be a problem as listed earlier in this section. In the case of a new installation, be sure to check the distance of the link to verify that the link is within specification. Remove any attenuation on the link. If the link still runs slowly, there can be a problem with:

• The interface

• An interface that is incorrectly mapped through the Telco

• An interface that you must change to a more powerful optic (out of distance specification)

Troubleshoot SONET Errors

Issue the show controllers srp command to troubleshoot physical SONET errors. This section provides a sample output of the command.

Note that there are two sets of statistics for each side of the ring. All the counters for both sides must be zero. These counters can have non-zero values without a problem with the link when:

• The link first comes up

• The fiber is removed or inserted

• The router reloads

If you find non-zero values, you must clear the counters, and recheck the values in the output from show controllers srp. If the error counts increment, there is a problem.

hswan-12410-3a#show controllers srp 0/0
SRP0/0 - Side A (Outer Rx, Inner Tx) 
 !--- Start of side A of the node.

SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0 

!--- Section counters must be zero.

LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0 

!--- Line counters must be zero.

PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0 

!--- Path counters must be zero.

  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0 

!--- Path counters must be zero.


Active Defects: None 
! -- A stable link should show "None"
Active Alarms:  None 
! -- A stable link should show "None"
Alarm reporting enabled for: SLOS SLOF PLOP

Framing           : SONET  
 !--- Framing type for this side of the node.

Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal 
 !--- Clock source for this side of the node.

Framer loopback   : None 
 !--- Shows whether the node has a software loop enabled.
 
Path trace buffer : Stable 
  Remote hostname : hswan-12016-2a 
 !--- Name of the remote node to which the SRP link is connected.
 
  Remote interface: SRP12/0 

!--- Remote interface to which the SRP link is connected.
 
  Remote IP addr  : 100.1.1.5 

!--- Remote interface to which the SRP link is connected. 

  Remote side id  : B 

!--- Remote side to which the link is connected. !--- Must be the opposite to local side!


BER thresholds:           SF = 10e-3  SD = 10e-6 

!--- Number of errors it has to receive to cause an Alarm.

IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6 

!--- Number of errors it has to receive to cause an Alarm.

TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6 

!--- Number of errors it has to receive to cause an Alarm.



SRP0/0 - Side B (Inner Rx, Outer Tx) 

!--- Start of side B of the node. Same layout/output as side A.

SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP 

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1 
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable 
  Remote hostname : hswan-10720-3a
  Remote interface: SRP1/1
  Remote IP addr  : 100.1.1.4
  Remote side id  : A      

BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

LOF and LOS Errors

Loss of Frame (LOF) errors occur when there are more than 3 ms of severely errored framing defects on the incoming SONET signal. Loss of Signal (LOS) errors occur when an all zeros pattern is detected on the incoming SONET signal for 19 (+/-3) microseconds or longer. LOS is also reported if the signal is lost (if the power is out of specification).

Both LOF and LOS are section errors and usually indicate that there is a problem between the node and the next SONET device (usually a SONET multiplexer [MUX] if going to a Telco network).

BIP(B1), BIP(B2), and BIP(B3) Errors

B1, B2, and B3 errors are the section, line, and path Bit Interleaved Parity errors that usually come into the interface. These values usually indicate either a problem with the link or the far end equipment. To troubleshoot, perform a hard loop back test on the interface. See the Hard Loopback Test section of this document for details.

AIS, RDI, and FEBE Errors

When a SONET network device detects LOF or LOS, the device sends an alarm indication signal (AIS) message to notify the downstream device, and a remote defect indication (RDI) message to notify the upstream device. The same is true for B2 and B3 errors, but these errors are reported as Far End Block Error Path (FEBE) errors.

If the show controllers srp command on router A sees FEBE errors, then you can infer that the device on the other end of this link has B2 or B3 errors, and reports the errors back to router A to indicate errors that come either from router A or the link.

Receipt of FEBE or remote defect indication (RDI) alarms does not point necessarily to a problem with the local interface. The fiber span can cause the errors. Again, a hard loopback test indicates whether there are errors. See the Hard Loopback Test section of this document for details.

LOP, NEWPTR, PSE and NSE Errors

Loss of Pointer (LOP), NEW SONET Pointer (NEWPTR), Positive Stuff Event (PSE) and Negative Stuff Event (NSE) errors indicate clocking errors with the link. The part of the SONET frame that these errors look at are the H1 and H2 bytes. If the node reports any of these errors, check the circuit for clocking issues. Even if both nodes on a link are configured correctly, a clocking issue within the Telco SONET network can cause these errors.

Hard Loopback Test

Perform a hard loopback test in order to rule out a problem with the router. Here are the prerequisites for this test:

• You must be able to take down the span that you need to test.

• You must have access to the router.

• You must have a fiber strand to connect the Tx port and Rx port.

• You musthave sufficient attenuation to get the interface into the specification with the fiber strand.

Complete these steps:

1. Isolate the span you want to work on from the rest of the ring.

   Note:  This is very important! If the span is not cut off from the rest of the ring, the SONET loop creates a dead stop in the ring, and the ring does not pass traffic anymore. This dead spot has the potential to kill all IPS packets that go around the ring. In order to isolate the span, Cisco recommends that you test from the rest of the ring. Complete these steps:

   1. Get into interface configuration mode for the node that will have the SONET loop.

   2. Issue the srp ips request forced-switch [side] command for a manual wrap of the side that will have the SONET loop.

      For example, if you want to put the SONET loop on side A of the node, issue the srp ips request forced-switch a command. This causes side B to wrap. Side B is still a part of the ring and still passes traffic. With side B wrapped, you can still work on side A of the node, with no effect to the rest of the ring.

2. Isolate the node on the other side of the span from the ring in the same way as in Step 1 (a) and (b).

3. Unplug the circuit from the interface.

4. Put one end of the fiber strand into the Tx port.

5. Check the power level that comes out of the fiber strand to be sure the level is within specification for that interface.

   If the power level is too high, use attenuators to cut the power level until the level is within specification.

6. Plug the other end of the fiber strand into the Rx port of the card.

7. Change the clock source for this interface to internal.

8. Clear the counters.

9. Wait a couple of minutes.

10. Run the show controllers srp command and check for errors.

Here is the output from the show controllers srp command, taken when there was a hard loop on side A. The path trace buffer reflects the same information as side A, and confirms that the port is looped (same hostname, interface, IP address and side ID).

This is important because most loop tests require the show interface command to see if the interface is up/up (looped). SRP does not report information like this so you cannot use the show interface command to see if the port is looped.

When the interface is confirmed as looped, you can check the interface for errors. If the interface reports errors, double-check the power level and the fiber strand. After you do this, if the interface still reports errors, replace the interface:

hswan-12008-2b#show controllers srp 1/0
SRP1/0 - Side A (Outer RX, Inner TX)
SECTION
  LOF = 0          LOS    = 0                            BIP(B1) = 0
LINE
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B2) = 0
PATH
  AIS = 0          RDI    = 0          FEBE = 0          BIP(B3) = 0
  LOP = 0          NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None
Active Alarms:  None
Alarm reporting enabled for: SLOS SLOF PLOP

Framing           : SONET
Rx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16
Tx SONET/SDH bytes: (K1/K2) = 0/0        S1S0 = 0  C2 = 0x16  J0 = 0x1
Clock source      : Internal
Framer loopback   : None
Path trace buffer : Stable
  Remote hostname : hswan-12008-2b 

!--- Check that host name is matched to verify that interface is looped.

  Remote interface: SRP1/0 

!--- Check that interface matches to verify that interface is looped.

  Remote IP addr  : 150.150.150.3 

!--- Check that IP address matches to verify that interface is looped.

  Remote side id  : A 

!--- Check that remote side ID matches to verify that interface is looped.


BER thresholds:           SF = 10e-3  SD = 10e-6
IPS BER thresholds(B3):   SF = 10e-3  SD = 10e-6
TCA thresholds:           B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

Be sure to turn off the forced wraps once the span is ready to be put back into the ring.

Troubleshoot at Layer 2

Use this section to troubleshoot Layer 2 with SRP.

SRP IPS

SRP uses Intelligent Protection Switching (IPS) to communicate to other nodes on the SRP ring. IPS provides SRP rings with powerful self-healing capabilities that allow them to automatically recover from fiber facility or node failure by wrapping the traffic on the failed span.

Each node on the SRP ring sends topology packets around the outer ring so all nodes on the ring know with whom they can communicate. Issue the show srp topology command to verify whether topology packets are sent and received around the ring:

hswan-12008-2b#show srp topology

 Topology Map for Interface SRP6/0
  Topology pkt. sent every 5 sec. (next pkt. after 1 sec.)
  Last received topology pkt. 00:00:03 

!--- If this value is higher than the topology packet sent value !--- (5 seconds), topology packet drops occur somewhere on the ring.

  Nodes on the ring: 4
  Hops (outer ring)      MAC       IP Address      Wrapped Name
      0             0003.a09f.5700 100.1.1.2         No    hswan-12008-2b
      1             0001.c9ec.d300 100.1.1.5         No    hswan-12016-2a
      2             0000.5032.3037 100.1.1.1         No    hswan-12410-3a
      3             0006.d74a.f900 100.1.1.4         No    hswan-10720-3a

This example has four nodes on the ring, where the first node (hop 0) is the local node. The output of the show srp topology command changes with the ring as long as the ring still receives topology packets.

Importantly, this output of the show srp topology command indicates when the last topology packet was received:

Last received topology pkt. 00:00:04

This information does not age out over time. So, if this counter is anything over the default five seconds, topology packets are being lost on the ring somewhere.

Note: You can change this timer with the srp topology-timer command.

If the ring loses topology packets, the node information can be wrong, because the node saves the last topology packet it receives. To verify which nodes are connected together, use the show controllers srp commands path trace buffer information to see the neighbor to which the node is physically connected.

This section shows how to troubleshoot for wrong configurations with the show srp ips command. Ensure that IPS reports no ring wraps, and that there is IDLE, SHORT status reported on IPS messages transmitted and received. The IPS requests reported must also be IDLE. Any other status indicates a problem with the SONET link.

This is an example of good show srp ips command output:

hswan-12008-2b#show srp ips srp 6/0 

 IPS Information for Interface SRP6/0
 MAC Addresses
   Side A (Outer ring Rx) neighbor 0006.d74a.f900
   Side B (Inner ring Rx) neighbor 0001.c9ec.d300
   Node MAC address 0003.a09f.5700
 IPS State
   Side A not wrapped 

!--- Must be in a "not wrapped" state.

   Side B not wrapped 

!--- Must be in a "not wrapped" state.

   Side A (Inner ring Tx) IPS pkt. sent every 1 sec. (next pkt. after 1 sec.)
   Side B (Outer ring Tx) IPS pkt. sent every 1 sec. (next pkt. after 1 sec.)
   inter card bus enabled
   IPS WTR period is 60 sec. (timer is inactive)
   Node IPS State: idle  

!--- Must be idle.

 IPS Self Detected Requests           IPS Remote Requests
   Side A IDLE                          Side A IDLE 

!--- Side A reports good IDLE status.

   Side B IDLE                          Side B IDLE 

!--- Side B reports good IDLE status.

 IPS messages received
   Side A (Outer ring Rx) {0006.d74a.f900,IDLE,SHORT}, TTL 255 

!--- Side A receives good "IDLE,SHORT" status.

   Side B (Inner ring Rx) {0001.c9ec.d300,IDLE,SHORT}, TTL 255 

!--- Side B receives good "IDLE,SHORT" status.

 IPS messages transmitted
   Side A (Outer ring Rx) {0003.a09f.5700,IDLE,SHORT}, TTL 128 

!--- Side A transmits good "IDLE,SHORT" status.

   Side B (Inner ring Rx) {0003.a09f.5700,IDLE,SHORT}, TTL 128 

!--- Side B transmits good "IDLE,SHORT" status.

This is an example of a bad show srp ips command (where side B is wrapped because side A is down):

hswan-12008-2b#show srp ips

 IPS Information for Interface SRP1/0
 MAC Addresses
   Side A (Outer ring Rx) neighbor 0003.a09f.5480
   Side B (Inner ring Rx) neighbor 0048.dc8b.b300
   Node MAC address 0003.a09f.5480
 IPS State
   Side A not wrapped
   Side B wrapped 

!--- Side B is wrapped because A is down.

   Side A (Inner ring Tx) IPS pkt. sent every 1 sec. (next pkt. after 1 sec.)
   Side B (Outer ring Tx) IPS pkt. sent every 1 sec. (next pkt. after 1 sec.)
   inter card bus enabled
   IPS WTR period is 60 sec. (timer is inactive)
   Node IPS State: wrapped 

!--- One side is wrapped.

 IPS Self Detected Requests           IPS Remote Requests
   Side A SF                            Side A IDLE 

!--- Side A reports SF instead of IDLE. This indicates !--- an error condition on the ring.

   Side B IDLE                          Side B IDLE
 IPS messages received
   Side A (Outer ring Rx) none 

!--- Side A is down, and does not receive any IPS messages.

   Side B (Inner ring Rx) {00b0.8e96.b41c,SF,LONG}, TTL 253 

!--- Side B reports SF,LONG instead of IDLE,SHORT.

 IPS messages transmitted 
   Side A (Outer ring Rx) {0003.a09f.5480,SF,SHORT}, TTL 128
   Side B (Inner ring Rx) {0003.a09f.5480,SF,LONG}, TTL 128

Verify whether you have a correct Address Resolution Protocol (ARP) table with the show arp command:

hswan-12008-2b#show arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  100.1.1.4              59   0006.d74a.f900  SRP-A  SRP6/0
Internet  100.1.1.1             234   0000.5032.3037  SRP-B  SRP6/0
Internet  100.1.1.2               -   0003.a09f.5700  SRP2   SRP6/0
Internet  150.150.150.4           3   00b0.8e96.b41c  SRP-B  SRP1/0
Internet  150.150.150.2          30   0048.dc8b.b300  SRP-B  SRP1/0
Internet  150.150.150.3           -   0003.a09f.5480  SRP    SRP1/0
Internet  150.150.150.1          30   0030.b660.6700  SRP-B  SRP1/0

• SRP—SRP version 1 (OC12 SRP)

• SRP2—SRP version 2 (OC48 SRP)

• SRP-A—Node connected to side A of the SRP interface

• SPR-B—Node connected to side B of the SRP interface

Note: All the entries for SRP1/0 have a type of SRP-B. This is because side A is down, so the node learns everything from side B of the interface.

The SRP interface can also be in pass-through mode. In order to ascertain this, issue the show interface command. Pass-through mode is when both sides of the interface cannot pass traffic. For example, when the interface is administratively shut down or both sides miss SRP keepalives. This causes the card to become an optical repeater on the ring. An important point about pass-through mode is that this mode alone does not cause the ring to wrap. Therefore, shutdown of a node does not cause IPS problems (this is good to troubleshoot ring problems). Here is a sample output of the show interface command:

hswan-12008-2b#show interface srp 1/0
SRP1/0 is administratively down, line protocol is down 
  Hardware is SRP over SONET, address is 0003.a09f.5480 (bia 0003.a09f.5480)
  Internet address is 150.150.150.3/24
  MTU 4470 bytes, BW 622000 Kbit, DLY 100 usec, rely 255/255, load 1/255
  Encapsulation SRP,
  Side A: loopback not set
  Side B: loopback not set
     4 nodes on the ring   MAC passthrough set 
     Side A: not wrapped   IPS local: IDLE       IPS remote: IDLE
     Side B: not wrapped   IPS local: IDLE       IPS remote: IDLE
  Last input 00:00:10, output 00:00:09, output hang never
  Last clearing of "show interface" counters 00:00:03
  Queueing strategy: fifo
  Output queue 0/40, 0 drops; input queue 0/75, 0 drops
  5 minute input rate 0 bits/sec, 1 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     Side A received errors:
        0 input errors, 0 CRC, 0 ignored,
        0 framer runts, 0 framer giants, 0 framer aborts,
        0 mac runts, 0 mac giants, 0 mac aborts
     Side B received errors:
        0 input errors, 0 CRC, 0 ignored,
        0 framer runts, 0 framer giants, 0 framer aborts,
        0 mac runts, 0 mac giants, 0 mac aborts

SRP Alarms

For help with SRP alarm messages, refer to the Alarm Messages section of Cisco 10720 Internet Router Installation and Configuration Guide.

SRP Debugs

The show commands are normally enough to troubleshoot SRP problems. However, there are situations where you must switch on debugs. Here are the two most frequently used debug commands:

• debug srp ips

• debug srp topology

Use debug srp ips to view the IPS packets that go around the ring. As with the show srp ips command, both sides must have a status of IDLE,SHORT.

Here is a good debug srp ips example where the node receives packets from both the A and B side of the ring (first two lines). It also transmits (Tx) IDLE,SHORT messages to the neighbor nodes (last two lines).

*Nov 3 02:46:47.899: srp_process_ips_packet: SRP1/0, checksum 64620, ttl 255, B 

!--- Receives packet from side B.

*Nov 3 02:46:48.139: srp_process_ips_packet: SRP1/0, checksum 14754, ttl 255, A 

!--- Receives packet from side A.

*Nov 3 02:46:48.403: Tx pkt node SRP1/0 side A {IDLE, SHORT} 

!--- Transmits (Tx) IDLE,SHORT msg to neighbor on side A.

*Nov 3 02:46:48.403: Tx pkt node SRP1/0 side B {IDLE, SHORT} 

!--- Transmits(Tx) IDLE,SHORT msg to neighbor on side B.

Here is a bad example of debug srp ips command where side B is down and side A is wrapped:

*Jan 4 21:11:25.580: srp_process_ips_packet: SRP12/0, 
checksum 50326, ttl 253,A
*Jan 4 21:11:26.200: Tx pkt node SRP12/0 side A {SF, LONG} 

!--- Transmits (Tx) IDLE,SHORT (error) msg to neighbor on side A.

*Jan 4 21:11:26.200: Tx pkt node SRP12/0 side B {SF, SHORT} 

!--- Transmits (Tx) IDLE,SHORT (error) msg to neighbor on side B.

Another debug command you can use is debug srp topology. The debugs show the flow of the topology packets around the ring. Note that on the wrapped node the node_wrapped status is a 1.

Here is a good example of debug srp topology with no wraps on the ring:

*Jan 3 23:34:01.846: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:34:01.846: srp_forward_topology_map_packet: SRP12/0, len 20
*Jan 3 23:34:01.846: srp_input: pkt_hdr=0x0F002007, flags=0x00000003 
*Jan 3 23:34:01.846: srp_forward_topology_map_packet: SRP12/0, len 20
*Jan 3 23:34:02.266: srp_send_topology_map_packet: SRP12/0 on side B 
- Not Wrapped
*Jan 3 23:34:02.266: srp_send_topology_map_packet: SRP12/0 on side A 
- Not Wrapped
*Jan 3 23:34:02.266: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:34:02.266: srp_consume_topology_map_packet: SRP12/0, len 34
*Jan 3 23:34:02.266: 0, src node_wrapped 0, src mac_addr 0001.c9ec.d300 
 !--- If the node is not wrapped, the node_wrapped bit should be zero (0).

*Jan 3 23:34:02.266: 1, src node_wrapped 0, src mac_addr 0000.5032.3037 
*Jan 3 23:34:02.266: 2, src node_wrapped 0, src mac_addr 0006.d74a.f900
*Jan 3 23:34:02.266: 3, src node_wrapped 0, src mac_addr 0003.a09f.5700
topology changed = No
*Jan 3 23:34:02.266: 0, src node_wrapped 0, src mac_addr 0001.c9ec.d300
*Jan 3 23:34:02.266: 1, src node_wrapped 0, src mac_addr 0000.5032.3037
*Jan 3 23:34:02.266: 2, src node_wrapped 0, src mac_addr 0006.d74a.f900
*Jan 3 23:34:02.266: 3, src node_wrapped 0, src mac_addr 0003.a09f.5700
topology updated = No
*Jan 3 23:34:02.266: srp_input: pkt_hdr=0x0F002007, flags=0x00000003 
*Jan 3 23:34:02.930: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:34:02.930: srp_forward_topology_map_packet: SRP12/0, len 13
*Jan 3 23:34:02.930: srp_input: pkt_hdr=0x0F002007, flags=0x00000003 
*Jan 3 23:34:02.930: srp_forward_topology_map_packet: SRP12/0, len 27
*Jan 3 23:34:04.194: srp_input: pkt_hdr=0x0F002007, flags=0x00000003 
*Jan 3 23:34:04.194: srp_forward_topology_map_packet: SRP12/0, len 13
*Jan 3 23:34:04.194: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:34:04.194: srp_forward_topology_map_packet: SRP12/0, len 27

Here is a bad example of debug srp topology with the node wrapped:

*Jan 3 23:44:47.042: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:47.042: srp_forward_topology_map_packet: SRP12/0, len 20
*Jan 3 23:44:47.058: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:47.058: srp_forward_topology_map_packet: SRP12/0, len 20
*Jan 3 23:44:47.486: srp_send_topology_map_packet: SRP12/0 on side B 
- Wrapped
*Jan 3 23:44:47.486: srp_send_topology_map_packet: SRP12/0 on side A 
- Wrapped
*Jan 3 23:44:47.486: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:47.486: srp_consume_topology_map_packet: SRP12/0, len 34
*Jan 3 23:44:47.486: 0, src node_wrapped 1, src mac_addr 0001.c9ec.d300 
 !--- If the node is wrapped, the node_wrapped bit should be one (1).

*Jan 3 23:44:47.486: 1, src node_wrapped 1, src mac_addr 0000.5032.3037 
*Jan 3 23:44:47.486: 2, src node_wrapped 0, src mac_addr 0006.d74a.f900
*Jan 3 23:44:47.486: 3, src node_wrapped 0, src mac_addr 0003.a09f.5700
topology changed = No
*Jan 3 23:44:47.486: 0, src node_wrapped 1, src mac_addr 0001.c9ec.d300
*Jan 3 23:44:47.486: 1, src node_wrapped 1, src mac_addr 0000.5032.3037
*Jan 3 23:44:47.486: 2, src node_wrapped 0, src mac_addr 0006.d74a.f900
*Jan 3 23:44:47.486: 3, src node_wrapped 0, src mac_addr 0003.a09f.5700
topology updated = No
*Jan 3 23:44:47.486: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:48.182: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:48.182: srp_forward_topology_map_packet: SRP12/0, len 13
*Jan 3 23:44:48.186: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:48.186: srp_forward_topology_map_packet: SRP12/0, len 27
*Jan 3 23:44:49.362: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:49.362: srp_forward_topology_map_packet: SRP12/0, len 27
*Jan 3 23:44:49.362: srp_input: pkt_hdr=0x0F002007, flags=0x00000002 
*Jan 3 23:44:49.362: srp_forward_topology_map_packet: SRP12/0, len 13

SRP Frequently Asked Questions

Here are some frequently asked questions:

• Question 1: Can I use an SM link with an MM card or an MM link with an SM card?

  Answer: No, but remember that the Rx port is only concerned with the receipt of the correct power level.

• Question 2: Can I connect an OC12 SRP card to an OC48 SRP card?

  Answer: No. Not only are the speeds different, but the OC12 also uses SRP version 1 while OC48 uses SRP version 2.

• Question 3: I see my own information in my path trace buffer. What is wrong?

  Answer: There is a loop somewhere that points back to that side of the node. Find the loop and remove the loop if the loop must not be there.

Related Information

• Optical Networking Product Support
• Optical Technology Support
• Technical Support & Documentation - Cisco Systems