Contents

Introduction

Background

Relevant Commands

Path Failure detection

Various Examples of path fail traps and explanations

Related Cisco Support Community Discussions

Introduction

This article covers understanding and troubleshooting Evolved General Packet Radio Service Tunneling Protocol for Control Plane (EGTPC) / GTPC (3rd Generation Partnership Project Technical Specification 29.274), GPRS Tunnelling protocol User Plane (GTPv1-U) (3GPP TS 29.281), and most relevant to this article, Restoration Procedures (3GPP TS 23.007) issues , which revolve around such concepts as restart counters, path failures detection, and echo requests/responses. It includes a list of relevant commands and what to look for in the output, along with explanation of related configurables. This is certainly an area for which a lot of questions are raised. Note that troubleshooting specific call control issues is in itself a very extensive area of interest, but this article only mentions how to capture statistics for call control - anything beyond that (i.e. call setup issues) is not covered. Really this article focuses mainly on troubleshooting and verifying the peer connections, without which there would be no call control.  While written for operators, designers / implementers could also benefit from the content here.

Examples all use fake (changed) IP addresses from a real operator's network.

Background

Each side of a GTP connection (one of S5 ( Serving Gateway (SGW) - Packet Data Network Gateway (PGW)), S11 ( Multimedia Management Entity (MME) - SGW), S4 ( Serving GPRS Service Node (SGSN) - Gateway GPRS Service Node (GGSN)) maintains a restart counter that is designed to alert the peer if the originating node has restarted so that that peer can take the appropriate action. While a node can use a unique counter for each peer it talks to, the design tends to be to use the SAME counter for all the peers, since when the node restarts, the counter needs to be incremented for all peers, and so tracking a separate value on a peer basis serves no extra benefit since they all need to increase by one anyway. In fact using the same restart counter for all peers is helpful in troubleshooting a large network where all the peers can be checked to make sure that the restart counters received have the same (single) value for the node those peers are talking to.

The restart counter CAN be included in all types of control messages, while the restart counter MUST be included in all echo requests and responses. As soon as a restart counter change occurs on a node, the various peers are informed about it via the next control messages sent to the respective peers. When a restart occurs, per specification, the receiving side has the option of temporarily maintaining the calls it has for a short while or in dropping them, the latter which is typically what is the configured behavior.

In the case of using echo requests, the method is to send the messages on a configured interval, and if no response is received after a configured number of retries (each after a configured timeout), the path is considered to be down and the appropriate action taken.  Configuration allows for echos to be optionally sent, and this would apply to the peer as well, whose approach doesn't necessarily have to be the same, especially when the peer could be from a different vendor and/or different carrier. In fact with regards to Echo responses, a node is only obligated to respond to an Echo request - whether it receives echos or not and with what frequency is not significant - it just needs to send an echo response for each request, and to take action if the restart counter received changes. Of course as already mentioned, IF it is also sending echo requests and doesn't receive timely responses, it needs to take action for that scenario as well.

Note that path failure detection can also be done over the GTP-U (user plane) connection via Echo, but ONLY via timeout, since there is no restart counter included in GTP-U messages like there is for GTP-C as previously discussed. GTP-U may or may not be using the same path as by control messages. If it is the same path, then GTP-U detection may not be as valuable since GTP-C detection should be sufficient.

In summary, here are the ways that path failure detection can be done:

- Restart Counter Change (via control msg Req/Rsp or Echo)

- GTPC Echo Req Time out

- GTPU Echo Req Time out

- Control Req Message Time-out (usually not configured)

Relevant Commands

In order to troubleshoot any kind of path issues, familiarize with the following commands. Following is an explanation of the relevant details of each.

- show [egtpc-service | gtpu-service] all

- show egtpc peers [egtp-service <service>]

- egtpc test echo gtp-version {1 | 2} src-address <service address> peer-address <peer address>

- show egtpc statistics [[[ (sgw-address | pgw-address | mme-address) <address>] [demux-only]] [debug-only] [verbose] | path-failure-reasons]

- show gtpu statistics [peer-address <peer>]

- show gtpc statistics [ggsn-service <service>] [smgr-instance <instance>]

- show demux-mgr statistics <egtpinmgr | egtpegmgr | gtpumgr> all (SGW-egress | GGSN/ PGW,/SGW ingress | GTPU)

- EGTPCPathFail[Clear], EGTPUPathFail[Clear] SNMP traps

- show session disconnect-reasons - gtpc-path-failure, gtpu-path-failure, path-failure

Other commands for counting subscribers related to GTP:

show sub [summary] ggsn-service <service>

show sub [summary] gtpu-service <service>

show egtpc sessions [egtp-service <service>]

The basic command to check the status and configuration of all EGTP services is "show egtpc-service". Most of the output should be what is already known via the configuration, as well as the status should be STARTED. The unique piece of information gleaned is the "Restart Counter" which cannot be found anywhere else. Related command "show gtpu-service all" returns configurables for the GTPU service(s) that are associated with the EGTPC services.

"show egtpc peers" is very useful in identifying all of the peers, including statuses, restart counters, and current/max subscriber counts.

- Note that this command covers both SGW and SGSN peers, whereas in the past there was a separate command for SGSN/GGSN.

- Since this command includes all peer types, the peer type can be filtered on via the service name (Even though the keyword egtpc-service deceptively also takes a GGSN service name).

- If there are no subscribers with the peer, then the status is Inactive and GTPC Echo disabled. In that state, for EGTPC there MAY be some peers for which the Restart counter is reported, while most may be 0.  This occurs for situations where the peer is actively sending echo requests to which ASR5K should be responding. The restart counter can be determined by running the EGTPC test command (discussed next), though in the inactive state, the counter will not be updated in "show egtpc peers" even though it was received.   Meanwhile for GGSN, the restart counter is never forgotten.

[local]PGW-GGSN> show egtpc peers egtp-service GGSN1

|||||  Service                       Restart--+   No. of

|||||  ID                            Counter  |  restarts

|||||   |                                     |     |     Current   Max

vvvvv   v              Peer Address           v     v     sessions sessions

-----  --- -------------------------------- --- --- ----------- -- ---------

IDNKG   6                   203.0.113.69    165     0           0           1

IDNKG   6                   203.0.113.70     25     0           0           1

IDNKG   6                   203.0.113.71    165     0           0           1

IDNKG   6                   203.0.113.72     25     0           0           1

IDNKG   6                   203.0.113.73     53     0           0           1

IDNKG   6                   203.0.113.74     53     0           0           1

IDNKG   6                   192.0.2.9        46     0           0           2

IDNKG   6                   192.0.2.10       44     0           0           1

IDNKG   6                   192.0.2.11       47     0           0           2

IDNKG   6                   192.0.2.12       44     0           0           2

IDNKG   6                   192.0.2.13       54     0           0           1

IDNKG   6                   192.0.2.14       47     0           0           1

IDNKG   6                   209.165.200.225  35     1           0           3

IDNKG   6                   209.165.200.226  35     3           0           3

IDNK   6                    198.51.100.204  156     0           0           1

IDNK   6                    198.51.100.205  158     0           0           1

IDNK   6                    198.51.100.211   54     0           0           1

IDNK   6                    198.51.100.212   54     0           0           1

IDNK   6                    198.51.100.213   56     0           0           1

IDNK   6                    198.51.100.214   56     0           0           1

IDNK   6                    198.51.100.215   54     0           0           1

Note a couple peers below that have counts much higher than the other peers. These are for SGWs that are physically located in the same site or the sister site of the PGW to which the MME funnels most calls.

[local]PGW> show egtpc peers

Monday March 30 20:10:26 UTC 2015

+----Status:                 (I) - Inactive  (A) - Active

|+---GTPC Echo:              (D) - Disabled  (E) - Enabled

||+--Restart Counter Sent:   (S) - Sent      (N) - Not Sent

|||+-Peer Restart Counter:   (K) - Known     (U) - Unknown

||||+-Type of Node:          (S) - SGW       (P) - PGW

|||||                        (M) - MME       (G) - SGSN

|||||                        (L) - LGW       (E) - ePDG

|||||                        (C) - CGW

|||||                        (U) - Unknown

|||||  Service                       Restart--------+  No. of

|||||  ID                            Counter        |  restarts

|||||   |                                           |   |   Current       Max

vvvvv   v              Peer Address                 v   v   sessions    sessions

-----  --- --------------------------------------- --- --- ----------- ----------

IDNUS   5                            198.51.100.4   0   0           0           1

IDNKS   5                           198.51.100.33  75   0           0           1

IDNUS   5                           198.51.100.35   0   0           0           2

IDNUS   5                           198.51.100.65   0   0           0           1

IDNUS   5                           198.51.100.67   0   0           0           2

IDNUS   5                           198.51.100.68   0   0           0           2

IDNUS   5                           198.51.100.98   0   0           0           2

IDNUS   5                          198.51.100.100   0   0           0           1

IDNUS   5                          198.51.100.129   0   0           0           2

IDNUS   5                          198.51.100.130   0   0           0           2

IDNUS   5                          198.51.100.161   0   0           0           2

IDNUS   5                          198.51.100.194   0   0           0           2

IDNUS   5                          198.51.100.196   0   0           0           1

IDNUS   5                          198.51.100.225   0   0           0           5

IDNUS   5                          198.51.100.226   0   0           0           2

IDNUS   5                          198.51.100.228   0   0           0           1

IDNUS   5                            198.51.100.3   0   0           0           2

IDNUS   5                           198.51.100.36   0   0           0           1

IDNUS   5                           198.51.100.64   0   0           0           4

IDNUS   5                           198.51.100.66   0   0           0           2

IDNUS   5                          198.51.100.130   0   0           0           2

IDNKS   5                           198.51.100.24   2   0           0           2

AESKS   5                           198.51.100.56   3   0           1           5

IDNKS   5                           198.51.100.99  18   0           0           2

AESKS   5                          198.51.100.100   7   0           1           3

IDNKS   5                          198.51.100.131  46   0           0           2

IDNKS   5                          198.51.100.132  18   0           0           2

IDNKS   5                          198.51.100.163  41   0           0           3

IDNKS   5                          198.51.100.164   6   0           0           2

AESKS   5                          198.51.100.184  11   0           1           2

IDNKS   5                          198.51.100.216   7   0           0           2

AESKS   5                            203.0.113.24   5   0           3           8

IDNKS   5                            203.0.113.56   3   1           0          13

IDNKS   5                            203.0.113.88   2   0           0          18

IDNKS   5                           203.0.113.120   6   0           0           3

IDNKS   5                           203.0.113.152   3   0           0           4

AESKS   5                           203.0.113.184   2   0           1           9

IDNKS   5                           203.0.113.216   4   0           0           2

IDNKS   5                           203.0.113.248  10   0           0           1

IDNUS   5                             203.0.113.4   0   0           0           2

IDNUS   5                            203.0.113.33   0   0           0           4

IDNUS   5                            203.0.113.65   0   0           0           2

IDNKS   5                           203.0.113.121   5   0           0           3

IDNKS   5                           203.0.113.153   6   0           0           5

IDNUS   5                           203.0.113.162   0   0           0           3

IDNUS   5                            203.0.113.32   0   0           0           5

AESKS   5                            203.0.113.35   5   1           6          37

IDNUS   5                            203.0.113.64   0   0           0           1

IDNUS   5                            203.0.113.66   0   0           0           4

IDNUS   5                            203.0.113.97   0   0           0           1

AESKS   5                            203.0.113.98   4   0           1           4

IDNUS   5                           203.0.113.161   0   0           0           4

IDNUS   5                           203.0.113.193   0   0           0          10

IDNUS   5                           203.0.113.228   0   0           0          12

IDNUS   5                           203.0.113.233   0   0           0           4

AESKS   5                               192.0.2.4   9   0           3          10

AESKS   5                              192.0.2.24   6   0          10          19

AESKS   5                              192.0.2.32   8   0          83         116

AESKS   5                              192.0.2.33  55   0          14          21

AESKS   5                              192.0.2.65  18   0           1        2747

AESKS   5                              192.0.2.66   4   0           3        1639

AESKS   5                              192.0.2.97  44  39          33        1826

AESKS   5                              192.0.2.98   5   1           6        2075

IDNKS   5                             192.0.2.195   8   0           0           7

AESKS   5                               192.0.2.3  46   0        3230        5346

AESKS   5                               192.0.2.5  12   0          84         202

AESKS   5                               192.0.2.7   2   0          31         443

AESKS   5                              192.0.2.24   3   0         116         117

AESKS   5                              192.0.2.36  15   0        4194        6889

AESKS   5                              192.0.2.44   7   0         112         331

AESKS   5                              192.0.2.68  34   0        6104        9798

AESKS   5                              192.0.2.69   7   0        2259        5027

AESKS   5                              192.0.2.88  17   0        3899        8880

AESKS   5                             192.0.2.164  16   0      833131     1298070

AESKS   5                             192.0.2.184   3   0     1437053     2153317

AESKS   5                             192.0.2.193  21   0       11364       26577

AESKS   5                             192.0.2.195   5   0          50         162

AESKS   5                             192.0.2.226  21   0        1950        3931

AESKS   5                             192.0.2.227   6   0        4348        9332

Note that running this command on an Inter-Chassis Service Redundancy (ICSR) standby node is only relevant for active peers, in which case the numbers should be close, for example in the following pair:

PGW> show egtpc peers ... 20:51:52  (ACTIVE)

PGW-ICSR> show egtpc peers ... 20:51:51 (STANDBY)

Peer

RC

# sessions

Max Sess

Peer

RC

# sessions

Max Sess

AESKS

192.0.2.5

2

31

443

ADSKS

192.0.2.5

2

31

443

AESKS

192.0.2.24

3

117

123

ADNKS

192.0.2.24

3

117

123

AESKS

192.0.2.36

15

4223

6889

ADSKS

192.0.2.36

15

4223

6889

AESKS

192.0.2.44

7

119

331

ADSKS

192.0.2.44

7

119

331

AESKS

192.0.2.68

34

6174

9798

ADSKS

192.0.2.68

34

6171

9797

AESKS

192.0.2.69

7

2311

5027

ADSKS

192.0.2.69

7

2311

5027

AESKS

192.0.2.88

17

3914

8880

ADSKS

192.0.2.88

17

3913

8874

AESKS

192.0.2.164

16

833705

1298070

ADSKS

192.0.2.164

16

832037

1294412

AESKS

192.0.2.184

3

1438418

2153317

ADSKS

192.0.2.184

3

1435461

2147280

AESKS

192.0.2.193

21

11447

26577

ADSKS

192.0.2.193

21

11443

26568

AESKS

192.0.2.195

5

48

162

ADSKS

192.0.2.195

5

48

162

AESKS

192.0.2.226

21

1962

3931

ADSKS

192.0.2.226

21

1962

3931

AESKS

192.0.2.227

6

4465

9332

ADSKS

192.0.2.227

6

4466

9332

"egtpc test echo" is used to check a specific peer to see if it is reachable or not and must be run in the context where the service is defined.

- Use gtp-version 1 for GGSN-SGSN connections

- Using ping command is not a valid test, though if it is successful, it is known that there is some level of reachability.

- The peer restart counter (Recovery) is displayed.

- Running this command will increment the Tx echo request / Rx Echo response counters in "show egtpc statistics"

- The restart counter of the peer is reported after Recovery keyword (11 in example below)

- Warning: though this should not happen in normal circumstances, if a test echo is performed on a peer and it times out for whatever reason (if all is working then it shouldn't time out), all the calls would e dropped (even though things were working just fine up to the point of doing the test).

[Ingress]PGW> egtpc test echo gtp-version 2 src-address  209.165.201.1 peer-address 198.51.100.35

Monday November 03 15:08:12 UTC 2014

EGTPC test echo

---------------

Peer: 198.51.100.35   Tx/Rx:  1/1  RTT(ms): 1    (COMPLETE) Recovery: 11 (0x0B)

"show egtpc statistics" is for EGTPC (v2) and reports on success/failures of many EGTPC control messages (Request/Response, Tx/Rx) including Create Session, Modify Bearer, Delete Session, Downlink Data Notify, Release Access Bearer, Create Bearer, Update Bearer, Modify Bearer, Suspend, etc., and the associated responses. It includes all path management counters for Echo Request/Response Tx/Rx, and based on the timers, the growth of these timers can be predicted and used to troubleshoot if not incrementing as expected.

The following example shows echos and responses being initiated and responded to from both sides. Note that the SGW has been configured to send once per minute while the PGW has been configured for once every three minutes (both sides don't need to have the same setting) and so every three minutes there are twice the number of packets (6:16:33, 6:19:33). Keep in mind that although both sides are on the same "cycle" in this example, they do not have to be and can run independent of eachother.  The restart counters has been captured in a column as well. Note the Restart counter of 11 from the SGW's echo response matches the value from its response to the PGW's test echo from above (the test was done at different time than the packet capture but for the same peer pair).

The demux-only option captures all the path management counters (echo requests/responses) as well as control messages that never reach a sessmgr, which is generally Create Session Request/Response. Demux process etpinmgr residing on the Demux card is responsible for handling all echo request/response and initial handling of all call control messages before distributing them to the appropriate sessmgrs (randomly for new calls, and specifically for existing calls anchored on a specific sessmgr). The sessmgr-only option conversely only counts messages that get handled by sessmgrs, which does NOT include path management counters. Not specifying either qualifiers is the default of counting all values.

"show gtpc statistics" is the analogous command for SGSN/GGSN services (GTPC v1) (CPC, Update PDP Context, etc.)

"show gtpu statistics" reports stats on the user-plane traffic handled by the GTP-U service(s) that are associated with EGTPC service(s). One interesting counter is Error Indication Tx/Rx which would be sent when the receiving node has no record of the subscriber that should be associated with the Terminal Endpoint Identifier (TEID) of the packet in question which could happen for a number of reasons (The bulkstats schema/variable is GTPU1/ err-ind-tx). If the MME/SGW has not been informed that the PGW no longer has a binding to the subscriber, then calls may need to be manually cleared on the MME/SGW (or wait for them to timeout) to fully resolve  such an issue of orphaned calls.

The peer specific versions of the above stat commands are necessary when troubleshooting specific peers, which often is the case.

Path Failure detection

The EGTPCPathFail trap is the key for knowing when a path failure has occurred. Amongst some basic values that one would already assume, it reports the number of calls dropped, the old and new restart counters, and the reason for the failure. In addition, the session disconnect reason gtpc-path-failure will be incremented.

The EGTPCPathFailClear trap will indicate the restart counter along with the reason for the path clearing. The most recent StarOS v17 is better worded to say "clear reason" instead of "failure reason" (it cleared, not failed!) to avoid confusion.

The following are path failure counts that are reported from "show egtpc stat path-failure-reasons"

[local]PGW> show egtpc statistics path-failure-reasons

Reasons for path failure at EGTPC:

Echo Request  restart counter change:                3

Echo Response restart counter change:                0   echo-rsp-restart-counter-change

No Echo Response received:                           0   no-response-from-peer

Control message restart counter change at   demux:  23   Create Session Req Restart Counter changed or create-sess-restart-counter-change

Control message restart counter change at sessmgr:   0   cpc-restart-counter-change or upc-restart-counter-change

Total path failures detected:                       26

Notes:

- Control message restart counter at sessmgr could be any type of control message (not all listed here)

- Changed restart counter from the SGSN via the next Create PDP Context Request after that change(cpc-restart-counter-change)

- Changed restart counter from the SGSN via the next Update PDP Context Request after that change (which would be a handoff from another SGSN) (upc-restart-counter-change)

- Path clear failure reason sessions-add is a deceiving term - it is not a failure but a success, and it indicates that a new session has come in to establish the connection

- If the values reported for restart counters and session counts for the PathFail and PathClear traps do not seem to line up, open a ticket with Cisco for further explanation.

It is important to understand how EGTPC/GTPC and to a lesser degree EGTPU/GTPU (as user-plane failure detection is not always configured) path failure detection is implemented to monitor the connection between a GGSN or PGW and the various peer SGSN and SGWs. Normally the connections within a service providers network are fairly stable, in which case path failures are infrequent. On the flip side, experience has shown that GTPC paths amongst roaming SGSNs connecting to GGSNs may not be so stable, and so with poor connections the result is path failures happening at a fairly frequent basis.

The following config covers timers and transmission frequency for control and echo messages which can be used to detect path failures. Same concept applies to GGSN, SGW, MME, and EGTP services:

ggsn-service GGSN1

retransmission-timeout 20

For GTP control messages (versus the echo messages)

max-retransmissions 5

for BOTH GTP control and echo messages – how many times to retransmit before path failure detection trigger

echo-interval 120

An echo is sent every 2 minutes (120 seconds).

echo-retransmission-timeout 5

For Echos, if no response, it retransmits every 5 seconds

path-failure detection-policy gtp echo

Path failure detection is possible via gtp echo (versus not using this as a method of detecting path failure (but still sending echos)).

associate gtpu-service GGSN1_gtpu

Associates the named GTPU service GGSN1_gtpu with this GGSN service

bind ipv4-address 209.165.201.2

The user and control plane bind IP addresses are the same in the following example configs, but may not necessarily be configured as such with other service provider implementations.

[local]GGSN> show ggsn-service name GGSN1

Service name:           GGSN1

Context:                XGWin

Bind:                   Done

Local IPv4 Address:     209.165.201.2     Local IP Port: 2123

Retransmission Timeout: 20 (secs)

Max Retransmissions:    5

Restart Counter:        41

Echo Interval:          120 (secs)

GTPC Echo Mode                        : Default

GTPC Echo Retransmission Timeout      : 5

Guard Interval:         100 (secs)

Path Failure Detection on gtp msgs: Echo

Notes:

- Given time 0 being when an Echo request cycle starts (every 2 minutes), if there are no responses after a TOTAL of 30 seconds (6 attempts), the path fails (0 + 5sec + (5 * 5sec))

- Process egtpinmgr (gtpcmgr deprecated since v14) running on the demux card is responsible for processing the echo packets

- “show gtp statistics”, section “Path Management Messages” has statistics for the echo requests/responses for the control path in BOTH directions:

[Ingress]GGSN> show gtpc statistics

Monday October 27 23:02:07 UTC 2014

Path Management Messages:

Echo Request RX:        169951248   Echo Response TX:        169951248

Echo Request TX:         40838432   Echo Response RX:         40784820

Another method of path detection is on the user plane instead of the control plane, and this is governed by the following config.

gtpu-service GGSN1_gtpu

no echo-interval

max-retransmissions 4

echo-retransmission-timeout 5

path-failure detection-policy gtp echo

no path-failure clear-trap gtp echo

bind ipv4-address 209.165.201.2

exit

Notes:

- “no echo-interval” disables sending of echo requests for user plane and therefore CLI “path-failure detection-policy gtp echo” has no relevance – NO Path failure detection is done on GTPU

- GGSN still does respond to echo requests from the SGSN though

- “show gtpu stat” and “show demux-mgr stat gtpumgr” capture echo stats

[local]PGW-GGSN> show gtpu statistics

Wednesday October 29 23:43:22 UTC 2014

Path Management Messages:

Echo Request Rx:           84690303  Echo Response Rx:            0

Echo Request Tx:                  0  Echo Response Tx:     84690043

[local]PGW-GGSN> show demux-mgr statistics gtpumgr

Wednesday October 29 23:42:04 UTC 2014

Echo stats:

Echo Req Rcvd:               84690303

Echo Rsp Sent:               84690043

Various Examples of path fail traps and explanations

Usually the peer restart counter will increase by one when there is a restart. In this case, a bug in PGW caused it to override the restart counter to the value of the RC of another peer (first trap), and so when the next packet arrived over the connection, it treated it as a restart (second trap) thinking that the RC had changed. The path then clears on any successive packet (third trap). The point here is not to know about a specific bug but rather to try and be aware of any counter changes that cannot be explained.

Wed Jan 14 05:22:07 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.1, peer old restart counter 54,  peer new restart counter 12,  peer session count  1107240, failure reason restart-counter-change

Wed Jan 14 05:22:07 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.1, peer old restart counter 12,  peer new restart counter 54,  peer session count  1107207, failure reason create-sess-restart-counter-change

Wed Jan 14 05:22:07 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.1, peer restart counter 54,   peer session count  1107199, failure reason sessions-add

This example is from a GGSN where the peer SGSN has changed its RC from 66 to 67 via a Create PDP Context Request:

Mon Oct 27 11:48:54 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.201.2,  peer address 209.165.200.225, peer old restart counter 66,  peer new restart counter 67,  peer session count  0, failure reason cpc-restart-counter-change

Mon Oct 27 11:48:54 2014 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.201.2,  peer address 209.165.200.225, peer restart counter 67,   peer session count  1, failure reason sessions-add

This example is from a PGW where the peer ePDG has changed its RC from 69 to 70 via a Create Session Request:

Fri Jun 12 05:21:54 2015 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 2001:5555:8000:fff0:132:200::,  peer address 2001:5555:8000:ffe0:123:20f::, peer old restart counter 69,  peer new restart counter 70,  peer session count  0, failure reason create-sess-restart-counter-change

Fri Jun 12 05:21:55 2015 Internal trap notification 1113 (EGTPCPathFailClear) context PGWin, service EGTP1, interface type pgw-ingress, self address 2001:5555:8000:fff0:132:200::,  peer address 2001:5555:8000:ffe0:123:20f::, peer restart counter 70,   peer session count  1, clear reason sessions-add

In this example, the output from various PGW nodes all connected to the same SGW show that the SGW stopped becoming reachable from the PGWs per an EGTPCPathFail trap at the various PGWs within a short time frame of eachother. Note that all show the same old restart counter as was discussed earlier that normally a node uses the same counter for all its peers. In the case of no response, the new restart counter displayed is 9, but the new counter might better be printed as undefined since the connection is down as of when the trap is triggered and so having a restart counter doesn't really make sense.

Mon Dec 22 09:07:29 2014 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.4,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  151, failure reason no-response-from-peer

Mon Dec 22 09:08:54 2014 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.5,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  634, failure reason no-response-from-peer

Mon Dec 22 09:09:00 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.6,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  22, failure reason no-response-from-peer

Mon Dec 22 09:07:42 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.7,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  33, failure reason no-response-from-peer

Mon Dec 22 09:08:06 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.8,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  1, failure reason no-response-from-peer

Mon Dec 22 09:07:34 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.9,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  725, failure reason no-response-from-peer

Mon Dec 22 09:08:20 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.10,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  279978, failure reason no-response-from-peer

Mon Dec 22 09:06:37 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.11,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  23, failure reason no-response-from-peer

Mon Dec 22 09:07:29 2014 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.12,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  6, failure reason no-response-from-peer

Mon Dec 22 09:07:01 2014 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.13,  peer address 198.51.100.162, peer old restart counter 9,  peer new restart counter 9,  peer session count  5, failure reason no-response-from-peer

The following example shows a path failure and same new restart counter as in the previous example, but then a half hour later shows when the peer comes back up again and the counter has increased by one from 15 to 16. Over time the count increases for that peer:

Wed Jul 22 05:07:08 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.13,  peer address 198.51.100.162, peer old restart counter 15,  peer new restart counter 15,  peer session count  34, failure reason no-response-from-peer

Wed Jul 22 05:38:00 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.13,  peer address 198.51.100.162, peer restart counter 16,   peer session count  1, clear reason sessions-update

[local]PGW> show egtpc peers

Wednesday July 22 10:04:30 UTC 2015

|||||  Service           Restart--------+   No. of

|||||  ID                Counter        |   restarts

|||||   |                               |    |   Current       Max

vvvvv   v              Peer Address     v    v   sessions    sessions

-----  --- --------------------------------------- --- --- -----------

AESKS   5               198.51.100.162  16   1        1157        2186

[XGWin]PGW> egtpc test echo gtp-version 2 src-address 209.165.201.13 peer-address 198.51.100.162

Wednesday July 22 10:52:41 UTC 2015

EGTPC test echo

---------------

Peer: 198.51.100.162                          Tx/Rx:  1/1  RTT(ms): 51   (COMPLETE) Recovery: 16 (0x10)

This example shows a repeating path failure due to CPC restart for 3 specific SGSN peers to the GGSN. The output below shows just one of the peers 209.165.200.225, but they all have the same issue. Repeated run of "show egtpc peers" and "show gtpc stat sgsn-address" also shows the alternating values. "show gtpc stat" confirms no issue with connectivity with respect to echo requests and responses, which in this case are only being initiated by the GGSN (acceptable as discussed earlier). One sidebar point to be made here is that there are often multiple pieces of data that can be collected that all corroborate to the same conclusion - one can choose to collect the data that is the most conclusive with the minimal amount of effort.

show snmp trap history verbose | grep -E "209.165.201.31|209.165.200.225|209.165.200.246"

Mon Jun 22 22:20:51 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 129,  peer new restart counter 132,  peer session count  3, failure reason cpc-restart-counter-change

Mon Jun 22 22:20:51 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 132,   peer session count  4, clear reason sessions-add

Mon Jun 22 22:20:55 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 132,  peer new restart counter 129,  peer session count  3, failure reason cpc-restart-counter-change

Mon Jun 22 22:20:55 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 129,   peer session count  4, clear reason sessions-add

Mon Jun 22 22:20:58 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 129,  peer new restart counter 132,  peer session count  3, failure reason cpc-restart-counter-change

Mon Jun 22 22:20:58 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 132,   peer session count  4, clear reason sessions-add

Mon Jun 22 22:21:02 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 132,  peer new restart counter 129,  peer session count  2, failure reason cpc-restart-counter-change

Mon Jun 22 22:21:02 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 129,   peer session count  3, clear reason sessions-add

Mon Jun 22 22:21:03 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 129,  peer new restart counter 132,  peer session count  4, failure reason echo-rsp-restart-counter-change

Mon Jun 22 22:21:03 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 132,   peer session count  5, clear reason sessions-add

Mon Jun 22 22:21:06 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer old restart counter 132,  peer new restart counter 129,  peer session count  2, failure reason cpc-restart-counter-change

Mon Jun 22 22:21:06 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service GGSN1, interface type ggsn, self address 209.165.202.129,  peer address 209.165.200.225, peer restart counter 129,   peer session count  3, clear reason sessions-add

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225

|||||  Service                      Restart No. of

|||||  ID                           Counter restarts

|||||   |                              |     |        Current      Max

vvvvv   v              Peer Address    |     |       sessions    sessions

|     |            |

v     v            v

Monday June 22 23:09:26 UTC 2015

AESKG   6             209.165.200.225 129 86339           2          57

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225

Monday June 22 23:09:28 UTC 2015

AESKG   6             209.165.200.225 129 86341           3          57

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225

Monday June 22 23:09:29 UTC 2015

AESKG   6             209.165.200.225 132 86342           7          57

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225

Monday June 22 23:09:30 UTC 2015

AESKG   6             209.165.200.225 132 86344           9          57

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225

Monday June 22 23:09:31 UTC 2015

AESKG   6             209.165.200.225 129 86345          11          57

[XGWin]GGSN> show egtpc peers | grep 209.165.200.225

Monday June 22 23:09:32 UTC 2015

AESKG   6             209.165.200.225 129 86345           7          57

[XGWin]GGSN> show gtpc statistics sgsn-address 209.165.200.225

Monday June 22 23:07:36 UTC 2015

SGSN Address:    209.165.200.225    Status:          Active

Total Restarts:  86287             Restart Counter: 129

Session Stats:

Total Current:                  2   S6b Assume Positive:             0

...

Path Management Messages:

Echo Request RX:                0   Echo Response TX:                0

Echo Request TX:            22919   Echo Response RX:            22917

Looking at other peers in the subnet, they all show a RC = 132 all the time, while the broken peers alternate between 129 and 132 as just shown, which points towards these SGSN peers as potentially passing the wrong RC in the broken scenarios:

Restart

Counter

AESKG   6        209.165.200.225 129 85115         8          57  <==

IDNKG   6        209.165.200.226 132  21           0           3

AESKG   6        209.165.200.227 132  60          47          92

AESKG   6        209.165.200.228 132  53          24          59

AESKG   6        209.165.200.229 132  69          92         126

AESKG   6        209.165.200.232 132  66          21          30

AESKG   6        209.165.200.236 132  43           8          21

AESKG   6        209.165.200.238 132  49          18          32

AESKG   6        209.165.200.239 132  39           1           6

AENKG   6        209.165.200.240 132  17           1           5

AESKG   6        209.165.200.241 132  67           6          43

AESKG   6        209.165.200.242 132  56          22          29

AENKG   6        209.165.200.243 132  40           2           9

AENKG   6        209.165.200.246 129 17713         3          24  <==

AESKG   6        209.165.200.247 132  64          29          45

AESKG   6        209.165.200.250 132  62          42          56

Recall from earlier, another approach to issues like this is to check other GGSNs (or PGWs for LTE) to see if they have the same issue with those seemingly problematic SGSN peers, and in this case they do also. For GGSN3 the first run of the command showed values 132 for both peers but another run showed 129 for one of the peers. Multiple runs (not shown here for brevity) would show all GGSNs bouncing between the two RCs for those peers over time:

[local]GGSN1> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"

Tuesday June 23 11:37:23 UTC 2015

AENKG   7               209.165.200.246 129 20147           1          29

AESKG   7               209.165.200.225 129 92853           2          51

[local]GGSN2> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"

Tuesday June 23 11:37:23 UTC 2015

AENKG   6               209.165.200.246 129 20450           4          17

AESKG   6               209.165.200.225 129 94021           3          54

[local]GGSN3> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"

Tuesday June 23 11:37:26 UTC 2015

AENKG   6               209.165.200.246 132 20211           1          39

AESKG   6               209.165.200.225 132 91996           3          59

[local]GGSN4> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"

Tuesday June 23 11:37:25 UTC 2015

AENKG   6               209.165.200.246 129 20309           1          24

AESKG   6               209.165.200.225 132 92694           9          57

[local]GGSN3> show egtpc peers | grep -E "209.165.200.246|209.165.200.225"

Tuesday June 23 11:44:35 UTC 2015

AENKG   6                209.165.200.246 129 20242           3          39

AESKG   6                209.165.200.225 132 92224           1          59

Finally to confirm the issue is the SGSN changing the RC, monitor subscriber menu option by SGSN peer shows changing RCs. When it does change,  all calls would be disconnected with path-failure as the disconnect reason. A screenshot of a packet capture also shows this:

[XGWin]GGSN> mon sub

Monday June 22 23:34:22 UTC 2015

y) By SGSN IP Address

IP Address: [209.165.200.225   ]

INBOUND>>>>>  23:35:32:688 Eventid:47000(3)

GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)

TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)

Sequence Number:: 0x1DB8 (7608)

...

IMSI: 300420078559902

Recovery: 0x81 (129)

...

***CONTROL*** 23:36:00:363 Eventid:10285

Disconnect Reason: path-failure

INBOUND>>>>>  23:36:48:414 Eventid:47000(3)

GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)

TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)

Sequence Number:: 0x3E2D (15917)

...

IMSI: 300420125984926

Recovery: 0x84 (132)

...

INBOUND>>>>>  23:37:28:337 Eventid:47000(3)

GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)

TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)

Sequence Number:: 0x3517 (13591)

...

IMSI: 300420094205377

Recovery: 0x84 (132)

...

Monday June 22 2015

INBOUND>>>>>  23:37:40:559 Eventid:47000(3)

GTPC Rx PDU, from 209.165.200.225:2123 to 209.165.202.129:2123 (176)

TEID: 0x00000000, Message type: GTP_CREATE_PDP_CONTEXT_REQ_MSG (0x10)

Sequence Number:: 0x4E47 (20039)

...

IMSI: 300420194755472

Recovery: 0x81 (129)

...

***CONTROL*** 23:37:40:755 Eventid:10285

Disconnect Reason: path-failure

This example shows a path failure where the problem was actually in the path and not the PGW or SGW. As a result, the RC remains the same when path re-establishes. In the second peer example the clear is a result of echo request establishment:

Thu May 28 20:47:47 2015 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.196, peer old restart counter 5,  peer new restart counter 5,  peer session count  549697, failure reason no-response-from-peer

Thu May 28 20:49:36 2015 Internal trap notification 1113 (EGTPCPathFailClear) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 198.51.100.196, peer restart counter 5,   peer session count  1, clear reason sessions-add

Thu May 28 20:49:06 2015 Internal trap notification 1112 (EGTPCPathFail) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 203.0.113.98, peer old restart counter 4,  peer new restart counter 4,  peer session count  3, failure reason no-response-from-peer

Thu May 28 20:50:49 2015 Internal trap notification 1113 (EGTPCPathFailClear) context PGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 203.0.113.98, peer restart counter 4,   peer session count  0, clear reason echo-req

This example shows another anomaly where repeatedly after the old restart counter has reset to 2, it somehow ends up at 0 again when the path fail clears. This turned out to be a bug. Again the lesson here is if something appears odd, it should be reported.

Fri Mar 27 03:13:36 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer old restart counter 0,  peer new restart counter 2,  peer session count  1, failure reason create-sess-restart-counter-change

Fri Mar 27 03:13:37 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer restart counter 0,   peer session count  1, failure reason sessions-add

Sat Mar 28 02:20:45 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer old restart counter 0,  peer new restart counter 2,  peer session count  1, failure reason create-sess-restart-counter-change

Sat Mar 28 02:20:47 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer restart counter 0,   peer session count  1, failure reason sessions-add

Sat Mar 28 17:14:06 2015 Internal trap notification 1112 (EGTPCPathFail) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer old restart counter 0,  peer new restart counter 2,  peer session count  1, failure reason create-sess-restart-counter-change

Sat Mar 28 17:14:06 2015 Internal trap notification 1113 (EGTPCPathFailClear) context XGWin, service EGTP1, interface type pgw-ingress, self address 209.165.201.1,  peer address 192.0.2.5, peer restart counter 0,   peer session count  1, failure reason sessions-add