Enhanced Application-Aware Routing

Table 1. Feature History

Feature Name

Release Information

Description

Enhanced Application-Aware Routing

Cisco IOS XE Catalyst SD-WAN Release 17.12.1a and later.

Cisco Catalyst SD-WAN Manager Release 20.12.1 and later.

Without enhanced application-aware routing enabled, Cisco IOS XE Catalyst SD-WAN devices require several minutes to switch traffic from one network path to another to meet SLA requirements when the loss, latency, and jitter exceed specific threshold values.

Enabling enhanced application-aware routing speeds the detection of tunnel performance issues. This enables Cisco IOS XE Catalyst SD-WAN devices to redirect traffic away from tunnels that do not meet SLA requirements.

Information About Enhanced Application-Aware Routing

Without enhanced application-aware routing enabled, Cisco IOS XE Catalyst SD-WAN devicerequire several minutes to switch traffic from one network path to another to meet SLA requirements when the loss, latency, and jitter exceed specific threshold values. Enabling enhanced application-aware routing speeds the detection of tunnel performance issues. This enables Cisco IOS XE Catalyst SD-WAN device to redirect traffic away from tunnels that do not meet SLA requirements.

Overview of Enhanced Application-Aware Routing

BFD (Bidirectional Forwarding Detection) detects link failure conditions and gathers performance routing data (PfR), including loss, latency, and jitter information of Cisco Catalyst SD-WAN tunnels (both IPsec and GRE). Each BFD hello packet collects the following information:

Latency: RTT (Round trip time) between BFD echo request and reply.

Jitter: The variation in the delay of packet arrival times in a network. It is a measure of the irregularity in the timing of data packets as they are transmitted and received.

Loss: Number of echo requests that fail to receive a reply.

By default, with a BFD hello timer of 1 second, one sample of PfR data is collected every second. This PfR data is collected over the duration of the poll interval (default 10 minutes). During the poll interval, the average of each statistic is computed. To determine dynamic path decisions based on the thresholds specified in application-aware routing SLAs, a default multiplier of 6 is employed to review multiple averages of the poll-interval. A poll interval average refers to the average time duration between consecutive polling or measurement events in a network monitoring or performance measurement system. The poll interval average provides an indication of how frequently the system collects data or samples network metrics over a specific time-period.

Convergence time refers to the amount of time it takes for the network to recover and resume normal operations after a failure or disruption. However, the default convergence time for detection of slowly degrading WAN circuits is between 10 minutes and 1 hour. Even with the lowest recommended poll-interval of 2 minutes and 6 intervals, the convergence time is between 2 minutes and 12 minutes. Setting a very low poll interval can result in false positives of PfR and traffic instability due to insufficient sample data for loss, latency, and jitter measurements.

PfR Measurements

Table 2. PfR Measurements

Metric

Source

Description

Loss

BFD

Measured as loss of BFD packet at 1pps or one packet in n_app_probe_class (n-apc) sec​

If the application probe class (APC) configuration is not set, the loss of BFD packets occurs at a rate of 1 packet per second (1pps). With the APC configuration, the loss is reduced to 1 packet in N seconds.

For more information see, Application Probe Class.

Latency

BFD

RTT measurements 1 pps or one packet in n-apc sec​

Without the application probe class (APC) configuration, the loss of RTT packets occurs at a rate of 1 packet per second (1pps). With the APC configuration, the loss is reduced to 1 packet in N seconds.

Jitter

BFD

Variation in RTT​

Application-Aware Routing Design and Measurements

  • The default BFD hello-interval is 1 sec, and the app-route/SLA poll-interval is 10 mins:

    The BFD hello-interval refers to the frequency at which BFD (Bidirectional Forwarding Detection) protocol sends hello packets to detect the liveliness of a network path. By default, the hello-interval is set to 1 second. On the other hand, the app-route/SLA poll-interval determines how frequently the network monitoring system collects data or measures network metrics related to application routes or Service Level Agreements (SLAs). The default poll-interval for app-route/SLA is set to 10 minutes.

  • By default, the system calculates to 60 minutes using 1 pps x 600 sec x 6 buckets:

    Refers to the calculation of a default value for the poll-interval in minutes. It calculates the interval by multiplying 1 packet per second (pps) by 600 seconds (10 minutes) and then multiplying the result by 6 buckets. The resulting value is 60 minutes, which is the default poll-interval.

  • Experts suggest using a poll-interval of 120 seconds (2 minutes) and a multiplier of 5, which results in a 10-minute interval. This recommendation is often followed to achieve a specific monitoring frequency.

  • Reducing the poll-interval/multiplier helps improve detection time but may lead to false positives with a small number of samples for PfR metrics:

    Decreasing the poll-interval and/or the multiplier can enhance the speed at which network performance issues are detected. However, reducing these values may also increase the likelihood of false positives, which is that the system may incorrectly identify issues due to a small number of data samples. The detection time and the accuracy of PfR (Performance Routing) metrics must be balanced.

  • The only option is to improve the measurement accuracy at a faster rate by reducing the BFD Hello interval:

    To achieve a faster and more accurate measurement of network performance, the recommended approach is to decrease the BFD hello-interval. Network path liveliness refers to the condition of the connectivity and availability of network paths. By reducing the interval at which hello packets are exchanged, the liveliness of network paths can be detected more frequently, leading to improved measurement accuracy.

Benefits of Enhanced Application-Aware Routing

  1. Improved the PfR metrics (loss/latency/jitter) measurements by introducing inline data that allows for more accurate and detailed measurements of these metrics. Inline data refers to the traffic that is processed and inspected directly at the edge of the network, within the Cisco IOS XE Catalyst SD-WAN devices. Instead of routing all the traffic to a central location for analysis and security checks, inline data allows for real-time inspection and decision-making at the network edge.

  2. Quick Enhanced-App-Route Detection and SLA Enforcement, which involves reducing the PfR poll-interval to a very low value (minimum of 10 seconds). This allows the Cisco IOS XE Catalyst SD-WAN devices to quickly detect any slow degradation of circuits. If a circuit fails to meet the SLA threshold, the tunnels are swiftly switched out from SLA forwarding to ensure efficient and reliable network performance. SLA (Service Level Agreement) forwarding refers to the capability of the Cisco Catalyst SD-WAN solution to dynamically route network traffic based on predefined performance criteria or SLAs.

  3. The speed of SLA switch-over is improved.

  4. SLA Dampening is introduced for a smoother transition to SLA forwarding. Before implementing SLA forwarding again, the tunnel goes through a process called dampening, which helps prevent disruptions and instabilities. This ensures a smooth transition back to SLA, minimizing any negative effects on network performance.

  5. Enhancements are made to measure loss, latency, and jitter.

Guidelines of Enhanced Application-Aware Routing

  • Both GRE and IPSEC tunnels are supported.

  • All existing TLOCs and WAN interface types, including physical, sub interface, loopback bind, dialer, and LTE interfaces, are supported.

  • TLOC Extension tunnels are supported.

  • Both IPv4 and IPv6 underlay tunnels are supported.

  • SLA update and switchover occur at a minimum interval of 10 seconds.

  • Tunnel scale is not impacted, with minimal impact on memory and performance.

  • Support is provided with and without app-probe class configuration in SLA classes.

  • SLA dampening is supported.

Compatibility With Cisco IOS XE Catalyst SD-WAN devices Not Running Enhanced Application-Aware Routing

  1. In the following scenario:

    • On the local side: The Cisco IOS XE Catalyst SD-WAN device is upgraded to Cisco IOS XE Catalyst SD-WAN Release 17.12.1a and later and has EAAR (Enhanced Application-Aware Routing) enabled.

    • On the remote side: The Cisco IOS XE Catalyst SD-WAN device is not upgraded to Cisco IOS XE Catalyst SD-WAN Release 17.12.1a and the EAAR is not enabled.

    Then the system will fall back to using BFD based measurements where support compatibility with older releases and disabled features are present.

  2. If both the local and remote sides are using Cisco IOS XE Catalyst SD-WAN Release 17.12.1a but the EAAR feature is not enabled, the system will revert to using BFD based measurements.


Note


The EAAR feature is disabled by default to support existing deployments.


Supported Devices for Enhanced Application-Aware Routing

Cisco IOS XE Catalyst SD-WAN devices

Restrictions for Enhanced Application-Aware Routing

  • The branch device on which you enable this feature does not support loopback unbind mode. The loopback unbind mode refers to a network interface configuration in which the loopback device is disconnected from the network stack.

  • There is no per-queue measurement for GRE tunnels. Per queue measurement is used to monitor and analyze network traffic on a per-queue basis. It involves measuring and collecting various metrics and statistics for each individual queue in a network device or system. A queue is a buffer where packets are stored before they are transmitted or processed.

Prerequisites for Enhanced Application-Aware Routing

To enable application-aware routing on a Cisco IOS XE Catalyst SD-WAN device, enable enhanced application-aware routing on both the Cisco IOS XE Catalyst SD-WAN devices.

Configure Enhanced Application-Aware Routing

The procedures in this section describe how to deploy the enhanced app-aware routing configurations fromCisco Catalyst SD-WAN Manager to Cisco IOS XE Catalyst SD-WAN devices.

Configure Enhanced Application-Aware Routing Using a Feature Template in Cisco Catalyst SD-WAN Manager

  1. From the Cisco SD-WAN Manager menu, choose Configuration > Templates.

  2. Click Feature Templates.

  3. Click Add Template.

  4. Choose a device and click the Cisco System template under Basic Information.

  5. In the Enhanced App-Aware Routing field, click Global from the drop-down list and choose one of the following modes:

    Mode

    EAAR Poll Interval

    EAAR Poll Multiplier

    EAAR Poll Window

    SLA Dampening Multiplier

    SLA Dampening Window

    Aggressive

    10s

    6

    10s - 60s

    120

    20 mins

    Moderate

    60s

    5

    60s - 300s

    40

    40 mins

    Conservative

    300s

    6

    300s - 1800s

    12

    60 mins


    Note


    You can configure the enhanced application aware routing (EAAR) poll interval, poll multiplier, and SLA dampening multiplier only through CLI template.


  6. Click Save.

Configure Enhanced Application-Aware Routing Using a Configuration Group in Cisco Catalyst SD-WAN Manager

  1. From the Cisco SD-WAN Manager menu, choose Configuration > Configuration Groups.

  2. Choose a configuration group. Under Actions click Edit.

  3. Under Feature Profiles click System Profile.

  4. Choose basic and under Actions click Edit Feature.

  5. In the Edit Basic Feature page, use the Enhanced App-Route field and choose one of the modes as follows:

    Mode

    EAAR Poll Interval

    EAAR Poll Multiplier

    EAAR Poll Window

    SLA Dampening Multiplier

    SLA Dampening Window

    Aggressive

    10s

    6

    10s - 60s

    120

    20 mins

    Moderate

    60s

    5

    60s - 300s

    40

    40 mins

    Conservative

    300s

    6

    300s - 1800s

    12

    60 mins

  6. Click Save.

Configure Enhanced Application-Aware Routing Using a CLI Template

For more information about using CLI templates, see CLI Add-On Feature Templates and CLI Templates. By default, CLI templates execute commands in global configuration mode.

  1. Enable enhanced PfR measurements for SLA enforcement.

    bfd enhanced-app-route enable

    Enabling the application-aware routing feature on a Cisco IOS XE Catalyst SD-WAN device requires you to enable the PfR CLI on both the remote Cisco IOS XE Catalyst SD-WAN device and the local Cisco IOS XE Catalyst SD-WAN device.

    This feature involves two steps:

    1. The remote Cisco IOS XE Catalyst SD-WAN device must provide loss statistics to the local Cisco IOS XE Catalyst SD-WAN device.

    2. The local Cisco IOS XE Catalyst SD-WAN device then utilizes these metrics to enforce Service Level Agreements (SLAs).

  2. When the enhanced application aware PfR is enabled, the default poll-interval of 10 seconds and multiplier of 6 is used for SLA enforcement and switchover. To modify these settings, use the following configuration options:

    bfd enhanced-app-route pfr-poll-interval

    bfd enhanced-app-route pfr-multiplier <number>

    The aggressive mode setting for app route pfr multiplier is 6 by default. It is 5 for the moderate mode.

  3. Configure the SLA dampening time. This is the waiting time before returning the tunnel to SLA buckets after meeting the SLA. The default time is 120 seconds. Enable the SLA dampening when the enhanced PfR is enabled.

    bfd sla-dampening enable

    bfd sla-dampening multiplier <number>

    The aggressive mode setting for dampening multiplier is 120 by default.

Verify the Enhanced Application-Aware Routing Configuration

To verify the enhanced application-routing configuration and display the configured params for EAAR use the show sdwan app-route params command.

Device# show sdwan app-route params 
*EAAR = Enhanced Application-Aware Routing
Config:                  :Enabled   
Poll interval:           :10000     
Poll multiplier:         :6         

App route 
Poll interval:           :600000    
Poll multiplier:         :6         

SLA dampening  
Config:                  :Enabled   
Multiplier:              :120   

You can use the show sdwan bfd sessions alt command to highlight the flags for EAAR.

Device# show sdwan bfd sessions alt 
*Sus = Suspend
*GREinUDP = GREinUDP encap
*EAAR  = Enhanced Application-Aware Routing
*NA  = Flag Not Set
 
                                                      SOURCE TLOC      REMOTE TLOC                                  DST PUBLIC     DST PUBLIC                                               
SYSTEM IP        SITE ID   STATE  COLOR                  COLOR                   SOURCE IP         IP                      PORT        ENCAP  BFD-LD    FLAGS       UPTIME          
-------------------------------------------------------------------------------------------------------------------------------------------------------------
172.16.0.0    100       up          lte                           lte                           10.0.0.0            10.0.0.1         12367       ipsec      20013     NA            0:07:48:38      
172.16.0.1    100       up          lte                           lte                           10.0.0.0            10.0.0.1         12377       ipsec      20014     NA            0:07:48:39      
172.16.0.0    400       up          lte                           lte                           10.0.0.0            10.0.0.1         12366       ipsec      20015     NA            0:07:48:39      
172.16.0.1    500       up          lte                           lte                           10.0.0.0            10.0.0.1         12366       ipsec      20016     EAAR          0:07:48:39

You can use show sdwan app-route stats summary command to display the app-route (PfR) stats details for each tunnel, across different intervals of measurements, for every configured APC.

Device# show sdwan app-route stats summary   
app-route statistics 10.0.0.0 10.0.0.0 ipsec 12366 12367
 remote-system-ip         172.16.0.0
 local-color              lte
 remote-color             lte
 sla-class-index          0,1,2,3
 fallback-sla-class-index None
 enhanced-app-route       Enabled
 sla-dampening-index      4,5
 app-probe-class-list None
  mean-loss    0
  mean-latency 0
  mean-jitter  0
              TOTAL                      AVERAGE  AVERAGE     TX DATA     RX DATA    IPV6 TX        IPV6 RX
INDEX   PACKETS  LOSS     LATENCY  JITTER            PKTS          PKTS          DATA PKTS  DATA PKTS
--------------------------------------------------------------------------------------------------------------------------------------------------
0           664              0             0                0                       0                 0                 0                       0
1           663              0             0                0                       0                 0                 0                       0
2           666              0             0                0                       0                 0                 0                       0
3           664              0             0                0                       0                 0                 0                       0
4           662              0             0                0                       0                 0                 0                       0
5           664              0             0                0                       0                 0                 0                       0

Monitor Enhanced Application-Aware Routing Using Cisco Catalyst SD-WAN Manager

  1. From the Cisco Catalyst SD-WAN Manager menu, choose Monitor >Devices.

  2. Under Devices, choose a device.

  3. Click Real Time in the left pane.

  4. In the Device Options field, choose App Routes Statistics.

Troubleshooting Enhanced Application-Aware Routing

From the device:

Device# show sdwan run | include enhanced-app-route
bfd enhanced-app-route enable
bfd enhanced-app-route pfr-poll-interval 10000
bfd enhanced-app-route pfr-multiplier 6

show sdwan run | inc sla-dampening
bfd sla-dampening enable
bfd sla-dampening multiplier 12
Device# show sdwan app-route params
Enhanced app route
  Config:                  :Enabled <<< Enhanced app-aware routing enabled 
    Poll interval:           :10000
    Poll multiplier:         :6
App route
    Poll interval:           :600000
    Poll multiplier:         :6
SLA dampening
    Config:                  :Enabled
    Multiplier:              :120
Device# show platform hardware qfp active feature sdwan datapath pathmon summary 
Src IP          Dst IP          Src Port Dst Port   Encap    Uidb     Bfd Discrim PathMon
------          ------          -------- -------    ------   -------  ----------- ------
10.0.0.0      10.0.0.1      12346    12366      IPSEC    65527    20003       in/out
Device# show sdwan bfd sessions alt
*Sus = Suspend
*GREinUDP = GREinUDP encap
*EAAR  = Enhanced Application-Aware Routing
*NA  = Flag Not Set
                                       SOURCE TLOC      REMOTE TLOC                                      DST PUBLIC                      DST PUBLIC
SYSTEM IP        SITE ID   STATE       COLOR            COLOR            SOURCE IP                       IP                              PORT        ENCAP  BFD-LD    FLAGS             UPTIME
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
172.16.0.0        100       down        private1         lte              10.0.0.0                      10.0.0.1                      12367       ipsec  20011     EAAR                NA
172.16.0.1        500       down        private1         3g               10.0.0.0                      10.0.0.1                      12366       ipsec  20013     EAAR                NA
172.16.0.0        600       down        private1         3g               10.0.0.0                      10.0.0.1                      12366       ipsec  20007     EAAR                NA
Device# show sdwan app-route stats remote-system-ip 172.16.0.0 app-route statistics 10.0.0.0 10.0.0.1 ipsec 12366 12366
remote-system-ip         172.16.0.0        
local-color              private1
remote-color             3g
sla-class-index          0
fallback-sla-class-index None
enhanced-app-route       Enabled
sla-dampening-index      None