IOS Per VRF TACACS+ Troubleshooting

Introduction

TACACS+ is heavily used as the authentication protocol to authenticate users to network devices. More and more administrators are segregating their management traffic using VPN Routing and Forwarding (VRFs). By default, AAA on IOS uses the default routing table to send packets. This document describes how to configure and troubleshoot TACACS+ when the server is in a VRF.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• TACACS+

• VRFs

Components Used

This document is not restricted to specific software and hardware versions.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Feature Information

Essentially a VRF is a virtual routing table on the device. When IOS makes a routing decision if the feature or interface is using a VRF, routing decisions are made against that VRF routing table. Otherwise, the feature uses the global routing table. With this in mind, here is how you configure TACACS+ to use a VRF (relevant configuration in bold):

version 15.2
service config
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname vrfAAA
!
boot-start-marker
boot-end-marker
!
aaa new-model
!
aaa group server tacacs+ management
 server-private 192.0.2.4 key cisco
 server-private 192.0.2.5 key cisco
 ip vrf forwarding blue
 ip tacacs source-interface GigabitEthernet0/0
!
aaa authentication login default group management local
aaa authorization exec default group management if-authenticated 
aaa accounting exec default start-stop group management
!
aaa session-id common
!
no ipv6 cef
!
ip vrf blue
!
no ip domain lookup
ip cef
!
interface GigabitEthernet0/0
 ip vrf forwarding blue
 ip address 203.0.113.2 255.255.255.0
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 no ip address
 shutdown
 duplex auto
 speed auto
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip route vrf blue 0.0.0.0 0.0.0.0 203.0.113.1
!
line con 0
line aux 0
line vty 0 4
 transport input all

As you can see, there are no globally defined TACACS+ servers. If you are migrating the servers to a VRF, you can safely remove the globally configured TACACS+ servers.

Troubleshooting Methodology

1. Make sure you have the proper ip vrf forwarding definition under your aaa group server as well as the source interface for the TACACS+ traffic.

2. Check your vrf routing table and make sure there is a route to your TACACS+ server. The example above is used to display the vrf routing table:

   vrfAAA#show ip route vrf blue
   
   Routing Table: blue
   Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
          D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
          N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
          E1 - OSPF external type 1, E2 - OSPF external type 2
          i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
          ia - IS-IS inter area, * - candidate default, U - per-user static route
          o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
          + - replicated route, % - next hop override
   
   Gateway of last resort is 203.0.113.1 to network 0.0.0.0
   
   S*    0.0.0.0/0 [1/0] via 203.0.113.1
         203.0.0.0/24 is variably subnetted, 2 subnets, 2 masks
   C        203.0.113.0/24 is directly connected, GigabitEthernet0/0
   L        203.0.113.2/32 is directly connected, GigabitEthernet0/0

3. Can you ping your TACACS+ server? Remember this needs to be VRF specific as well:

   vrfAAA#ping vrf blue 192.0.2.4
   Type escape sequence to abort.
   Sending 5, 100-byte ICMP Echos to 102.0.2.4, timeout is 2 seconds:
   !!!!!
   Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

4. You can use the test aaa command to verify connectivity (you must use the new-code option at the end, legacy does not work):

   vrfAAA#test aaa group management cisco Cisco123 new-code 
   Sending password
   User successfully authenticated
   
   USER ATTRIBUTES
   
   username             "cisco"
   reply-message        "password: "

If the routes are in place and you see no hits on your TACACS+ server, make sure that the ACLs are allowing TCP port 49 to reach the server from the router or switch. If you get an authentication failure troubleshoot TACACS+ as normal, the VRF feature is just for the routing of the packet.

Data Analysis

If everything above looks correct, aaa and tacacs debugs can be enabled to troubleshoot the issue. Start with these debugs:

• debug tacacs

• debug aaa authentication

Here is an example of a debug where something is not configured properly, such as but no limited to:

• Missing TACACS+ source interface

• Missing ip vrf forwarding commands under the source interface or under the aaa group server

• No route to the TACACS+ server in the VRF routing table

Jul 30 20:23:16.399: TPLUS: Queuing AAA Authentication request 0 for processing
Jul 30 20:23:16.399: TPLUS: processing authentication start request id 0
Jul 30 20:23:16.399: TPLUS: Authentication start packet created for 0(cisco)
Jul 30 20:23:16.399: TPLUS: Using server 192.0.2.4
Jul 30 20:23:16.399: TPLUS(00000000)/0: Connect Error No route to host
Jul 30 20:23:16.399: TPLUS: Choosing next server 192.0.2.5
Jul 30 20:23:16.399: TPLUS(00000000)/0: Connect Error No route to host

Here is a successful connection:

Jul 30 20:54:29.091: AAA/AUTHEN/LOGIN (00000000): Pick method list 'default' 
Jul 30 20:54:29.091: TPLUS: Queuing AAA Authentication request 0 for processing
Jul 30 20:54:29.091: TPLUS: processing authentication start request id 0
Jul 30 20:54:29.091: TPLUS: Authentication start packet created for 0(cisco)
Jul 30 20:54:29.091: TPLUS: Using server 192.0.2.4
Jul 30 20:54:29.091: TPLUS(00000000)/0/NB_WAIT/2B2DC1AC: Started 5 sec timeout
Jul 30 20:54:29.095: TPLUS(00000000)/0/NB_WAIT: socket event 2
Jul 30 20:54:29.095: TPLUS(00000000)/0/NB_WAIT: wrote entire 25 bytes request
Jul 30 20:54:29.095: TPLUS(00000000)/0/READ: socket event 1
Jul 30 20:54:29.095: TPLUS(00000000)/0/READ: Would block while reading
Jul 30 20:54:29.099: TPLUS(00000000)/0/READ: socket event 1
Jul 30 20:54:29.099: TPLUS(00000000)/0/READ: read entire 12 header bytes (expect 16 bytes data)
Jul 30 20:54:29.099: TPLUS(00000000)/0/READ: socket event 1
Jul 30 20:54:29.099: TPLUS(00000000)/0/READ: read entire 28 bytes response
Jul 30 20:54:29.099: TPLUS(00000000)/0/2B2DC1AC: Processing the reply packet
Jul 30 20:54:29.099: TPLUS: Received authen response status GET_PASSWORD (8)
Jul 30 20:54:29.099: TPLUS: Queuing AAA Authentication request 0 for processing
Jul 30 20:54:29.099: TPLUS: processing authentication continue request id 0
Jul 30 20:54:29.099: TPLUS: Authentication continue packet generated for 0
Jul 30 20:54:29.099: TPLUS(00000000)/0/WRITE/2B2DC1AC: Started 5 sec timeout
Jul 30 20:54:29.099: TPLUS(00000000)/0/WRITE: wrote entire 25 bytes request
Jul 30 20:54:29.103: TPLUS(00000000)/0/READ: socket event 1
Jul 30 20:54:29.103: TPLUS(00000000)/0/READ: read entire 12 header bytes (expect 6 bytes data)
Jul 30 20:54:29.103: TPLUS(00000000)/0/READ: socket event 1
Jul 30 20:54:29.103: TPLUS(00000000)/0/READ: read entire 18 bytes response
Jul 30 20:54:29.103: TPLUS(00000000)/0/2B2DC1AC: Processing the reply packet
Jul 30 20:54:29.103: TPLUS: Received authen response status PASS (2)

Common Problems

The most common problem is the configuration. Many times the admin puts in the aaa group server, but does not update the aaa lines to point to the server group. Instead of:

aaa authentication login default group management local
aaa authorization exec default group management if-authenticated 
aaa accounting exec default start-stop group management

The admin will have put in:

aaa authentication login default grout tacacs+ local
aaa authorization exec default group tacacs+ if-authenticated 
aaa accounting exec default start-stop group tacacs+

Simply update the configuration with the correct server group.

A second common problem is a user receives this error when trying to add ip vrf forwarding under the server group:

% Unknown command or computer name, or unable to find computer address

This means the command was not found. If this occurs make sure the version of IOS supports per-VRF TACACS+. Here are some common minimum versions:

• 12.3(7)T

• 12.2(33)SRA1

• 12.2(33)SXI

• 12.2(33)SXH4

• 12.2(54)SG

Related Information

• Technical Support & Documentation - Cisco Systems