Troubleshoot Address ACI Fault Code F046

Introduction

This document describes next steps for remediation of ACI fault F0467: invalid-vlan, invalid-path or encap-already-in-use.

Background Information

ACI Fault F0467 is flagged in different scenarios, but shows a distinct cause for each one.

The most common cause values seen with ACI Fault F0467 are:

• invalid-vlan
• invalid-path
• encap-already-in-use

All causes of ACI Fault F0467 can affect the deployment of vlans on the switch node interfaces.

Intersight Connected ACI Fabrics

This fault is actively monitored as part of Proactive ACI Engagements.

If you have an Intersight connected ACI fabric, a Service Request was generated on your behalf to indicate that instances of this fault were found within your Intersight Connected ACI fabric.

ACI Fault F0467 Scenarios

Invalid VLAN Configuration invalid-vlan

Scenario

• New EPG configured with encap VLAN 421
• Physical  domain assigned to EPG
• Static port binding for VLAN 421 on EPG
• Fault F0467 - flagged against the switch node with a pointer to the EPG
• Fault debug message contains invalid-vlan:vlan-x :Either the EpG is not associated with a domain or the domain does not have this vlan assigned to it.

The fault description explicitly states "Either the EpG is not associated with a domain or the domain does not have this vlan assigned to it".

APIC# moquery -c faultInst -f 'fault.Inst.code=="F0467"' | grep lc_EPG
descr : Configuration failed for uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG node 103 eth1/13 due to Invalid VLAN Configuration, debug message: invalid-vlan: vlan-421 :Either the EpG is not associated with a domain or the domain does not have this vlan assigned to it;
dn : topology/pod-1/node-103/local/svc-policyelem-id-0/uni/epp/fv-[uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG]/node-103/stpathatt-[eth1/13]/nwissues/fault-F0467

Potential Cause: Associated VLAN Pool Does Not Contain the Required VLAN

Access Encap VLAN 421 is not deployed on the leaf node:

Node-103# show vlan encap-id 421 extended 
<<< Empty >>>   

The Static path to EPG association is not created:

APIC# moquery -c l2RtDomIfConn | grep lc_EPG | grep dn 
<<< Empty >>>  

Domain lc_phys_dom is associated it to the lc_EPG EPG:

APIC# moquery -c fvRsDomAtt | grep -A 25 lc_EPG | grep rn
rn : rsdomAtt-[uni/phys-lc_phys_dom]

Domain to Vlan Pool association  exists:

APIC# moquery -c infraRsVlanNs | grep -A 15 lc_phys_dom | grep tDn
tDn : uni/infra/vlanns-[lc_vlan_pool]-static

Vlan Pool lc_vlan_pool is ranged to only include VLAN 420.

APIC# moquery -c fvnsEncapBlk | grep lc_vlan_pool 
dn : uni/infra/vlanns-[lc_vlan_pool]-static/from-[vlan-420]-to-[vlan-420]

vlan 421 is not in the previous pool, hence the error "invalid-vlan: vlan-421 :Either the EpG is not associated with a domain or the domain does not have this vlan assigned to it."

In the block diagram referenced earlier, this specific vlan pool reference is highlighted:

Add the missing vlan 421 to the specific vlan range.

Vlan Pool to Encap and Domain associations (Fabric > Access Policies > Pool > VLAN > lc_vlan_pool):

Vlan pool range verification after adding vlan 421:

APIC# moquery -c fvnsEncapBlk | grep lc_vlan_pool 
dn : uni/infra/vlanns-[lc_vlan_pool]-static/from-[vlan-420]-to-[vlan-420]
dn : uni/infra/vlanns-[lc_vlan_pool]-static/from-[vlan-421]-to-[vlan-421]

Potential Cause: VLAN Pool with required VLAN Not Associated to Domain  

Fabric > Access Policies > Physical and External Domains> Physical Domains > lc_phys_dom:

[+] Domain to Vlan Pool association:

APIC# moquery -c infraRsVlanNs | grep -A 15 lc_phys_dom | grep tDn
<< EMPTY >>

Fix: Include missing vlan association

Fabric > Access Policies > Physical and External Domains> Physical Domains > lc_phys_dom:

Invalid Path Configuration invalid-path

Scenario

• EPG configured
• Domain assigned to EPG
• Static port binding created on EPG for VLAN 420, node 103 eth 1/13
• Fault F0467 - flagged against the switch node with a pointer to the EPG
• Fault debug message contains invalid-path: Either the EpG/L3Out is not associated with a domain or the domain does not have this interface assigned to it.

This fault is raised when a switch/port/VLAN declaration is made without the corresponding access policies in place to allow that configuration to be applied properly.

Depending on the description of this fault, a different element of the access policy relationship is possibly missing.

EPG - lc_EPG to Fault association at Tenants > lc_TN > lc_AP > lc_EPG > Faults > Fault:

The affected EPG, switch node id, and port number are in the fault description and dn:

APIC# moquery -c faultInst -f 'fault.Inst.code=="F0467"' | grep lc_EPG
descr            : Configuration failed for uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG node 103 eth1/13 due to Invalid Path Configuration, debug message: invalid-path: Either the EpG/L3Out is not associated with a domain or the domain does not have this interface assigned to it;
dn               : topology/pod-1/node-103/local/svc-policyelem-id-0/uni/epp/fv-[uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG]/node-103/stpathatt-[eth1/13]/nwissues/fault-F0467

Quick Start Isolation

Confirm if the vlan is deployed. If not, these commands can be run to isolate the configuration error.

In these commands, lc_EPG is the EPG name used for output filtering. 

Encap-vlan is NOT deployed on the leaf node:

Node-103# show vlan encap-id 420 extended 
<<< Empty >>>   

[1] Static path to EPG association policy is empty:

APIC# moquery -c l2RtDomIfConn | grep lc_EPG | grep dn
<<< Empty >>>

[2] Domain to EPG association:

APIC# moquery -c fvRsDomAtt | grep -A 25 lc_EPG | grep rn
rn : rsdomAtt-[uni/phys-lc_phys_dom]

[3] Domain to Vlan Pool association:

APIC# moquery -c infraRsVlanNs | grep -A 15 lc_phys_dom | grep tDn
tDn : uni/infra/vlanns-[lc_vlan_pool]-static

[4] Vlan pool range verification:

APIC# moquery -c fvnsEncapBlk | grep lc_vlan_pool 
dn : uni/infra/vlanns-[lc_vlan_pool]-static/from-[vlan-420]-to-[vlan-420]

[5] Domain to AAEP association:

APIC# moquery -c infraRtDomP | grep lc_phys_dom
dn : uni/phys-lc_phys_dom/rtdomP-[uni/infra/attentp-lc_AAEP]

[6] AAEP to Interface Policy Group association(IPG):

rtp-aci08-apic1# moquery -c infraRtAttEntP | grep lc_AAEP
dn           : uni/infra/attentp-lc_AAEP/rtattEntP-[uni/infra/funcprof/accportgrp-lc_IPG]

[7]  IPG to Interface Selector association:

APIC# moquery -c infraRsAccBaseGrp | grep -B 15 lc_IPG | grep dn
dn : uni/infra/accportprof-lead103_IP/hports-lc_Interface_Selector-typ-range/rsaccBaseGrp

[8]  Interface Profile to Switch Profile association:

APIC# moquery -c infraRsAccPortP | grep leaf103_IP | grep dn
dn : uni/infra/nprof-leaf103_SP/rsaccPortP-[uni/infra/accportprof-leaf103_IP] 

The invalid-path cause is seen if any of the required Access Policy associated are missing given the static path configuration. Walk through the potential causes in this order to verify the Access Policies hop-by-hop:

1. Missing Domain to AAEP association
2. Missing AAEP to IPG association 
3. Missing IPG to Interface Selector association  
4. Missing Interface Selector to Interface Profile association 
5. Missing Interface Profile to Switch Profile association

Potential Cause: Missing Domain to AAEP Association 

Fabric > Access Policies > Policies > Global > AAEP > lc_AAEP:

[+] Static path to EPG association policy is empty:

APIC# moquery -c l2RtDomIfConn | grep lc_EPG | grep dn 
<< EMPTY >>

[+] Domain to AAEP association:

APIC# moquery -c infraRtDomP | grep lc_phys_dom
<< EMPTY >>

Fix: Vlan Pool to Encap and Domain associations (Fabric > Access Policies > Pool > VLAN > lc_vlan_pool)

Fabric > Access Policies > Physical and External Domains> Physical Domains > lc_phys_dom:

Potential Cause: Missing AAEP to IPG Association

IPG to AAEP Association

Fabric > Access Policies > Interfaces > Leaf Interfaces > Policy Groups > Leaf Access Port > lc_IPG:

[+] Static path to EPG association policy is empty:

APIC# moquery -c l2RtDomIfConn | grep lc_EPG | grep dn 
<< EMPTY >>

[+] IPG to AAEP association is empty:

APIC# moquery -c infraRsAttEntP | grep -A 15 lc_IPG | grep tDn
<< EMPTY >>

Fix: Missing AAEP to IPG association

Fabric > Access Policies > Interfaces > Leaf Interfaces > Policy Groups > Leaf Access Port > lc_IPG:

[+] IPG to AAEP association 

APIC# moquery -c infraRsAttEntP | grep -A 15 lc_IPG | grep tDn
tDn : uni/infra/attentp-lc_AAEP

Potential Cause: Missing IPG to Interface Selector Association

Interface Selector to Interface Policy Group Association

Fabric > Access Policies > Interfaces > Leaf Interfaces > Profiles > leaf103_IP > lc_Interface_Selector:

[+]  IPG to Interface Selector association

APIC# moquery -c infraRsAccBaseGrp | grep -B 15 lc_IPG | grep dn
<< EMPTY >>

Fix: Interface Selector to Interface Policy Group association

[+]  IPG to Interface Selector association:

APIC# moquery -c infraRsAccBaseGrp | grep -B 15 lc_IPG | grep dn
dn : uni/infra/accportprof-lead103_IP/hports-lc_Interface_Selector-typ-range/rsaccBaseGrp

Potential Cause: Missing Interface Selector to Interface Profile Association

Interface Profile to Interface Selector Association

Fabric > Access Policies > Interfaces > Leaf Interfaces > Profiles > leaf103_IP:

Troubleshooting:

APIC# moquery -c infraHPortS | grep leaf103_IP
<< EMPTY >>

Fix: Interface Profile to Interface Selector association

APIC# moquery -c infraHPortS | grep leaf103_IP
dn : uni/infra/accportprof-leaf103_IP/hports-lc_Interface_Selector-typ-range

Potential Cause: Missing Interface Profile to Switch Profile Association

Interface Profile to Switch Profile Association

Fabric > Access Policies > Switches > Leaf Switches > Profiles > leaf103_SP

APIC# moquery -c infraRsAccPortP | grep leaf103_IP | grep dn
<< EMPTY >>

Fix: Leaf Profile to Interface Selector Profiles association

[+]  Interface Profile to Switch Profile association:

APIC# moquery -c infraRsAccPortP | grep leaf103_IP | grep dn
dn : uni/infra/nprof-leaf103_SP/rsaccPortP-[uni/infra/accportprof-leaf103_IP]

Encap Already Used in Another EPG: encap-already-in-use

Scenario

By default, VLANs have a global scope. A given VLAN ID can only be used for a single EPG on a given leaf switch.

Any attempt to re-use the same VLAN on multiple EPGs within a given leaf switch results in an encap-already-in-use F0467 fault.

EPG to Fault association at Tenants > lc_TN > lc_AP > lc_EPG > Faults > Fault:

APIC# moquery -c faultInst -f 'fault.Inst.code=="F0467"' | grep lc_EPG
changeSet : configQual:encap-already-in-use, configSt:failed-to-apply, debugMessage:encap-already-in-use: Encap (vlan-420) is already in use by lc_TN_Dup:lc_APP:lc_EPG;, temporaryError:no
descr : Configuration failed for uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG node 103 eth1/13 due to Encap Already Used in Another EPG, debug message: encap-already-in-use: Encap (vlan-420) is already in use by lc_TN_Dup:lc_APP:lc_EPG;
dn : topology/pod-1/node-103/local/svc-policyelem-id-0/uni/epp/fv-[uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG]/node-103/stpathatt-[eth1/13]/nwissues/fault-F0467

Quick Start Isolation

[+] You can confirm the encap already in use on a different tenant lc_TN_Dup:

Node-103# show vlan extended | egrep "Encap|----|vlan-420"
 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 3    lc_TN_Dup:lc_APP:lc_EPG          vlan-420         Eth1/13

Remediation Options

• Option 1:Use a different vlan number no in use on the leaf or VPC pair.
• Option 2: Use the same vlan on a different leaf or VPC pair that does not have Vlan trying to be deployed.
• Option 3: Remove static port association on duplicated EPG , this allows the new deployment.
• Option 4: In ACI versions prior to the v1.1 release, a given VLAN encapsulation maps to only a single EPG on a leaf switch. If there is a second EPG which has the same VLAN encapsulation on the same leaf switch, the ACI raises this fault.

Starting with the v1.1 release, you can deploy multiple EPGs with the same VLAN encapsulation on a given leaf switch (or FEX), in the Per Port VLAN configuration  guide:

• https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/2-x/L2_config/b_Cisco_APIC_Layer_2_Configuration_Guide/b_Cisco_APIC_Layer_2_Configuration_Guide_chapter_011.html#concept_BC396E1CBB7D4687A9CBBECDDD43DE11

Additional Details

Successful Configuration Reference

This section can be used as a reference guide for what a complete configuration with a functional setup looks like.

EPG to Static Path Association

Tenants > lc_TN > lc_AP > lc_EPG > Static Ports:

[+] Static port to EPG association policy:

APIC# moquery -c l2RtDomIfConn | grep lc_EPG | grep dn 
dn : topology/pod-1/node-103/sys/ctx-[vxlan-2195458]/bd-[vxlan-16416666]/vlan-[vlan-420]/rtfvDomIfConn-[uni/epp/fv-[uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG]/node-103/stpathatt-[eth1/13]/conndef/conn-[vlan-420]-[0.0.0.0]]

EPG to AAEP Association

Fabric > Access Policies > Policies > Global > AAEP > lc_AAEP:

APIC# moquery -c fvIfConn -f 'fv.IfConn.encap=="vlan-420"' | grep dn 
dn : uni/epp/fv-[uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG]/node-103/attEntitypathatt-[lc_AAEP]/conndef/conn-[vlan-420]-[0.0.0.0]

EPG to Domain Association

Tenants > lc_TN > lc_AP > lc_EPG > Domains:

[+] Domain lc_phys_dom has been associated it to EPG.

APIC# moquery -c fvRsDomAtt | grep -A 25 lc_EPG | grep rn
rn : rsdomAtt-[uni/phys-lc_phys_dom]

Domain to AAEP and Vlan Pool Associations

Fabric > Access Policies > Physical and External Domains> Physical Domains > lc_phys_dom:

[+] Domain to AAEP association:

APIC# moquery -c infraRtDomP | grep lc_phys_dom
dn : uni/phys-lc_phys_dom/rtdomP-[uni/infra/attentp-lc_AAEP]

[+] Domain to Vlan Pool association 

APIC# moquery -c infraRsVlanNs | grep -A 15 lc_phys_dom | grep tDn
tDn : uni/infra/vlanns-[lc_vlan_pool]-static

Vlan Pool to Encap Block and Domain Associations

Fabric > Access Policies > Pool > VLAN > lc_vlan_pool:

[+] Vlan pool range verification:

APIC# moquery -c fvnsEncapBlk | grep lc_vlan_pool 
dn : uni/infra/vlanns-[lc_vlan_pool]-static/from-[vlan-420]-to-[vlan-420]

[+] Domains where the lc_vlan_pool is been used:

APIC# moquery -c fvnsRtVlanNs | grep lc_vlan_pool
dn           : uni/infra/vlanns-[lc_pool]-dynamic/rtinfraVlanNs-[uni/phys-lc_phys_dom]

AAEP to Domain Association

Fabric > Access Policies > Policies > Global > AAEP > lc_AAEP:

APIC# moquery -c infraRsDomP | grep lc_AAEP 
dn : uni/infra/attentp-lc_AAEP/rsdomP-[uni/phys-lc_phys_dom]

IPG to AAEP Association

Fabric > Access Policies > Interfaces > Leaf Interfaces > Policy Groups > Leaf Access Port > lc_IPG:

[+] IPG to AAEP association:

APIC# moquery -c infraRsAttEntP | grep -A 15 lc_IPG | grep tDn
tDn : uni/infra/attentp-lc_AAEP

Leaf Profile to Interface Selector Association

Fabric > Access Policies > Interfaces > Leaf Interfaces > Profiles > leaf103_IP:

APIC# moquery -c infraHPortS | grep leaf103_IP
dn : uni/infra/accportprof-leaf103_IP/hports-lc_Interface_Selector-typ-range

Interface Selector to Interface Policy Group Association

Fabric > Access Policies > Interfaces > Leaf Interfaces > Profiles > leaf103_IP > lc_Interface_Selector:

[+]  IPG to Interface Selector association:

APIC# moquery -c infraRsAccBaseGrp | grep -B 15 lc_IPG | grep dn
dn : uni/infra/accportprof-lead103_IP/hports-lc_Interface_Selector-typ-range/rsaccBaseGrp

Leaf Interface Profile to Interface Selectors and Leaf Switch Profile Associations

Fabric > Access Policies > Switches > Leaf Switches > Profiles > leaf103_SP:

[+] Leaf Interface Profile to Switch Profile association:

APIC# moquery -c infraRsAccPortP | grep leaf103_IP | grep dn
dn : uni/infra/nprof-leaf103_SP/rsaccPortP-[uni/infra/accportprof-leaf103_IP]

[+] Switch Profile to Switch Prolicy Group association:

APIC# moquery -c infraRsAccNodePGrp | grep -A 8 leaf103_SP | grep tDn
tDn : uni/infra/funcprof/accnodepgrp-leaf103_SPG

Vlan Deployment Verification

Scenario

• Access Encap VLAN 420 is deployed on node 103 - E1/13
• All relevent Access Policy and EPG configuration deployed

Check ACI Fabric VLAN Deployment via APIC

A moquery against class fvIcConn can be filtered on VLAN encaps of interest to show every EPG/Switch/Interface combination where the VLAN has been deployed.

APIC# moquery -c fvIfConn -f 'fv.IfConn.encap=="vlan-420"' | grep dn 
dn               : uni/epp/fv-[uni/tn-lc_TN/ap-lc_APP/epg-lc_EPG]/node-103/stpathatt-[eth1/13]/conndef/conn-[vlan-420]-[0.0.0.0]

Check VLAN Deployment via Switch CLI

Run show vlan extended on any switch to check which VLANs are currently deployed on a switch, along with which EPG and Interface the VLAN is tied to.

The encap-id xx filter is available on ACI Release 4.2 and later.

Node-103# show vlan encap-id 420 extended 

 VLAN Name                             Encap            Ports                    
 ---- -------------------------------- ---------------- ------------------------ 
2    lc_TN:lc_APP:lc_EPG              vlan-420         Eth1/13                  

Check Platform Independent VLAN Deployment via Switch CLI

Every VLAN in an ACI Switch Node is mapped to some Platform Independent (PI) VLAN which is a value local to each Switch Node. 

Access Encaps are mapped to a PI VLAN called an 'FD VLAN', whereas Bridge Domains are mapped to a PI Vlan called a "BD VLAN."

Run show system internal epm vlan all on a switch to display the list of vlans deployed on the leaf.

Node-103# show vlan extended | egrep "Encap|----|1/13"
 VLAN Name                             Encap            Ports                    
 ---- -------------------------------- ---------------- ------------------------ 
2    lc_TN:lc_APP:lc_EPG              vlan-420         Eth1/13                   --> FD vlan 2               
18   lc_TN:lc_BD                      vxlan-16416666   Eth1/13                   --> BD vlan 18

The FD vlan and BD vlan to interface porogramming can be validated with a show interface command:

Node-103# show interface eth 1/13 trunk | grep -A 2 Allowed           
 Port           Vlans Allowed on Trunk                                             
-----------------------------------------------------------------------------------
 Eth1/13        2,18

Check SVI VLAN Deployment

If validating a layer-3 vlan with a BD SVI, moquery class fvSubnet to get the IP address of the subnet:

APIC# moquery -c fvSubnet | grep lc_BD
dn : uni/tn-lc_TN/BD-lc_BD/subnet-[10.10.10.254/24]

Then check show ip interface brief again and check for the matching ip address to validate the vlan and expected VRF.

In this example, validation is for "BD VLan 18" from the previous CLI output example:

Node-103# show ip interface brief
...
 
IP Interface Status for VRF "lc_TN:lc_VRF"(16)
Interface            Address              Interface Status
vlan18               10.10.10.254/24   protocol-up/link-up/admin-up

Reference Diagrams

High Level Programming Sequence for a Static Path Binding

This high level sequence summarizes the steps involved from VLAN Static Path API call, to Switch Node VLAN Deployment.

Access Policy Relationship Block Diagram

This block diagram shows the relationship between Access Policies to ensure a successful Switch Node VLAN deployment.

Standalone NXOS Commands mapped to Access Policies

Every network engineer has actually been working with the idea of access policies; only they have been defined as text in a file via a CLI interface of a standalone device.

When a fault F0467 is seen, it is important to first understand the access policies and ensure they are configured correctly.

VLAN Verification Command Cheat Sheet

Each command output provides a variable that is used for the next command in the list.

These commands were referenced throughout this document to troubleshoot the different scenarios.

Related Information

• https://www.ciscolive.com/on-demand/on-demand-library.html?&currentTab=session&search=BRKDCN-3900
• https://www.ciscolive.com/on-demand/on-demand-library.html?&currentTab=session&search=BRKACI-2770
• https://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/aci/apic/sw/4-x/troubleshooting/Cisco_TroubleshootingApplicationCentricInfrastructureSecondEdition.pdf