Configuring Infra for Cisco ACI Sites

Prerequisites and Guidelines

The following sections describe the steps necessary to configure the general as well as site-specific fabric Infra settings.

Before you proceed with Infra configuration, you must have configured and added the sites as described in previous sections, which includes:

  • Configuring each site's fabric access policies.

  • Configuring direct communication and routable subnets for sites with remote leaf switches.

In addition, keep in mind the following:

  • Any infrastructure changes such as adding and removing spine switches or spine node ID changes require a Multi-Site Orchestrator fabric connectivity information refresh described in the Refreshing Site Connectivity Information as part of the general Infra configuration procedures.

  • The Overlay Unicast TEP, Overlay Multicast TEP, and BGP-EVPN Router-IDs IP addresses assigned on the Orchestrator should not be taken from the address space of the original fabric's Infra TEP pool or from the 0.x.x.x range.

Configuring Infra: General Settings

This section describes how to configure general Infra settings for all the sites.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left sidebar, select General Settings.

Step 5

Configure control plane BGP.

  1. From the BGP Peering Type dropdown, choose either full-mesh or route-reflector.

    The route-reflector option is effective only when all sites are part of the same BGP Autonomous System (AS).

  2. In the Keepalive Interval (Seconds) field, enter the keep alive interval seconds.

    We recommend keeping the default value.

  3. In the Hold Interval (Seconds) field, enter the hold interval seconds.

    We recommend keeping the default value.

  4. In the Stale Interval (Seconds) field, enter stale interval seconds.

    We recommend keeping the default value.

  5. Choose whether you want to turn on the Graceful Helper option.

  6. In the Maximum AS Limit field, enter the maximum AS limit.

  7. In the BGP TTL Between Peers field, enter the BGP TTL between peers.

Refreshing Site Connectivity Information

Any infrastructure changes, such as adding and removing spines or changing spine node IDs, require a Multi-Site fabric connectivity site refresh. This section describes how to pull up-to-date connectivity information directly from each site's APIC.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.
Step 2

In the Main menu, select Infrastructure > Infra Configuration.

Step 3

In the top right of the main Infra Configuration view, click the Configure Infra button.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, click the Reload Site Data button to pull fabric information from the APIC.

Step 6

(Optional) In the Confirmation dialog, check the box if you want to remove configuration for decommissioned spine switch nodes.

If you choose to enable this checkbox, all configuration info for any currently decommissioned spine switches will be removed from the database.

Step 7

Finally, click Yes to confirm and load the connectivity information.

This will discover any new or removed spines and all site-related fabric connectivity will be re-imported from the APIC.

Configuring Infra: On-Premises Site Settings

This section describes how to configure site-specific Infra settings for on-premises sites.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific on-premises site.

Step 5

In the right <Site> Settings pane, enable the Multi-Site knob to manage the site from the Orchestrator.

Step 6

(Optional) Enable the CloudSec Encryption knob encryption for the site.

CloudSec Encryption provides inter-site traffic encryption. The "Infrastructure Management" chapter in the Cisco Multi-Site Configuration Guide covers this feature in detail.

Step 7

Specify the Overlay Multicast TEP.

This address is used for the inter-site L2 BUM and L3 multicast traffic. This IP address is deployed on all spine switches that are part of the same fabric, regardless of whether it is a single pod or multi-pod fabric.

Step 8

Specify the BGP Autonomous System Number.

Step 9

Specify the BGP Password.

Step 10

Specify the OSPF Area ID.

When configuring the Multi-Site infra OSPF details, we recommend that you use OSPF Area 0. If you use an Area ID other than 0, in the next step configure it as a regular OSPF area type and not a stub area type.

Step 11

Select the OSPF Area Type from the dropdown menu.

The OSPF area type can be one of the following:

  • nssa

  • regular

  • stub

Step 12

Select the external routed domain from the dropdown menu.

Choose an external router domain that you have created in the Cisco APIC GUI.
Step 13

Configure OSPF settings for the site.

You can either click an existing policy (for example, msc-ospf-policy-default ) to modify it or click +Add Policy to add a new OSPF policy. Then in the Add/Update Policy window, specify the following:

  • In the Policy Name field, enter the policy name.

  • In the Network Type field, choose either broadcast, point-to-point, or unspecified.

    The default is broadcast.

  • In the Priority field, enter the priority number.

    The default is 1.

  • In the Cost of Interface field, enter the cost of interface.

    The default is 0.

  • From the Interface Controls dropdown menu, choose one of the following:

    • advertise-subnet

    • bfd

    • mtu-ignore

    • passive-participation

  • In the Hello Interval (Seconds) field, enter the hello interval in seconds.

    The default is 10.

  • In the Dead Interval (Seconds) field, enter the dead interval in seconds.

    The default is 40.

  • In the Retransmit Interval (Seconds) field, enter the retransmit interval in seconds.

    The default is 5.

  • In the Transmit Delay (Seconds) field, enter the transmit delay in seconds.

    The default is 1.

Step 14

(Optional) Configure SR-MPLS settings for the site.

If the site is connected via an MPLS network, enable the SR-MPLS Connectivity knob and provide the Segment Routing global block (SRGB) range.

The Segment Routing Global Block (SRGB) is the range of label values reserved for Segment Routing (SR) in the Label Switching Database (LSD). These values are assigned as segment identifiers (SIDs) to SR-enabled nodes and have global significance throughout the domain.

The default range is 16000-23999.

If you enable MPLS connectivity for the site, you will need to configure additional settings as described in the "Sites Connected via SR-MPLS" chapter of the Cisco Multi-Site Configuration Guide for ACI Fabrics.

Configuring Infra: Pod Settings

This section describes how to configure pod-specific settings in each site.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.
Step 2

In the Main menu, click Sites.

Step 3

In the Sites view, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, select a pod.
Step 6

In the right POD Properties pane, add the Overlay Unicast TEP for the POD.

This IP address is deployed on all spine switches that are part of the same pod and used for intersite known unicast traffic.
Step 7

Click +Add TEP Pool to add a routable TEP pool.

The routable TEP pools are used for public IP addresses for inter-site connectivity.
Step 8

Repeat the procedure for every pod in the site.

Configuring Infra: Spine Switches

This section describes how to configure spine switches in each site for Cisco Multi-Site.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.
Step 2

In the Main menu, click Sites.

Step 3

In the Sites view, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, select a spine switch within a pod.
Step 6

In the right <Spine> Settings pane, click +Add Port.

Step 7

In the Add Port window, enter the following information:

  • In the Ethernet Port ID field, enter the port ID, for example 1/29.

  • In the IP Address field, enter the IP address/netmask.

    MSO creates a sub-interface with VLAN 4 with the specified IP ADDRESS under the specified PORT.

  • In the MTU field, enter the MTU. You can specify either inherit or a value between 576 and 9000.

    MTU of the spine port should match MTU on IPN side.

  • In the OSPF Policy field, choose the OSPF policy for the switch that you have configured in Configuring Infra: On-Premises Site Settings.

    OSPF settings in the OSPF policy you choose should match on IPN side.

  • For OSPF Authentication, you can pick either none or one of the following:

    • MD5

    • Simple
Step 8

Enable BGP Peering knob.

In a single Pod fabric with more than two spine switches, BGP peering should only be enabled on a pair (for redundancy) of spine switches called BGP Speakers. All other spine switches should have BGP peering disabled and will function as BGP Forwarders.

In a Multi-Pod fabric BGP peering should only be enabled on a couple of BGP speaker spine switches, each deployed in a different Pod. All other spines switches should have BGP peering disabled and function as BGP forwarders.

Step 9

In the BGP-EVPN Router-ID field, provide the IP address used for BGP-eVPN session between sites.

Step 10

Repeat the procedure for every spine switch.

Deploying Infra Configuration

This section describes how to deploy the Infra configuration to each APIC site.

Procedure

In the top right of the main pane, click Deploy to deploy the configuration.