Chapter 1. New and Changed Information

- New and Changed Information

New and Changed Information

The following table provides an overview of the significant changes to the organization and features in this guide from the release the guide was first published to the current release. The table does not provide an exhaustive list of all changes made to the guide.

Latest Updates

Release

New Feature or Update

Where Documented

3.3(1)

First release of this document.

--

Chapter 2. Deploying Multi-Site Orchestrator

- Deployment Overview

- Prerequisites and Guidelines

- Installing Multi-Site Orchestrator Application Using App Store

- Installing Multi-Site Orchestrator Application Manually

Deployment Overview

Beginning with Release 3.2(1), you must deploy the Cisco Multi-Site Orchestrator (MSO) as an application in Cisco Nexus Dashboard.

Note

While Release 3.2(1) supported only the physical form factor of Nexus Dashboard, Release 3.3(1) and later can be deployed in physical, virtual, or cloud Nexus Dashboard clusters.

If you are upgrading from a release prior to release 3.2(1), familiarize yourself with deployment overview described in this section, then follow the instructions in Migrating Existing Cluster to Nexus Dashboard.

Cisco Nexus Dashboard is a central management console for multiple data center sites and a common platform for hosting Cisco data center applications, such as the Nexus Dashboard Orchestrator or Nexus Insights. Nexus Dashboard provides a common platform and modern technology stack for these micro-services-based applications, simplifying the life cycle management of the different modern applications and reducing the operational overhead to run and maintain those applications.

Each Nexus Dashboard cluster consists of 3 master nodes. You can also deploy additional worker nodes to enable horizontal scaling and a standby node for easy cluster recovery in case of a master node failure.

For detailed information about Nexus Dashboard cluster initial deployment and configuration, see Cisco Nexus Dashboard Deployment Guide.

For more information about using Nexus Dashboard, see the Cisco Nexus Dashboard User Guide.

This document describes initial installation requirements and procedures for the Nexus Dashboard Orchestrator service. Detailed configuration and use case information is available from the Cisco Multi-Site Configuration Guide for Cisco ACI or Cisco Multi-Site Configuration Guide for Cisco DCNM for your release and the Cisco Cloud APIC use case documents, depending on the type of fabrics you plan to manage.

Prerequisites and Guidelines

Nexus Dashboard

You must have Cisco Nexus Dashboard cluster deployed and its fabric connectivity configured, as described in Cisco Nexus Dashboard Deployment Guide before proceeding with any additional requirements and the Multi-Site Orchestrator application installation described here.

Orchestrator Release

Minimum Nexus Dashboard Release

Release 3.3(1) and later

Cisco Nexus Dashboard, Release 2.0.2h

Nexus Dashboard Networks

When first configuring Nexus Dashboard, you will need to provide two IP addresses for the two Nexus Dashboard interfaces—one connected to the Data Network and the other to the Management Network. The data network is used for the nodes' clustering and Cisco fabrics traffic. The management network is used to connect to the Cisco Nexus Dashboard GUI, CLI, or API.

The two interfaces can be in the same or different subnets. In addition, each network's interfaces across different nodes in the cluster can also be in different subnets.

Connectivity between the nodes is required on both networks with the round trip time (RTT) not exceeding 150ms for Multi-Site Orchestrator. Other application running in the same Nexus Dashboard cluster may have lower RTT requirements and you must always use the lowest RTT requirement when deploying multiple applications in the same Nexus Dashboard cluster. We recommend consulting the Cisco Nexus Dashboard Deployment Guide for more information.

When Multi-Site Orchestrator app is deployed in Nexus Dashboard, it uses each of the two networks for different purposes as shown in the following table:

MSO Traffic Type

Nexus Dashboard Network

Any traffic to and from:

- Cisco APIC

- Cisco DCNM

- Any other remote devices or controllers

Data network

Intra-cluster communication

Data network

Audit log streaming (Splunk/syslog)

Management network

Remote backup

Management network

Nexus Dashboard Cluster Sizing

Nexus Dashboard supports co-hosting of applications. Depending on the type and number of applications you choose to run, you may be required to deploy additional worker nodes in your cluster. For cluster sizing information and recommended number of nodes based on specific use cases, see the Cisco Nexus Dashboard Capacity Planning tool.

If you plan to host other applications in addition to the Multi-Site Orchestrator, ensure that you deploy and configure additional Nexus Dashboard nodes based on the cluster sizing tool recommendation, as described in the Cisco Nexus Dashboard User Guide, which is also available directly from the Nexus Dashboard GUI.

Note

This release of Multi-Site Orchestrator can be co-hosted with other applications on physical Nexus Dashboard clusters only. If you are deploying the Multi-Site Orchestrator application in a virtual or cloud Nexus Dashboard cluster, you must not install other applications in the same cluster.

Network Time Protocol (NTP)

Multi-Site Orchestrator uses NTP for clock synchronization, so you must have an NTP server configured in your environment.

- Hardware Requirements For ACI Fabrics

- Hardware Requirements For DCNM Fabrics

Hardware Requirements For ACI Fabrics

Spine Switch Requirements

Multi-Site requires second generation (Cloud Scale) spine switches for intersite connectivity. All Cloud Scale spine switches supported by a given ACI release are supported by Multi-Site Orchestrator.

Nexus 9000 first generation switches are not supported for Multi-Site intersite connectivity, but can still be used within a single fabric as long as that fabric is running an APIC release prior to 5.0(1).

Refer to the ACI-mode Switches Hardware Support Matrix for the complete list of supported spines for each release.

Leaf Switch Requirements

Multi-Site has no dependency on the fabrics' leaf switches and as such supports the same leaf switch models as the Cisco APIC. The full list of supported hardware is available in the ACI-mode Switches Hardware Support Matrix.

IPN Connectivity Across Sites

The following figure shows how spine switches supported with ACI Multi-Site are connected to the intersite network.

You can choose to mix spine switches supported by Multi-Site with switches that are not supported within the same Cisco APIC fabric, but only the supported switches can connect to the intersite network as shown in the following figure.

Hardware Requirements For DCNM Fabrics

Spine Switch Requirements

The following table summarizes the hardware requirements for EVPN Multi-Site Architecture:

- Cisco Nexus 9300 EX platform

- Cisco Nexus 9300 FX platform

- Cisco Nexus 9300 FX2 platform

- Cisco Nexus 9300-GX platform

- Cisco Nexus 9332C platform

- Cisco Nexus 9364C platform

- Cisco Nexus 9500 platform with X9700-EX line card

- Cisco Nexus 9500 platform with X9700-FX line card

The hardware requirements for the site-internal BGP Route Reflector (RR) and VTEP of a VXLAN BGP EVPN site remain the same as those without the EVPN Multi-Site Border Gateways (BGW). This document does not cover the hardware and software requirements for the VXLAN EVPN site-internal network.

Installing Multi-Site Orchestrator Application Using App Store

This section describes how to install Cisco Multi-Site Orchestrator application in an existing Cisco Nexus Dashboard cluster.

Before you begin

- Ensure that you meet the requirements and guidelines described in Prerequisites and Guidelines.

- The Cisco DC App Center must be reachable from the Nexus Dashboard via the Management Network directly or using a proxy configuration. Nexus Dashboard proxy configuration is described in the Nexus Dashboard User Guide.

If you are unable to establish the connection to the DC App Center, skip this section and follow the steps described in Installing Multi-Site Orchestrator Application Manually.

- The App Store allows you to install the latest version of the application only.

If you want to install a version prior to Release 3.3(1), see the Multi-Site Orchestrator Installation Guide specific to that release for the available deployment options and procedures.

Procedure

Step 1

Log in to the Nexus Dashboard GUI

Step 2

Navigate to the App Store and choose Multi-Site Orchestrator app.

1. From the left navigation menu, select Service Catalog.

2. Select the App Store tab.

3. In the Multi-Site Orchestrator tile, click Install.

Step 3

In the License Agreement window that opens, click Agree and Download.

Step 4

Wait for the application to be downloaded to the Nexus Dashboard and deployed.

It may take up to 30 minutes for the application to replicate to all nodes and all services to fully deploy.

Step 5

Enable the app.

After installation is complete, the application will remain in the Disabled state by default and you must enable it.

To enable the app, click the ... menu on the app and select Enable.

Step 6

Launch the app.

To launch the app, simply click Open on the application tile in the Nexus Dashboard's Service Catalog page.

The single sign-on (SSO) feature allows you to log in to the application using the same credentials as you used for the Nexus Dashboard.

Installing Multi-Site Orchestrator Application Manually

This section describes how to manually upload and install Cisco Multi-Site Orchestrator application in an existing Cisco Nexus Dashboard cluster.

Before you begin

- Ensure that you meet the requirements and guidelines described in Prerequisites and Guidelines.

Procedure

Step 1

Download the Cisco Multi-Site Orchestrator application.

1. Browse to the Multi-Site Orchestrator page on DC App Center:

https://dcappcenter.cisco.com/nexus-dashboard-orchestrator.html

2. From the Version drop-down, choose the version you want to install and click Download.

3. Click Agree and download to accept the license agreement and download the image.

Step 2

Log in to your Cisco Nexus Dashboard dashboard.

When deploying an app, you need to install it in only one of the Nexus Dashboard nodes, the application will be replicated to the other nodes in the cluster automatically. So you can log in to any one of your Nexus Dashboard nodes using its management IP address.

Step 3

Upload the app image.

1. In the left navigation bar, click Service Catalog.

2. Select the Installed Services tab.

3. In the top right of the main pane, select Actions > Upload App.

Step 4

Upload the image file to the Nexus Dashboard cluster.

1. Choose the location of the image.

If you downloaded the application image to your system, choose Local.

If you are hosting the image on a server, choose Remote.

2. Choose the file.

If you chose Local in the previous substep, click Select File and select the app image you downloaded.

If you chose Remote, provide the full URL to the image file, for example http://<ip-address>:<port>/<full-path>/cisco-mso-<version>.aci.

3. Click Upload to add the app to the cluster.

Step 5

Wait for the application to be downloaded to the Nexus Dashboard and deployed.

It may take up to 30 minutes for the application to replicate to all nodes and all services to fully deploy.

Step 6

Enable the app.

After installation is complete, the application will remain in the Disabled state by default and you must enable it.

To enable the app, click the ... menu on the app and select Enable.

Step 7

Launch the app.

To launch the app, simply click Open on the application tile in the Nexus Dashboard's Service Catalog page.

The single sign-on (SSO) feature allows you to log in to the application using the same credentials as you used for the Nexus Dashboard.

Part I. Day-0 Operations for ACI Fabrics

- Configuring Cisco ACI Sites

- Adding and Deleting Sites

- Configuring Infra General Settings

- Configuring Infra for Cisco APIC Sites

- Configuring Infra for Cisco Cloud APIC Sites

- Deploying Infra Configuration for ACI Sites

Chapter 3. Configuring Cisco ACI Sites

- Pod Profile and Policy Group

- Configuring Fabric Access Policies for All APIC Sites

- Configuring Sites That Contain Remote Leaf Switches

- Cisco Mini ACI Fabrics

Pod Profile and Policy Group

In each site's APIC, you must have one POD profile with a POD policy group. If your site does not have a POD policy group you must create one. Typically, these settings will already exist as you will have configured them when you first deployed the fabric.

Procedure

Step 1

Log in to the site's APIC GUI.

Step 2

Check that the POD profile contains a POD policy group.

Navigate to Fabric > Fabric Policies > Pods > Profiles > Pod Profile default.

Step 3

If necessary, create a POD policy group.

1. Navigate to Fabric > Fabric Policies > Pods > Policy Groups.

2. Right-click Policy Groups and select Create Pod Policy Group.

3. Enter the appropriate information and click Submit.

Step 4

Assign the new POD policy group to the default POD profile.

1. Navigate to Fabric > Fabric Policies > Pods > Profiles > Pod Profile default

2. Select the default profile.

3. Choose the new pod policy group and click Update.

Configuring Fabric Access Policies for All APIC Sites

Before your APIC fabrics can be added to and managed by the Multi-Site Orchestrator, there is a number of fabric-specific access policies that you must configure on each site.

- Configuring Fabric Access Global Policies

- Configuring Fabric Access Interface Policies

Configuring Fabric Access Global Policies

This section describes the global fabric access policy configurations that must be created for each APIC site before it can be added to and managed by the Multi-Site Orchestrator.

Procedure

Step 1

Log in directly to the site's APIC GUI.

Step 2

From the main navigation menu, select Fabric > Access Policies.

You must configure a number of fabric policies before the site can be added to the Multi-Site Orchestrator. From the APIC's perspective, this is something you do just like you would if you were connecting a bare-metal host, where you would configure domains, AEPs, policy groups, and interface selectors; you must configure the same options for connecting the spine switch interfaces to the inter-site network for all the sites that will be part of the same Multi-Site domain.

Step 3

Specify the VLAN pool.

The first thing you configure is the VLAN pool. We use Layer 3 sub-interfaces tagging traffic with VLAN-4 to connect the spine switches to the inter-site network.

1. In the left navigation tree, browse to Pools > VLAN.

2. Right-click the VLAN category and choose Create VLAN Pool.

In the Create VLAN Pool window, specify the following:

- For the Name field, specify the name for the VLAN pool, for example msite.

- For Allocation Mode, specify Static Allocation.

- And for the Encap Blocks, specify just the single VLAN 4. You can specify a single VLAN by entering the same number in both Range fields.

Step 4

Configure Attachable Access Entity Profiles (AEP).

1. In the left navigation tree, browse to Global Policies > Attachable Access Entity Profiles.

2. Right-click the Attachable Access Entity Profiles category and choose Create Attachable Access Entity Profiles.

In the Create Attachable Access Entity Profiles window, specify the name for the AEP, for example msite-aep.

3. Click Next and Submit

No additional changes, such as interfaces, are required.

Step 5

Configure domain.

The domain you configure is what you will select from the Multi-Site Orchestrator when adding this site.

1. In the left navigation tree, browse to Physical and External Domains > External Routed Domains.

2. Right-click the External Routed Domains category and choose Create Layer 3 Domain.

In the Create Layer 3 Domain window, specify the following:

- For the Name field, specify the name the domain, for example msite-l3.

- For Associated Attachable Entity Profile, select the AEP you created in Step 4.

- For the VLAN Pool, select the VLAN pool you created in Step 3.

3. Click Submit.

No additional changes, such as security domains, are required.

What to do next

After you have configured the global access policies, you must still add interfaces policies as described in Configuring Fabric Access Interface Policies.

Configuring Fabric Access Interface Policies

This section describes the fabric access interface configurations that must be done for the Multi-Site Orchestrator on each APIC site.

Before you begin

You must have configured the global fabric access policies, such as VLAN Pool, AEP, and domain, in the site's APIC, as described in Configuring Fabric Access Global Policies.

Procedure

Step 1

Log in directly to the site's APIC GUI.

Step 2

From the main navigation menu, select Fabric > Access Policies.

In addition to the VLAN, AEP, and domain you have configured in previous section, you must also create the interface policies for the fabric's spine switch interfaces that connect to the Inter-Site Network (ISN).

Step 3

Configure a spine policy group.

1. In the left navigation tree, browse to Interface Policies > Policy Groups > Spine Policy Groups.

This is similar to how you would add a bare-metal server, except instead of a Leaf Policy Group, you are creating a Spine Policy Group.

2. Right-click the Spine Policy Groups category and choose Create Spine Access Port Policy Group.

In the Create Spine Access Port Policy Group window, specify the following:

- For the Name field, specify the name for the policy group, for example Spine1-PolGrp.

- For the Link Level Policy field, specify the link policy used between your spine switch and the ISN.

- For CDP Policy, choose whether you want to enable CDP.

- For the Attached Entity Profile, select the AEP you have configured in previous section, for example msite-aep.

3. Click Submit.

No additional changes, such as security domains, are required.

Step 4

Configure a spine profile.

1. In the left navigation tree, browse to Interface Policies > Profiles > Spine Profiles.

2. Right-click the Spine Profiles category and choose Create Spine Interface Profile.

In the Create Spine Interface Profile window, specify the following:

- For the Name field, specify the name for the profile, for example Spine1-ISN.

- For Interface Selectors, click the + sign to add the port on the spine switch that connects to the ISN. Then in the Create Spine Access Port Selector window, provide the following:

- For the Name field, specify the name for the port selector, for example Spine1-ISN.

- For the Interface IDs, specify the switch port that connects to the ISN, for example 5/32.

- For the Interface Policy Group, choose the policy group you created in the previous step, for example Spine1-PolGrp.

Then click OK to save the port selector.

3. Click Submit to save the spine interface profile.

Step 5

Configure a spine switch selector policy.

1. In the left navigation tree, browse to Switch Policies > Profiles > Spine Profiles.

2. Right-click the Spine Profiles category and choose Create Spine Profile.

In the Create Spine Profile window, specify the following:

- For the Name field, specify the name for the profile, for example Spine1.

- For Spine Selectors, click the + to add the spine and provide the following:

- For the Name field, specify the name for the selector, for example Spine1.

- For the Blocks field, specify the spine node, for example 201.

3. Click Update to save the selector.

4. Click Next to proceed to the next screen.

5. Select the interface profile you have created in the previous step

For example Spine1-ISN.

6. Click Finish to save the spine profile.

Configuring Sites That Contain Remote Leaf Switches

Starting with Release 2.1(2), the Multi-Site architecture supports APIC sites with Remote Leaf switches. The following sections describe guidelines, limitations, and configuration steps required to allow Multi-Site Orchestrator to manage these sites.

- Remote Leaf Guidelines and Limitations

- Configuring Routable Subnets for Remote Leaf Switches

- Enabling Direct Communication for Remote Leaf Switches

Remote Leaf Guidelines and Limitations

If you want to add an APIC site with a Remote Leaf to be managed by the Multi-Site Orchestrator, the following restrictions apply:

- You must upgrade your Cisco APIC to Release 4.2(4) or later.

- Only physical Remote Leaf switches are supported in this release

- Only -EX and -FX or later switches are supported as Remote Leaf switches for use with Multi-Site

- Remote Leaf is not supported with back-to-back connected sites without IPN switches

- Remote Leaf switches in one site cannot use another site's L3Out

- Stretching a bridge domain between one site and a Remote Leaf in another site is not supported

You must also perform the following tasks before the site can be added to and managed by the Multi-Site Orchestrator:

- You must enable Remote Leaf direct communication and configure routable subnets directly in the site's APIC, as described in the following sections.

- You must add the routable IP addresses of Cisco APIC nodes in the DHCP-Relay configuration applied on the interfaces of the Layer 3 routers connecting to the Remote Leaf switches.

The routable IP address of each APIC node is listed in the Routable IP field of the System > Controllers > <controller-name> screen of the APIC GUI.

Configuring Routable Subnets for Remote Leaf Switches

Before you can add a site that contains one or more Remote Leaf switches to the Multi-Site Orchestrator, you must configure routable subnets for the pod with which the Remote Leaf nodes are associated.

Procedure

Step 1

Log in directly to the site's APIC GUI.

Step 2

From the menu bar, select Fabric > Inventory.

Step 3

In the Navigation pane, click Pod Fabric Setup Policy.

Step 4

In the main pane, double-click the pod where you want to configure the subnets.

Step 5

In the Routable Subnets area, click the + sign to add a subnet.

Step 6

Enter the IP and Reserve Address Count, set the state to Active or Inactive, then click Update to save the subnet.

When configuring routable subnets, you must provide a netmask between /22 and /29.

Step 7

Click Submit to save the configuration.

Enabling Direct Communication for Remote Leaf Switches

Before you can add a site that contains one or more Remote Leaf switches to the Multi-Site Orchestrator, you must configure direct remote leaf communication for that site. Additional information about remote leaf direct communication feature is available in the Cisco APIC Layer 3 Networking Configuration Guide. This section outlines the steps and guidelines specific to the integration with Multi-Site.

Note

Once you enable Remote Leaf switch direct communication, the switches will function in the new mode only

Procedure

Step 1

Log in directly to the site's APIC.

Step 2

Enable direct traffic forwarding for Remote Leaf switches.

1. From the menu bar, navigate to System > System Settings.

2. From the left side bar, select Fabric Wide Setting.

3. Check the Enable Remote Leaf Direct Traffic Forwarding checkbox.

Note

You cannot disable this option after you enable it.

4. Click Submit to save the changes.

Cisco Mini ACI Fabrics

Cisco Multi-Site supports Cisco Mini ACI fabrics as typical on-premises sites without requiring any additional configuration. This section provides a brief overview of Mini ACI fabrics, detailed info on deploying and configuring this type of fabrics is available in Cisco Mini ACI Fabric and Virtual APICs.

Cisco ACI, Release 4.0(1) introduced Mini ACI Fabric for small scale deployment. Mini ACI fabric works with Cisco APIC cluster consisting of one physical APIC and two virtual APICs (vAPIC) running in virtual machines. This reduces the physical footprint and cost of the APIC cluster, allowing ACI fabric to be deployed in scenarios with limited rack space or initial budget, such as a colocation facility or a single-room data center, where a full-scale ACI installations may not be practical due to physical footprint or initial cost.

The following diagram shows an example of a mini Cisco ACI fabric with a physical APIC and two virtual APICs (vAPICs):

Cisco Mini ACI Fabric

Chapter 4. Adding and Deleting Sites

- Cisco MSO and APIC Interoperability Support

- Adding Cisco ACI Sites

- Removing Sites

- Cross Launch to Fabric Controllers

Cisco MSO and APIC Interoperability Support

Cisco Multi-Site Orchestrator (MSO) does not require a specific version of APIC to be running in all sites. The APIC clusters in each site as well as the MSO itself can be upgraded independently of each other and run in mixed operation mode as long as the fabric can be on-boarded to the Nexus Dashboard where the Multi-Site Orchestrator application is installed. As such, we recommend that you always upgrade to the latest release of the Multi-Site Orchestrator.

However, keep in mind that if you upgrade the MSO before upgrading the APIC clusters in one or more sites, some of the new MSO features may not yet be supported by an earlier APIC release. In that case a check is performed on each template to ensure that every configured option is supported by the target sites.

The check is performed when you save a template or deploy a template. If the template is already assigned to a site, any unsupported configuration options will not be saved; if the template is not yet assigned, you will be able to assign it to a site, but not be able to save or deploy the schema if it contains configuration unsupported by that site.

In case an unsupported configuration is detected, an error message will show, for example: This APIC site version <site-version> is not supported by MSO. The minimum version required for this <feature> is <required-version> or above.

The following table lists the features and the minimum required APIC release for each one:

Note

While some of the following features are supported on earlier Cisco APIC releases, Release 4.2(4) is the earliest release that can be on-boarded to the Nexus Dashboard and managed by this release of Multi-Site Orchestrator.

Feature

Minimum APIC Version

ACI Multi-Pod Support

Release 4.2(4)

Service Graphs (L4-L7 Services)

Release 4.2(4)

External EPGs

Release 4.2(4)

ACI Virtual Edge VMM Support

Release 4.2(4)

DHCP Support

Release 4.2(4)

Consistency Checker

Release 4.2(4)

vzAny

Release 4.2(4)

Host Based Routing

Release 4.2(4)

CloudSec Encryption

Release 4.2(4)

Layer 3 Multicast

Release 4.2(4)

MD5 Authentication for OSPF

Release 4.2(4)

EPG Preferred Group

Release 4.2(4)

Intersite L3Out

Release 4.2(4)

EPG QoS Priority

Release 4.2(4)

Contract QoS Priority

Release 4.2(4)

Single Sign-On (SSO)

Release 5.0(1)

Multicast Rendezvous Point (RP) Support

Release 5.0(1)

Transit Gateway (TGW) support for AWS and Azure Sites

Release 5.0(1)

SR-MPLS Support

Release 5.0(1)

Cloud LoadBalancer High Availability Port

Release 5.0(1)

Service Graphs (L4-L7 Services) with UDR

Release 5.0(2)

3rd Party Device Support in Cloud

Release 5.0(2)

Cloud Loadbalancer Target Attach Mode Feature

Release 5.1(1)

Support security and service insertion in Azure for non-ACI networks reachable through Express Route

Release 5.1(1)

CSR Private IP Support

Release 5.1(1)

Extend ACI policy model and automation for Cloud native services in Azure

Release 5.1(1)

Flexible segmentation through multiple VRF support within a single VNET for Azure

Release 5.1(1)

Private Link automation for Azure PaaS and third-party services

Release 5.1(1)

Openshift 4.3 IPI on Azure with ACI-CNI

Release 5.1(1)

Cloud Site Underlay Configuration

Release 5.2(1)

Adding Cisco ACI Sites

This section describes how to add a Cisco APIC or Cloud APIC site using the Nexus Dashboard GUI and then enable that site to be managed by Multi-Site Orchestrator.

Before you begin

- If you are adding on-premises ACI site, you must have completed the site-specific configurations in each site's APIC, as described in previous sections in this chapter.

- You must ensure that the site(s) you are adding are running Release 4.2(4) or later.

Procedure

Step 1

Log in to the Nexus Dashboard GUI

Step 2

Add a new site.

1. From the left navigation menu, select Sites.

2. In the top right of the main pane, select Actions > Add Site.

Step 3

Provide site information.

1. For Site Type, select ACI or Cloud ACI depending on the type of ACI fabric you are adding.

2. Provide the controller information.

You need to provide the Host Name/IP Address, User Name, and Password. for the APIC controller currently managing your ACI fabrics.

For on-premises ACI sites managed by Cisco APIC, if you plan to use this site with Day-2 Operations applications such as Nexus Insights, you must also provide the In-Band EPG name used to connect the Nexus Dashboard to the fabric you are adding. Otherwise, if you will use this site with Multi-Site Orchestrator only, you can leave this field blank.

3. Click Add to finish adding the site.

At this time, the sites will be available in the Nexus Dashboard, but you still need to enable them for Multi-Site Orchestrator management as described in the following steps.

Step 4

Repeat the previous steps for any additional ACI sites.

Step 5

From the Nexus Dashboard's Service Catalog, open the Multi-Site Orchestrator application.

You will be automatically logged in using the Nexus Dashboard user's credentials.

Step 6

In the Multi-Site Orchestrator GUI, manage the sites.

1. From the left navigation menu, select Infrastructure > Sites.

2. In the main pane, change the State from Unmanaged to Managed for each fabric that you want the MSO to manage.

Removing Sites

This section describes how to disable site management for one or more sites using the Multi-Site Orchestrator GUI. The sites will remain present in the Nexus Dashboard.

Before you begin

You must ensure that all templates associated with the site you want to remove are not deployed.

Procedure

Step 1

Open the Multi-Site Orchestrator GUI.

You can open the MSO application from the Nexus Dashboard's Service Catalog. You will be automatically logged in using the Nexus Dashboard user's credentials.

Step 2

Remove the site's underlay configuration.

1. From the left navigation menu, select Infrastructure > Infra Configuration.

2. In the main pane, click Configure Infra.

3. In the left sidebar, select the site you want to unmanage.

4. In right bar, under Overlay Configuration tab, disable the ACI Multi-Site knob.

5. In the right sidebar, select the Underlay Configuration tab.

6. Remove all underlay configurations from the site.

7. Click Deploy to deploy underlay and overlay configuration changes to the site.

Step 3

In the Multi-Site Orchestrator GUI, disable the sites.

1. From the left navigation menu, select Infrastructure > Sites.

2. In the main pane, change the State from Managed to Unmanaged for each fabric that you want the MSO to stop managing.

Note

If the site is associated with one or more deployed templates, you will not be able to change its state to Unmanaged until you undeploy those templates.

Step 4

Delete the site from Nexus Dashboard.

If you no longer want to manage this site or use it with any other applications, you can delete the site from the Nexus Dashboard as well.

Note

Note that the site must not be currently in use by any of the applications installed in your Nexus Dashboard cluster.

1. From the left navigation menu of the Nexus Dashboard GUI, select Sites.

2. Select one or more sites you want to delete.

3. In the top right of the main pane, select Actions > Delete Site.

4. Provide the site's login information and click OK.

The site will be removed from the Nexus Dashboard.

Cross Launch to Fabric Controllers

Multi-Site Orchestrator currently supports a number of configuration options for each type of fabrics. For many additional configuration options, you may need to log in directly into the fabric's controller.

You can cross launch into the specific site controller's GUI from the MSO's Infrastucture > Sites screen by selecting the actions (...) menu next to the site and clicking Open in user interface. Note that cross-launch works with out-of-band (OOB) management IP of the fabric.

If your Nexus Dashboard and the fabric are configured for remote user authentication, you will be logged in automatically into the fabric's controller using the same log in information as the Nexus Dashboard user.

Chapter 5. Configuring Infra General Settings

- Infra Configuration Dashboard

- Configuring Infra: General Settings

Infra Configuration Dashboard

The Infra Configuration page displays an overview of all sites and inter-site connectivity in your Multi-Site Orchestrator deployment and contains the following information:

Infra Configuration Overview

1. The General Settings tile displays information about BGP peering type and its configuration.

This is described in detail in the next section.

2. The On-Premises tiles display information about every on-premises site that is part of your Multi-Site domain along with their number of pods and spine switches, OSPF settings, and overlay IPs.

You can click on the Pods tile that displays the number of pods in the site to show information about the Overlay Unicast TEP addresses of each Pod.

This is described in detail in Configuring Infra for Cisco APIC Sites.

3. The Cloud tiles display information about every cloud site that is part of your Multi-Site domain along with their number of regions and basic site information.

This is described in detail in Configuring Infra for Cisco Cloud APIC Sites.

The following sections describe the steps necessary to configure the general fabric Infra settings. Fabric-specific requirements and procedures are described in the following chapters based on the specific type of fabric you are managing.

Before you proceed with Infra configuration, you must have configured and added the sites as described in previous sections.

In addition, any infrastructure changes such as adding and removing spine switches or spine node ID changes require a Multi-Site Orchestrator fabric connectivity information refresh described in the Refreshing Site Connectivity Information as part of the general Infra configuration procedures.

Configuring Infra: General Settings

This section describes how to configure general Infra settings for all the sites.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left sidebar, select General Settings.

Step 5

Configure Control Plane BGP.

1. Select the Control Plane BGP tab.

2. Choose BGP Peering Type.

- full-mesh—All border gateway switches in each site will establish peer connectivity with remote sites' border gateway switches.

In full-mesh configuration, Multi-Site Orchestrator uses the spine switches for ACI managed fabrics and border gateways for DCNM managed fabrics.

- route-reflector—The route-reflector option allows you to specify one or more control-plane nodes to which each site establishes MP-BGP EVPN sessions. The use of route-reflector nodes avoids creating MP-BGP EVPN full mesh adjacencies between all the sites managed by MSO.

For ACI fabrics, the route-reflector option is effective only for fabrics that are part of the same BGP ASN.

3. In the Keepalive Interval (Seconds) field, enter the keep alive interval seconds.

We recommend keeping the default value.

4. In the Hold Interval (Seconds) field, enter the hold interval seconds.

We recommend keeping the default value.

5. In the Stale Interval (Seconds) field, enter stale interval seconds.

We recommend keeping the default value.

6. Choose whether you want to turn on the Graceful Helper option.

7. Provide the Maximum AS Limit.

We recommend keeping the default value.

8. Provide the BGP TTL Between Peers.

We recommend keeping the default value.

The following settings are for cloud sites' inter-site connectivity:

1. Provide the OSPF Area ID field.

This is OSPF area ID used by cloud sites for on-premises ISN peering, which you previously configured in the Cloud APIC for inter-site connectivity in earlier Multi-Site Orchestrator releases.

2. Click +Add IP Address to add one or more external subnet pools.

This subnet will be used to address the IPsec tunnel interfaces and loopbacks of the Cloud Routers used for on-premises connectivity, which you previously configured in the Cloud APIC for inter-site connectivity in earlier Multi-Site Orchestrator releases.

The subnets must not overlap with other on-premises TEP pools, should not begin with 0.x.x.x or 0.0.x.x, and should have a network mask between /16 and /24, for example 30.29.0.0/16.

Step 6

Provide the IPN Devices information.

When you configure inter-site underlay connectivity between on-premises and cloud sites as described in later sections, you will need to select an on-premises IPN device which will establish connectivity to the cloud CSRs. These IPN devices must first be defined here before they are available in the on-premises site configuration screen, which is described in more detail in Configuring Infra: On-Premises Site Settings.

1. Select the IPN Devices tab.

2. Click Add IPN Device.

3. Provide the Name and the IP Address of the IPN device.

The IP address you provide will be used as the tunnel peer address from the Cloud APIC's CSRs, not the IPN device's management IP address.

4. Click the check mark icon to save the device information.

5. Repeat this step for any additional IPN devices you want to add.

Chapter 6. Configuring Infra for Cisco APIC Sites

- Refreshing Site Connectivity Information

- Configuring Infra: On-Premises Site Settings

- Configuring Infra: Pod Settings

- Configuring Infra: Spine Switches

Refreshing Site Connectivity Information

Any infrastructure changes, such as adding and removing spines or changing spine node IDs, require a Multi-Site fabric connectivity site refresh. This section describes how to pull up-to-date connectivity information directly from each site's APIC.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the Main menu, select Infrastructure > Infra Configuration.

Step 3

In the top right of the main Infra Configuration view, click the Configure Infra button.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, click the Refresh button to pull fabric information from the APIC.

Step 6

(Optional) For on-premises sites, in the Confirmation dialog, check the box if you want to remove configuration for decommissioned spine switch nodes.

If you choose to enable this checkbox, all configuration info for any currently decommissioned spine switches will be removed from the database.

Step 7

Finally, click Yes to confirm and load the connectivity information.

This will discover any new or removed spines and all site-related fabric connectivity will be re-imported from the APIC.

Configuring Infra: On-Premises Site Settings

This section describes how to configure site-specific Infra settings for on-premises sites.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific on-premises site.

Step 5

Provide the Overlay Configuration.

1. In the right <Site> Settings pane, select the Overlay Configuration tab.

2. In the right <Site> Settings pane, enable the Multi-Site knob.

This defines whether the overlay connectivity is established between this site and other sites.

3. (Optional) Enable the CloudSec Encryption knob encryption for the site.

CloudSec Encryption provides inter-site traffic encryption. The "Infrastructure Management" chapter in the Cisco Multi-Site Configuration Guide covers this feature in detail.

4. Specify the Overlay Multicast TEP.

This address is used for the inter-site L2 BUM and L3 multicast traffic. This IP address is deployed on all spine switches that are part of the same fabric, regardless of whether it is a single pod or Multi-Pod fabric.

This address should not be taken from the address space of the original fabric's Infra TEP pool or from the 0.x.x.x range.

5. (Optional) From the External Routed Domain dropdown, select the domain you want to use.

Choose an external router domain that you have created in the Cisco APIC GUI. For more information, see the Cisco APIC Layer 3 Networking Configuration Guide specific to your APIC release.

6. Specify the BGP Autonomous System Number.

7. (Optional) Specify the BGP Password.

8. (Optional) Enable SR-MPLS Connectivity for the site.

If the site is connected via an MPLS network, enable the SR-MPLS Connectivity knob and provide the Segment Routing global block (SRGB) range.

The Segment Routing Global Block (SRGB) is the range of label values reserved for Segment Routing (SR) in the Label Switching Database (LSD). These values are assigned as segment identifiers (SIDs) to SR-enabled nodes and have global significance throughout the domain.

The default range is 16000-23999.

If you enable MPLS connectivity for the site, you will need to configure additional settings as described in the "Sites Connected via SR-MPLS" chapter of the Cisco Multi-Site Configuration Guide for ACI Fabrics.

Step 6

Provide the Underlay Configuration.

1. In the right <Site> Settings pane, select the Underlay Configuration tab.

2. Select the OSPF Area Type from the dropdown menu.

The OSPF area type can be one of the following:

- nssa

- regular

3. Configure OSPF settings for the site.

You can either click an existing policy (for example, msc-ospf-policy-default ) to modify it or click +Add Policy to add a new OSPF policy. Then in the Add/Update Policy window, specify the following:

- In the Policy Name field, enter the policy name.

- In the Network Type field, choose either broadcast, point-to-point, or unspecified.

The default is broadcast.

- In the Priority field, enter the priority number.

The default is 1.

- In the Cost of Interface field, enter the cost of interface.

The default is 0.

- From the Interface Controls dropdown menu, choose one of the following:

- advertise-subnet

- bfd

- mtu-ignore

- passive-participation

- In the Hello Interval (Seconds) field, enter the hello interval in seconds.

The default is 10.

- In the Dead Interval (Seconds) field, enter the dead interval in seconds.

The default is 40.

- In the Retransmit Interval (Seconds) field, enter the retransmit interval in seconds.

The default is 5.

- In the Transmit Delay (Seconds) field, enter the transmit delay in seconds.

The default is 1.

Step 7

Configure inter-site connectivity between on-premises and cloud sites.

If you do not need to create inter-site connectivity between on-premises and cloud sites, for example if your deployment contains only cloud or only on-premises sites, skip this step.

When you configure underlay connectivity between on-premises and cloud sites, you need to provide an IPN device IP address to which the Cloud APIC's CSRs establish a tunnel and then configure the cloud site's infra settings.

1. Click +Add IPN Device to specify an IPN device.

2. From the dropdown, select one of the IPN devices you defined previously.

The IPN devices must be already defined in the General Settings > IPN Devices list, as described in Configuring Infra: General Settings

3. Configure inter-site connectivity for cloud sites.

Any previously configured connectivity from the cloud sites to this on-premises site will be displayed here, but any additional configuration must be done from the cloud site's side as described in Configuring Infra for Cisco Cloud APIC Sites.

What to do next

While you have configured all the required inter-site connectivity information, it has not been pushed to the sites yet. You need to deploy the configuration as described in Deploying Infra Configuration

Configuring Infra: Pod Settings

This section describes how to configure pod-specific settings in each site.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the Main menu, click Sites.

Step 3

In the Sites view, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, select a pod.

Step 6

In the right POD Properties pane, add the Overlay Unicast TEP for the POD.

This IP address is deployed on all spine switches that are part of the same pod and used for intersite known unicast traffic.

Step 7

Click +Add TEP Pool to add an external routable TEP pool.

The external routable TEP pools are used to assign a set of IP addresses that are routable across the ISN to APIC nodes, spine switches, and border leaf nodes. This is required to enable the intersite L3Out functionality.

External TEP pools previously assigned to the fabric on APIC are automatically inherited by MSO and displayed in the GUI when the fabric is added to the Multi-Site domain.

Step 8

Repeat the procedure for every pod in the site.

Configuring Infra: Spine Switches

This section describes how to configure spine switches in each site for Cisco Multi-Site.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the Main menu, click Sites.

Step 3

In the Sites view, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, select a spine switch within a pod.

Step 6

In the right <Spine> Settings pane, click +Add Port.

Step 7

In the Add Port window, enter the following information:

- In the Ethernet Port ID field, enter the port ID, for example 1/29.

- In the IP Address field, enter the IP address/netmask.

MSO creates a sub-interface with VLAN 4 with the specified IP ADDRESS under the specified PORT.

- In the MTU field, enter the MTU. You can specify either inherit, which would configure an MTU of 9150B, or choose a value between 576 and 9000.

MTU of the spine port should match MTU on IPN side.

- In the OSPF Policy field, choose the OSPF policy for the switch that you have configured in Configuring Infra: On-Premises Site Settings.

OSPF settings in the OSPF policy you choose should match on IPN side.

- For OSPF Authentication, you can pick either none or one of the following:

- MD5

- Simple

Step 8

Enable BGP Peering knob.

In a single Pod fabric with more than two spine switches, BGP peering should only be enabled on a pair (for redundancy) of spine switches called BGP Speakers. All other spine switches should have BGP peering disabled and will function as BGP Forwarders.

In a Multi-Pod fabric BGP peering should only be enabled on a couple of BGP speaker spine switches, each deployed in a different Pod. All other spines switches should have BGP peering disabled and function as BGP forwarders.

Step 9

In the BGP-EVPN Router-ID field, provide the IP address used for BGP-eVPN session between sites.

Step 10

Repeat the procedure for every spine switch.

Chapter 7. Configuring Infra for Cisco Cloud APIC Sites

- Refreshing Cloud Site Connectivity Information

- Configuring Infra: Cloud Site Settings

Refreshing Cloud Site Connectivity Information

Any infrastructure changes, such as CSR and Region addition or removal, require a Multi-Site fabric connectivity site refresh. This section describes how to pull up-to-date connectivity information directly from each site's APIC.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the Main menu, select Infrastructure > Infra Configuration.

Step 3

In the top right of the main Infra Configuration view, click the Configure Infra button.

Step 4

In the left pane, under Sites, select a specific site.

Step 5

In the main window, click the Refresh button to discover any new or changed CSRs and regions.

Step 6

Finally, click Yes to confirm and load the connectivity information.

This will discover any new or removed CSRs and regions.

Step 7

Click Deploy to propagate the cloud site changes to other sites that have connectivity to it.

After you refresh a cloud site's connectivity and CSRs or regions are added or removed, you need to deploy infra configuration so other sites that have underlay connectivity to that cloud site get updated configuration.

Configuring Infra: Cloud Site Settings

This section describes how to configure site-specific Infra settings for Cloud APIC sites.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the Main menu, select Infrastructure > Infra Configuration.

Step 3

In the top right of the main pane, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific cloud site.

Step 5

Provide the Overlay Configuration.

1. In the right <Site> Settings pane, select the Overlay Configuration tab.

2. In the right <Site> Settings pane, enable the Multi-Site knob.

This defines whether the overlay connectivity is established between this site and other sites.

3. (Optional) Specify the BGP Password.

Step 6

Provide the Underlay Configuration.

1. In the right <Site> Settings pane, select the Underlay Configuration tab.

2. Click Add Connectivity.

3. From the Site dropdown, select the site to which you want to establish connectivity.

4. From the Connection Type dropdown, choose the type of connection between the sites.

The following options are available:

- Public Internet—connectivity between the two sites is established via the Internet.

This type is supported between any two cloud sites or between a cloud site and an on-premises site.

- Private Connection—connectivity is established using a private connection between the two sites.

This type is supported between a cloud site and an on-premises site.

- Cloud Backbone—connectivity is established using cloud backbone.

This type is supported between two cloud sites of the same type, such as Azure-to-Azure or AWS-to-AWS.

If you have multiple types of sites (on-premises, AWS, and Azure), different pairs of site can use different connection type.

5. (Optional) Enable IPsec.

The following options are available:

- For Public Internet connectivity, IPsec is always enabled.

- For Cloud Backbone connectivity, IPsec is always disabled.

- For Private Connection, you can choose to enable or disable IPsec.

6. If IPsec is enabled, choose the IKE Version for it.

Internet Key Exchange (IKE) is a protocol used to establish security association for IPsec. You can choose which version of the protocol to use: IKEv1 (Version 1) or IKEv2 (Version 1) depending on your configuration.

7. Click Save to save the inter-site connectivity configuration.

When you save connectivity information from Site1 to Site2, the reverse connectivity is automatically created from Site2 to Site1, which you can see by selecting the other site and checking the Underlay Configuration tab.

8. Repeat this step to add inter-site connectivity for other sites.

When you establish underlay connectivity from Site1 to Site2, the reverse connetivity is done automatically for you.

However, if you also want to establish inter-site connectivity from Site1 to Site3, you must repeat this step for that site as well.

What to do next

While you have configured all the required inter-site connectivity information, it has not been pushed to the sites yet. You need to deploy the configuration as described in Deploying Infra Configuration

Chapter 8. Deploying Infra Configuration for ACI Sites

- Deploying Infra Configuration

- Enabling Connectivity Between On-Premises and Cloud Sites

Deploying Infra Configuration

This section describes how to deploy the Infra configuration to each APIC site.

Procedure

Step 1

In the top right of the main pane, click Deploy and choose the appropriate option to deploy the configuration.

If you have configured only on-premises or only cloud sites, simply click Deploy to deploy the Infra configuration.

However, if you have both, on-premises and cloud site, the following two additional options become available:

- Deploy & Download IPN Device config files: Pushes the configuration to both the on-premises APIC site and the Cloud APIC site and enables the end-to-end interconnect between the on-premises and the cloud sites.

In addition, this option downloads a zip file that contains configuration information that you will use to enable connectivity between the Cisco Cloud Services Router (CSR) deployed in your cloud sites and the on-premises IPsec termination device. A followup screen appears that allows you to select all or some of the configuration files to download.

- Download IPN Device config files only: Downloads a zip file that contains configuration information that you will use to enable connectivity between the Cisco Cloud Services Router (CSR) without deploying the configuration.

Step 2

In the confirmation window, click Yes.

The Deployment started, refer to left menu for individual site deployment status message will indicate that Infra configuration deployment began and you can verify each site's progress by the icon displayed next to the site's name in the left pane.

What to do next

The Infra overlay and underlay configuration settings are now deployed to all sites' controllers and cloud CSRs. The last remaining step is to configure your IPN devices with the tunnels for cloud CSRs as descrbied in Refreshing Site Connectivity Information.

Enabling Connectivity Between On-Premises and Cloud Sites

If you have only on-premises or only cloud sites, you can skip this section.

This section describes how to enable connectivity between on-premises APIC sites and Cloud APIC sites.

By default, the Cisco Cloud APIC will deploy a pair of redundant Cisco Cloud Services Router 1000Vs. The procedures in this section creates two tunnels, one IPsec tunnel from the on-premises IPsec device to each of these Cisco Cloud Services Router 1000Vs. If you have multiple on-premises IPsec devices, you will need to configure the same tunnels to the CSRs on each of the on-premises devices.

The following information provides commands for Cisco Cloud Services Router 1000V as your on-premises IPsec termination device. Use similar commands if you are using a different device or platform.

Procedure

Step 1

Gather the necessary information that you will need to enable connectivity between the CSRs deployed in the cloud site and the on-premises IPsec termination device.

You can get the required configuration details using either the Deploy & Download IPN Device config files or the Download IPN Device config files only option in Multi-Site Orchestrator as part of the procedures provided in Deploying Infra Configuration.

Step 2

Log into the on-premises IPsec device.

Step 3

Configure the tunnel for the first CSR.

Details for the first CSR are available in the configuration files for the ISN devices you downloaded from the Multi-Site Orchestrator, but the following fields describe the important values for your specific deployment:

- <first-csr-tunnel-ID>—unique tunnel ID that you assign to this tunnel.

- <first-csr-ip-address>—public IP address of the third network interface of the first CSR.

The destination of the tunnel depends on the type of underlay connectivity:

- The destination of the tunnel is the public IP of the cloud router interface if the underlay is via public internet

- The destination of the tunnel is the private IP of the cloud router interface if the underlay is via private connectivity, such as DX on AWS or ER on Azure

- <first-csr-preshared-key>—preshared key of the first CSR.

- <onprem-device-interface>—interface that is used for connecting to the Cisco Cloud Services Router 1000V deployed in Amazon Web Services.

- <onprem-device-ip-address>—IP address for the <interface> interface that is used for connecting to the Cisco Cloud Services Router 1000V deployed in Amazon Web Services.

- <peer-tunnel-for-onprem-IPsec-to-first-CSR>—peer tunnel IP address for the on-premises IPsec device to the first cloud CSR.

- <process-id> —OSPF process ID.

- <area-id>—OSPF area ID.

The following example shows intersite connectivity configuration using the IKEv2 protocol supported starting with Multi-Site Orchestrator, Release 3.3(1) and Cloud APIC, Release 5.2(1). If you are using IKEv1, the IPN configuration file you downloaded form MSO may look slightly differently, but the principle remains the same.

crypto ikev2 proposal ikev2-proposal-default
encryption aes-cbc-256 aes-cbc-192 aes-cbc-128
integrity sha512 sha384 sha256 sha1
group 24 21 20 19 16 15 14 2
exit

crypto ikev2 policy ikev2-policy-default
proposal ikev2-proposal-default
exit

crypto ikev2 keyring key-ikev2-infra:overlay-1-<first-csr-tunnel-id>
peer peer-ikev2-keyring
address <first-csr-ip-address>
pre-shared-key <first-csr-preshared-key>
exit
exit

crypto ikev2 profile ikev2-infra:overlay-1-<first-csr-tunnel-id>
match address local interface <onprem-device-interface>
match identity remote address <first-csr-ip-address> 255.255.255.255
identity local address <onprem-device-ip-address>
authentication remote pre-share
authentication local pre-share
keyring local key-ikev2-infra:overlay-1-<first-csr-tunnel-id>
lifetime 3600
dpd 10 5 on-demand
exit

crypto ipsec transform-set infra:overlay-1-<first-csr-tunnel-id> esp-gcm 256
mode tunnel
exit

crypto ipsec profile infra:overlay-1-<first-csr-tunnel-id>
set pfs group14
set ikev2-profile ikev2-infra:overlay-1-<first-csr-tunnel-id>
set transform-set infra:overlay-1-<first-csr-tunnel-id>
exit

interface tunnel 2001
ip address <peer-tunnel-for-onprem-IPsec-to-first-CSR> 255.255.255.252
ip virtual-reassembly
tunnel source <onprem-device-interface>
tunnel destination <first-csr-ip-address>
tunnel mode ipsec ipv4
tunnel protection ipsec profile infra:overlay-1-<first-csr-tunnel-id>
ip mtu 1400
ip tcp adjust-mss 1400
ip ospf <process-id> area <area-id>
no shut
exit

Example:

crypto ikev2 proposal ikev2-proposal-default
encryption aes-cbc-256 aes-cbc-192 aes-cbc-128
integrity sha512 sha384 sha256 sha1
group 24 21 20 19 16 15 14 2
exit

crypto ikev2 policy ikev2-policy-default
proposal ikev2-proposal-default
exit

crypto ikev2 keyring key-ikev2-infra:overlay-1-2001
peer peer-ikev2-keyring
address 52.12.232.0
pre-shared-key 1449047253219022866513892194096727146110
exit
exit

crypto ikev2 profile ikev2-infra:overlay-1-2001
! Please change GigabitEthernet1 to the appropriate interface
match address local interface GigabitEthernet1
match identity remote address 52.12.232.0 255.255.255.255
identity local address 128.107.72.62
authentication remote pre-share
authentication local pre-share
keyring local key-ikev2-infra:overlay-1-2001
lifetime 3600
dpd 10 5 on-demand
exit

crypto ipsec transform-set infra:overlay-1-2001 esp-gcm 256
mode tunnel
exit

crypto ipsec profile infra:overlay-1-2001
set pfs group14
set ikev2-profile ikev2-infra:overlay-1-2001
set transform-set infra:overlay-1-2001
exit

! These tunnel interfaces establish point-to-point connectivity between the on-prem device and the cloud Routers
! The destination of the tunnel depends on the type of underlay connectivity:
! 1) The destination of the tunnel is the public IP of the cloud Router interface if the underlay is via internet
! 2) The destination of the tunnel is the private IP of the cloud Router interface if the underlay is via private
connectivity like DX on AWS or ER on Azure

interface tunnel 2001
ip address 5.5.1.26 255.255.255.252
ip virtual-reassembly
! Please change GigabitEthernet1 to the appropriate interface
tunnel source GigabitEthernet1
tunnel destination 52.12.232.0
tunnel mode ipsec ipv4
tunnel protection ipsec profile infra:overlay-1-2001
ip mtu 1400
ip tcp adjust-mss 1400
! Please update process ID according with your configuration
ip ospf 1 area 0.0.0.1
no shut
exit

Step 4

Repeat the previous step for the 2nd and any additional CSRs that you need to configure.

Step 5

Verify that the tunnels are up on your on-premises IPsec device.

Use the following command to display the status. If you do not see that both tunnels are shown as up, verify the information that you entered in the steps in this section to determine where you might have an issue. Do not proceed to the next section until you see that both tunnels are shown as up.

ISN_CSR# show ip interface brief | include Tunnel
Interface              IP-Address      OK? Method Status                Protocol
Tunnel1000             30.29.1.2         YES manual up                    up
Tunnel1001             30.29.1.4         YES manual up                    up

Part II. Day-0 Operations for DCNM Fabrics

- Adding and Deleting Sites

- Configuring Infra for Cisco DCNM Sites

Chapter 9. Adding and Deleting Sites

- Adding Cisco DCNM Sites

- Removing Sites

- Cross Launch to Fabric Controllers

Adding Cisco DCNM Sites

This section describes how to add a DCNM site using the Nexus Dashboard GUI and then enable that site to be managed by Multi-Site Orchestrator.

Before you begin

- You must ensure that the site(s) you are adding are running Cisco DCNM, Release 11.5(1) or later.

Procedure

Step 1

Log in to the Nexus Dashboard GUI

Step 2

Add a new site.

1. From the left navigation menu, select Sites.

2. In the top right of the main pane, select Actions > Add Site.

Step 3

Provide site information.

1. For Site Type, select DCNM.

2. Provide the DCNM controller information.

You need to provide the Host Name/IP Address of the in-band (eth2) interface, User Name, and Password. for the DCNM controller currently managing your DCNM fabrics.

3. Click Select Sites to select the specific fabrics managed by the DCNM controller.

The fabric selection window will open.

Step 4

Select the fabrics you want to add to the Nexus Dashboard.

1. Check one or more fabrics that you want to be available to the applications running in your Nexus Dashboard.

2. Click Select.

Step 5

In the Add Site window, click Add to finish adding the sites.

At this time, the sites will be available in the Nexus Dashboard, but you still need to enable them for Multi-Site Orchestrator management as described in the folowing steps.

Step 6

Repeat the previous steps for any additional DCNM controllers.

Step 7

From the Nexus Dashboard's Service Catalog, open the Multi-Site Orchestrator application.

You will be automatically logged in using the Nexus Dashboard user's credentials.

Step 8

In the Multi-Site Orchestrator GUI, manage the sites.

1. From the left navigation menu, select Infrastructure > Sites.

2. In the main pane, change the State from Unmanaged to Managed for each fabric that you want the MSO to manage.

If the fabric you are managing is part of a DCNM Multi-Site Domain (MSD), it will have a Site ID already associated with it. In this case, simply changing the State to Managed will manage the fabric.

However, if the fabric is not part of a DCNM MSD, you will also be prompted to provide a Fabric ID for the site when you change its state to Managed.

Note

If you want to manage both kinds of fabrics, those that are part of an existing MSD and those that are not, you must on-board the MSD fabrics first, followed by any standalone fabrics.

Removing Sites

This section describes how to disable site management for one or more sites using the Multi-Site Orchestrator GUI. The sites will remain present in the Nexus Dashboard.

Before you begin

You must ensure that all templates associated with the site you want to remove are not deployed.

Procedure

Step 1

Open the Multi-Site Orchestrator GUI.

You can open the MSO application from the Nexus Dashboard's Service Catalog. You will be automatically logged in using the Nexus Dashboard user's credentials.

Step 2

Remove the site's underlay configuration.

1. From the left navigation menu, select Infrastructure > Infra Configuration.

2. In the main pane, click Configure Infra.

3. In the left sidebar, select the site you want to unmanage.

4. In right bar, under Overlay Configuration tab, disable the ACI Multi-Site knob.

5. In the right sidebar, select the Underlay Configuration tab.

6. Remove all underlay configurations from the site.

7. Click Deploy to deploy underlay and overlay configuration changes to the site.

Step 3

In the Multi-Site Orchestrator GUI, disable the sites.

1. From the left navigation menu, select Infrastructure > Sites.

2. In the main pane, change the State from Managed to Unmanaged for each fabric that you want the MSO to stop managing.

Note

If the site is associated with one or more deployed templates, you will not be able to change its state to Unmanaged until you undeploy those templates.

Step 4

Delete the site from Nexus Dashboard.

If you no longer want to manage this site or use it with any other applications, you can delete the site from the Nexus Dashboard as well.

Note

Note that the site must not be currently in use by any of the applications installed in your Nexus Dashboard cluster.

1. From the left navigation menu of the Nexus Dashboard GUI, select Sites.

2. Select one or more sites you want to delete.

3. In the top right of the main pane, select Actions > Delete Site.

4. Provide the site's login information and click OK.

The site will be removed from the Nexus Dashboard.

Cross Launch to Fabric Controllers

Multi-Site Orchestrator currently supports a number of configuration options for each type of fabrics. For many additional configuration options, you may need to log in directly into the fabric's controller.

You can cross launch into the specific site controller's GUI from the MSO's Infrastucture > Sites screen by selecting the actions (...) menu next to the site and clicking Open in user interface. Note that cross-launch works with out-of-band (OOB) management IP of the fabric.

If your Nexus Dashboard and the fabric are configured for remote user authentication, you will be logged in automatically into the fabric's controller using the same log in information as the Nexus Dashboard user.

Chapter 10. Configuring Infra for Cisco DCNM Sites

- Prerequisites and Guidelines

- Configuring Infra: General Settings

- Refreshing Site Connectivity Information

- Configuring Infra: DCNM Site Settings

- Deploying Infra Configuration

Prerequisites and Guidelines

The following sections describe the steps necessary to configure the general as well as site-specific fabric Infra settings.

Before you proceed with Infra configuration, you must have added the sites as described in previous sections.

In addition, keep in mind the following:

- Adding or removing border gateway switches requires a Multi-Site Orchestrator fabric connectivity information refresh described in the Refreshing Site Connectivity Information as part of the general Infra configuration procedures.

Configuring Infra: General Settings

This section describes how to configure general Infra settings for all the sites.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left sidebar, select General Settings.

Step 5

Configure Control Plane BGP.

1. Select the Control Plane BGP tab.

2. Choose BGP Peering Type.

- full-mesh—All border gateway switches in each site will establish peer connectivity with remote sites' border gateway switches.

- route-server—The route-server option allows you to specify one or more control-plane nodes to which each site establishes MP-BGP EVPN sessions. The route-server nodes perform a function similar to traditional BGP route-reflectors, but for EBGP (and not iBGP) sessions. The use of route-server nodes avoids creating MP-BGP EVPN full mesh adjacencies between all the VXLAN EVPN sites managed by MSO.

3. If you set the BGP Peering Type to route-server, click +Add Route Server to add one or more route servers.

In the Add Route Server window that opens:

- From the Site dropdown, select the site you want to connect to the route server.

- The ASN field will be auto-populated with the site's ASN.

- From the Core Router Device dropdown, select the route server to which you want to connect.

- From the Interface dropdown, select the interface on the core router device.

You can add up to 4 route servers. If you add multiple route servers, every site will establish MP-BGP EVPAN adjacencies to every route server.

4. Leave the Keepalive Interval (Seconds), Hold Interval (Seconds), Stale Interval (Seconds), Graceful Helper, Maximum AS Limit, and BGP TTL Between Peers fields at default values as they are relevant for Cisco ACI fabrics only.

5. Skip the OSPF Area ID and External Subnet Pool fields at default values as they are relevant for Cisco Cloud ACI fabrics only.

Step 6

Skip the IPN Devices tab settings.

The settings under the IPN Devices tab are for Cisco ACI inter-site connectivity between on-premises APIC and Cloud APIC sites. You can skip these settings when managing Cisco DCNM sites only.

Step 7

Configure DCNM Settings.

1. Select the DCNM Settings tab.

2. Provide the L2 VXLAN VNI Range.

3. Provide the L3 VXLAN VNI Range.

4. Provide the Multi-Site Routing Loopback IP Range.

This field is used to auto-populate the Multi-Site TEP field for each fabric, which is described in Configuring Infra: DCNM Site Settings.

For sites that were previously part of a Multi-Site Domain (MSD) in DCNM, this field will be pre-populated with the previously defined value.

5. Provide the Anycast Gateway MAC.

Refreshing Site Connectivity Information

Infrastructure changes, such as adding and removing border gateway switches, require a Multi-Site Orchestrator fabric connectivity refresh. This section describes how to pull up-to-date connectivity information directly from each site's controller.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left sidebar, under Sites, select a specific site.

Step 5

In the main window, click the Refresh button to pull fabric information from the controller.

Step 6

(Optional) In the Confirmation dialog, check the box if you want to remove configuration for decommissioned border gateway switches.

If you choose to enable this checkbox, all configuration info for any currently decommissioned border gateway switches will be removed from the database.

Step 7

Finally, click Yes to confirm and load the connectivity information.

This will discover any new or removed spines and all site-related fabric connectivity will be re-imported from the site's controller.

Configuring Infra: DCNM Site Settings

This section describes how to configure site-specific Infra settings for on-premises sites.

Procedure

Step 1

Log in to the Cisco Multi-Site Orchestrator GUI.

Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific DCNM.

Step 5

In the right <Site> Settings sidebar, specify the Multi-Site VIP.

This address is used for the inter-site L2 BUM and L3 multicast traffic. This IP address is deployed on all border gateway switches that are part of the same fabric.

Note

If the site you are configuring is part of the DCNM Multi-Site Domain (MDS), this field will be pre-populated with the information imported from DCNM. In this case, changing the value and re-deploying the infra configuration, will impact traffic between the sites that are part of the MDS.

You can choose to Auto Allocate this field, which will allocate the next available address from the Multi-Site Routing Loopback IP Range you defined in previous section.

Step 6

Within the <fabric-name> tile, select the border gateway.

Step 7

In the right <border-gateway> setting sidebar, specify the BGP-EVPN ROUTER-ID and BGW PIP.

For border gateways that are part of a vPC domain, you must also specify a VPC VIP

Step 8

Click Add Port to configure the port that connects to the IPN.

Note

This release does not support importing the port configuration from the DCNM. If the site you are configuring is already part of the DCNM Multi-Site Domain (MDS), you must use the same values that are already configured in DCNM.

Provide the following information specific to your deployment for the port that connects this border gateway to a core switch or another border gateway:

- From the Ethernet Port ID dropdown, select the port that connects to the IPN.

- In the IP Address field, enter the IP address and netmask.

- In the Remote Address field, provide the IP address of the remote device to which the port is connected.

- In the Remote ASN field, provide the remote site's ID.

- In the MTU field, enter the port's MTU.

MTU of the spine port should match MTU on IPN side.

You can specify either inherit or a value between 576 and 9000.

- For BGP Authentication, you can pick either None or Simple (MD5).

If you select Simple, you must also provide the Authentication Key.

Deploying Infra Configuration

This section describes how to deploy the Infra configuration to each DCNM site.

Before you begin

You must have the general and site-specific infra configurations completed as described in previous sections of this chapter.

Procedure

Step 1

Ensure that there are no configuration conflicts or resolve them if necessary.

The Deploy button will be disabled and a warning will be displayed if there are any configuration conflicts from the already configured settings in each site. For example, if a VRF or network with the same name exists in multiple sites but uses different VNI in each site.

In case of configuration conflicts:

1. Click Click to View link in the conflict notification pop-up.

2. Note down the specific configurations that are causing the conflicts.

For example, in the following report, there are ID mismatches between VRFs and networks in fab1 and fab2 sites.

3. Click the X button to close the report, then exit Infra configuration screen.

4. Unmanage the site in MSO, as described in Removing Sites.

You do not need to remove the site from the Nexus Dashboard, simply unmanage it in MSO GUI.

5. Resolve the existing configuration conflicts.

6. Manage the site again, as described in Adding Cisco DCNM Sites.

Since the site is already added in Nexus Dashboard, simply enable it for management in MSO.

7. Verify that all conflicts are resolved and the Deploy button is available.

Step 2

Deploy configuration.

1. In the top right of the Fabric Connectivity Infra screen, choose the appropriate Deploy option to deploy the configuration.

If you are configuring only DCNM sites, simply click Deploy to deploy the Infra configuration.

2. Wait for configuration to be deployed.

When you deploy infra configuration, MSO will signal the DCNM to configure the underlay and the EVPN overlay between the border gateways.

When configuration is successfully deployed, you will see a green checkmark next to the site in the Fabric Connectivity Infra screen:

Part III. Upgrading Multi-Site Orchestrator

- Upgrading or Downgrading MSO Application

- Migrating Existing Cluster to Nexus Dashboard

Chapter 11. Upgrading or Downgrading MSO Application

- Overview

- Prerequisites and Guidelines

- Upgrading MSO Application Using Cisco App Store

- Upgrading MSO Application Manually

Overview

The following sections describe how to upgrade or downgrade Cisco Multi-Site Orchestrator, Release 3.2(1) or later that is deployed in Cisco Nexus Dashboard.

If you are running an earlier release deployed in VMware ESX VMs or Cisco Application Services Engine, you must deploy a brand new cluster and then transfer the configuration from your existing cluster, as described in the "Migrating Existing Cluster to Nexus Dashboard" chapter of the Multi-Site Orchestrator Deployment Guide.

Prerequisites and Guidelines

Before you upgrade or downgrade your Cisco Multi-Site Orchestrator cluster:

- Stateful upgrades from releases prior to Release 3.2(1) are not supported.

If you are upgrading from an earlier release, skip the rest of this chapter and follow the instructions described in the "Migrating Existing Cluster to Nexus Dashboard" section of the Multi-Site Orchestrator Deployment Guide.

- Ensure that your current Nexus Dashboard cluster is healthy.

You can check the Nexus Dashboard cluster health in one of two ways:

- By logging into your Nexus Dashboard GUI and verifying system status in the System Overview page.

- By logging into any one of the nodes directly as rescue-user and running the following command:

# acs health
All components are healthy

- Ensure that your current Cisco Multi-Site Orchestrator is running properly.

You can check the status of you Multi-Site Orchestrator application by navigating to Settings > System Status:

Then ensure that the status of all nodes and services is healthy:

- You can upgrade the MSO application in one of two ways:

- Using the Nexus Dashboard's App Store, as described in Upgrading MSO Application Using Cisco App Store.

In this case, the Cisco DC App Center must be reachable from the Nexus Dashboard via the Management Network directly or using a proxy configuration. Nexus Dashboard proxy configuration is described in the Nexus Dashboard User Guide.

Note

The App Store allows you to upgrade to the latest available version of the application only.

- By manually uploading the new app image, as described in Upgrading MSO Application Manually.

You can use this approach if you are unable to establish the connection to the DC App Center or if you want to upgrade to a version of the application that is not the latest available release.

- If you plan to add and manage new Cloud APIC sites after you upgrade your Multi-Site Orchestrator to release 3.3(1) or later, you must ensure that they are running Cloud APIC release 5.2(1) or later.

On-boarding and managing Cloud APIC sites running earlier releases is not support in Multi-Site Orchestrator 3.3(1).

- Downgrading to releases prior to release 3.3(1) is not supported.

If you want to downgrade to an earlier release, you must deploy a new Multi-Site Orchestrator cluster on a platform supported by the earlier release, then restore the older configuration backup. Restoring backups created on Release 3.3(1) or later to an older MSO cluster is not supported.

If you downgrade to an earlier release of Multi-Site Orchestrator, you must also downgrade all Cloud APIC sites to a release prior to Release 5.2(1).

Upgrading MSO Application Using Cisco App Store

This section describes how to upgrade Cisco Multi-Site Orchestrator, Release 3.2(1) or later.

Before you begin

- Ensure that you have completed the prerequisites described in Prerequisites and Guidelines.

- Ensure that Cisco DC App Center is reachable from the Nexus Dashboard via the Management Network directly or using a proxy configuration.

Nexus Dashboard proxy configuration is described in the Nexus Dashboard User Guide

Procedure

Step 1

Log in to your Nexus Dashboard..

Step 2

From the left navigation menu, select Service Catalog.

Step 3

Upgrade the application using the App Store.

1. In the Service Catalog screen, select the App Store tab.

2. In the Multi-Site Orchestrator tile, click Upgrade.

3. In the License Agreement window that opens, click Agree and Download.

Step 4

Wait for the new image to initialize.

It may take up to 20 minutes for the new application image to become available.

Step 5

Activate the new image.

1. In the Service Catalog screen, select the Installed Services tab.

2. In the top right of the Multi-Site Orchestrator tile, click the menu (...) and choose Available Versions.

3. In the available versions window, click Activate next to the new image.

Note

Do not Disable the currently running image before activating the new image. The image activation process will recognize the currently running image and perform the upgrade workflows necessary for the currently running app version.

It may take up to 20 additional minutes for all the application services to start and the GUI to become available. The page will automatically reload when the process is completed.

Step 6

(Optional) Delete the old application image.

You can choose to retain the old application version in case you ever want to downgrade. Or you can delete it as described in this step.

1. In the Service Catalog screen, select the Installed Services tab.

2. In the top right of the Multi-Site Orchestrator tile, click the menu (...) and choose Available Versions.

3. In the available versions window, click the delete icon next to the image you want to delete.

It may take up to 20 additional minutes for all the application services to start and the GUI to become available. The page will automatically reload when the process is completed.

Step 7

Launch the app.

To launch the app, simply click Open on the application tile in the Nexus Dashboard's Service Catalog page.

The single sign-on (SSO) feature allows you to log in to the application using the same credentials as you used for the Nexus Dashboard.

Upgrading MSO Application Manually

This section describes how to upgrade Cisco Multi-Site Orchestrator, Release 3.2(1) or later.

Before you begin

- Ensure that you have completed the prerequisites described in Prerequisites and Guidelines.

Procedure

Step 1

Download the target release image.

1. Browse to the Multi-Site Orchestrator page on DC App Center:

https://dcappcenter.cisco.com/nexus-dashboard-orchestrator.html

2. From the Version drop-down, choose the version you want to install and click Download.

3. Click Agree and download to accept the license agreement and download the image.

Step 2

Log in to your Nexus Dashboard..

Step 3

Upload the image to your Nexus Dashboard.

1. From the left navigation menu, select Service Catalog.

2. In the Nexus Dashboard's Service Catalog screen, select the Installed Services tab.

3. From the Actions menu in the top right of main pane, select Upload App.

4. In the Upload App window, choose the location of the image

If you downloaded the application image to your system, choose Local.

If you are hosting the image on a server, choose Remote.

5. Choose the file.

If you chose Local in the previous substep, click Select File and select the app image you downloaded.

If you chose Remote, provide the full URL to the image file, for example http://<ip-address>:<port>/<full-path>/cisco-mso-<version>.aci.

6. Click Upload to add the app to the cluster.

A new tile will appear with the upload progress bar. Once the image upload is completed, the Nexus Dashboard will recognize the new image as an existing application and add it as a new version.

Step 4

Wait for the new image to initialize.

It may take up to 20 minutes for the new application image to become available.

Step 5

Activate the new image.

1. In the Service Catalog screen, select the Installed Services tab.

2. In the top right of the Multi-Site Orchestrator tile, click the menu (...) and choose Available Versions.

3. In the available versions window, click Activate next to the new image.

Note

Do not Disable the currently running image before activating the new image. The image activation process will recognize the currently running image and perform the upgrade workflows necessary for the currently running app version.

It may take up to 20 additional minutes for all the application services to start and the GUI to become available. The page will automatically reload when the process is completed.

Step 6

(Optional) Delete the old application image.

You can choose to retain the old application version in case you ever want to downgrade. Or you can delete it as described in this step.

1. In the Service Catalog screen, select the Installed Services tab.

2. In the top right of the Multi-Site Orchestrator tile, click the menu (...) and choose Available Versions.

3. In the available versions window, click the delete icon next to the image you want to delete.

It may take up to 20 additional minutes for all the application services to start and the GUI to become available. The page will automatically reload when the process is completed.

Step 7

Launch the app.

To launch the app, simply click Open on the application tile in the Nexus Dashboard's Service Catalog page.

The single sign-on (SSO) feature allows you to log in to the application using the same credentials as you used for the Nexus Dashboard.

Chapter 12. Migrating Existing Cluster to Nexus Dashboard

- Overview

- Prerequisites and Guidelines

- Back Up Existing Cluster Configuration

- Prepare New Cluster

- Restore Configuration in the New Cluster

- Upgrade Cloud Sites

- Update MSO Infra Configuration

- Resolve Configuration Drifts

Overview

This release of Multi-Site Orchestrator must be deployed as an application in Cisco Nexus Dashboard. The previously supported VMware ESX virtual appliance and Cisco Application Services Engine form factors are now deprecated.

The following sections describe how to migrate an existing Cisco Multi-Site Orchestrator to Release 3.3(1) on Nexus Dashboard platform.

If your MSO cluster is already deployed in Nexus Dashboard, follow the steps described in Upgrading or Downgrading MSO Application instead.

Migration Workflow

The following list provides a high level overview of the migration process and the order of tasks you will need to perform.

A video demonstrating the MSO-specific steps is available at Migrating from MSO 3.1 to MSO 3.3 on Nexus Dashboard. Note that the video does not replace a complete list of requirements and steps listed in this chapter, such as Nexus Dashboard deployment and Cloud APIC site upgrades.

- Back up existing Multi-Site Orchestrator configuration and disconnect or bring down the existing MSO cluster.

If you deploy a brand new Nexus Dashboard cluster rather than upgrade an existing cluster, we recommend preserving the existing MSO cluster until the new cluster is deployed and configuration is restored.

- Deploy a Nexus Dashboard cluster using physical, virtual, or cloud form factor.

- (Optional) Configure the Nexus Dashboard cluster with additional nodes if required for application co-hosting.

- (Optional) Configure remote authentication servers in the Nexus Dashboard if required by your existing Multi-Site Orchestrator deployment.

- On-board the APIC, Cloud APIC, or DCNM sites that you currently manage from the Multi-Site Orchestrator to the Nexus Dashboard.

- Install the Multi-Site Orchestrator application in the Nexus Dashboard.

- Restore the configuration backup in the new MSO application installed in the Nexus Dashboard.

- Upgrade cloud sites to Cloud APIC release 5.2(x) one site at a time.

You will upgrade a site's Cloud APIC, then that site's CSRs, then repeat the procedure for each additional site.

- Update Infra configuration settings in Multi-Site Orchestrator.

Prerequisites and Guidelines

Because the new platform is vastly different in how it implements clustering and infrastructure, site management, and user management, the migration process involves parallel deployment of a new Nexus Dashboard platform and manual transfer of the current configuration database from your existing Multi-Site Orchestrator (MSO) cluster.

Before you migrate your existing cluster to Nexus Dashboard:

- If you have an existing physical Nexus Dashboard cluster with Multi-Site Orchestrator application release 3.2(x), you can skip this chapter and simply upgrade the cluster as described in the "Upgrading" chapter of the Cisco Nexus Dashboard Deployment Guide and then upgrade the Multi-Site Orchestrator application as described in Upgrading Multi-Site Orchestrator.

Note

Release 3.2(1) did not support on-boarding cloud sites. If you plan to add any Cloud APIC sites after the upgrade, ensure that they are running Cloud APIC release 5.2(1) or later.

- We recommend that you first familiarize yourself with the Nexus Dashboard platform and overall deployment overview and guidelines described in the Cisco Nexus Dashboard Deployment Guide and the Deploying Multi-Site Orchestrator chapter of this document.

- Ensure that your current MSO cluster is healthy.

You will create a backup of your existing configuration and then import it into the newly deployed MSO application in Nexus Dashboard.

Ensure that the cluster is healthy and existing IPsec intersite connectivity between cloud and on-premises sites is up.

- Ensure that your on-premises sites are running Cisco APIC release 4.2(4) or later.

Site management has moved from the MSO UI to the Nexus Dashboard common site management, which supports releases 4.2(4) or later. Fabric upgrades are described in detail in Cisco APIC Installation, Upgrade, and Downgrade Guide

- Ensure that your cloud sites are running Cisco Cloud APIC release 5.1(1).

Site management has moved from the MSO UI to the Nexus Dashboard common site management, which supports on-boarding cloud site releases 5.1(1) or later. Fabric upgrades are described in detail in Cisco APIC Installation, Upgrade, and Downgrade Guide

Note

However, you must not upgrade to the latest Cloud APIC 5.2(1) release before Multi-Site Orchestrator is migrated to the 3.3(1) release. If your cloud sites are running Cloud APIC 4.x or 5.0(x) releases, you must upgrade to a Cloud APIC 5.1(x) release before following the instructions in this chapter.

- If you manage any Cisco Cloud APIC sites, ensure that you deploy Multi-Site Orchestrator release 3.3(1) and import any existing configurations before you upgrade the cloud sites to Cloud APIC release 5.2(1) or later.

After MSO migration to Release 3.3 is completed, you must upgrade all cloud sites to Cloud APIC release 5.2(1).

- Downgrading to releases prior to release 3.3(1) is not supported.

If you want to downgrade to an earlier release, you must deploy a new Multi-Site Orchestrator cluster on a platform supported by the earlier release, then restore the older configuration backup. Restoring backups created on Release 3.3(1) or later to an older MSO cluster is not supported.

If you downgrade to an earlier release of Multi-Site Orchestrator, you must also downgrade all Cloud APIC sites to a release prior to Release 5.2(1).

Back Up Existing Cluster Configuration

The migration process includes creating a backup of current configuration from your existing Multi-Site Orchestrator cluster and then restoring that in the new Multi-Site Orchestrator application running in Nexus Dashboard.

This section describes how to back up your existing cluster configuration.

Before you begin

You must have the following completed:

- Familiarized yourself with the migration workflow order described in the Overview

- Reviewed and completed general prerequisites described in Prerequisites and Guidelines.

Procedure

Step 1

Log in to your existing Multi-Site Orchestrator.

Step 2

Backup existing deployment configuration.

1. From the left navigation pane, select Operations > Backups & Restore.

2. In the main window, click New Backup.

A New Backup window opens.

3. In the Name field, provide the name for the backup file.

The name can contain up to 10 alphanumeric characters, but no spaces or underscores (_).

4. Choose Local for the Backup Location.

5. Click Save to create the backup.

Step 3

Download the backup file from the existing Orchestrator.

If you created the backup using a remote location, you can skip this step.

In the main window, click the actions () icon next to the backup and select Download. This will download the backup file to your system.

Prepare New Cluster

This section describes how to prepare a Nexus Dashboard cluster for Multi-Site Orchestrator release 3.3(1).

It includes choosing and deploying an appropriate form factor of Nexus Dashboard cluster and establishing network connectivity from the cluster to each site you plan to manage from the Multi-Site Orchestrator.

Before you begin

You must have the following completed:

- Familiarized yourself with the migration workflow order described in the Overview

- Reviewed and completed general prerequisites described in Prerequisites and Guidelines.

- Existing configuration backed up as described in Back Up Existing Cluster Configuration.

Procedure

Step 1

Deploy a Nexus Dashboard release 2.0.2h cluster and configure fabric connectivity.

How you deploy or upgrade to Nexus Dashboard release 2.0.2h depends on the deployment type of your existing cluster:

- If you have an existing virtual Cisco Application Services Engine cluster with Multi-Site Orchestrator application, you must deploy a brand new virtual or cloud Nexus Dashboard cluster as described in the Cisco Nexus Dashboard Deployment Guide.

We also recommend completing the entire migration process before deleting the existing cluster.

- If you have an existing physical Cisco Application Services Engine cluster with Multi-Site Orchestrator application release 3.1(x), you must uninstall the existing application, then upgrade the cluster to Nexus Dashboard release 2.0.2h as described in the "Upgrading" chapter of the Cisco Nexus Dashboard Deployment Guide.

- If you have an existing physical Nexus Dashboard cluster with Multi-Site Orchestrator application release 3.2(x), you can upgrade the cluster as described in the "Upgrading" chapter of the Cisco Nexus Dashboard Deployment Guide and then upgrade the Multi-Site Orchestrator application as described in Upgrading Multi-Site Orchestrator and skip the rest of this chapter.

Note

Release 3.2(1) did not support on-boarding cloud sites. If you plan to add any Cloud APIC sites after the upgrade, ensure that they are running Cloud APIC release 5.2(1) or later.

Step 2

Ensure that your Nexus Dashboard cluster is appropriately scaled based on the fabric sizes and number of applications.

If you deployed a virtual or cloud form factor of the Nexus Dashboard, Multi-Site Orchestrator is the only application supported and the base 3-node cluster is sufficient, so you can skip this step.

If you deployed a physical Nexus Dashboard cluster and Multi-Site Orchestrator is the only application you plan to host, the base 3-node cluster is sufficient and you can skip this step.

However, if you deployed a physical Nexus Dashboard cluster and plan to co-host multiple applications, use the Cisco Nexus Dashboard Capacity Planning tool to determine the required cluster size for your specific use case. If you need to extend your cluster to support all required applications, see the Cisco Nexus Dashboard User Guide for information on deploying additional worker nodes.

Step 3

Install the MSO application in your Nexus Dashboard.

This process is described in detail in the Deploying Multi-Site Orchestrator chapter.

Step 4

On-board all sites to the Nexus Dashboard.

Site management has moved from the MSO UI to the Nexus Dashboard common site management. As such, you must on-board the same sites using the same names that were assigned to the sites when on-boarded on the original MSO cluster to the Nexus Dashboard GUI before migrating your existing configuration to the new cluster, as described in Adding and Deleting Sites. If any site that exists in you current deployment is not present in Nexus Dashboard (or it exists with a different name), the configuration restore during migration will fail with a Pre-restore check failed error message.

Note

After you add the sites to the Nexus Dashboard, you must not set them to Managed in the MSO application. The sites will be enabled for management automatically when you restore your configuration from backup.

Add a site:

1. From the left navigation menu, select Sites.

2. In the top right of the main pane, select Actions > Add Site.

If adding an ACI site, provide the following information:

1. For Site Type, select ACI or Cloud ACI depending on the type of ACI fabric you are adding.

2. Provide the controller information.

You need to provide the Host Name/IP Address, User Name, and Password. for the APIC controller currently managing your ACI fabrics. If MSO is the only application you plan to host, you can specify either the in-band or out-of-band address of the on-premises APIC; however, if you plan to host other applications, such as Nexus Insights, you must specify the in-band address.

For on-premises ACI sites managed by Cisco APIC, if you plan to use this site with Day-2 Operations applications such as Nexus Insights, you must also provide the In-Band EPG name used to connect the Nexus Dashboard to the fabric you are adding. Otherwise, if you will use this site with Multi-Site Orchestrator only, you can leave this field blank.

3. Click Add to finish adding the site.

At this time, the sites will be available in the Nexus Dashboard, but you still need to enable them for Multi-Site Orchestrator management as described in the following steps.

If adding a DCNM site, provide the following information:

1. For Site Type, select DCNM.

2. Provide the DCNM controller information.

You need to provide the Host Name/IP Address of the in-band (eth2) interface, User Name, and Password. for the DCNM controller currently managing your DCNM fabrics.

3. Click Select Sites to select the specific fabrics managed by the DCNM controller.

In the fabric selection window that opens, check one or more fabrics that you managed in your existing Multi-Site deployment and click Select.

Repeat this step to add all the sites from your existing Multi-Site deployment.

Step 5

Add any remote authentication servers you had configured in MSO to the Nexus Dashboard.

User management has moved from the MSO UI to the Nexus Dashboard common user management. As such, you must add the same remote users and authentication servers to the Nexus Dashboard, as described in the Cisco Nexus Dashboard User Guide.

Any local users you had previously configured directly in MSO will be added into the Nexus Dashboard automatically when you import the existing configuration backup.

Restore Configuration in the New Cluster

This section describes how deploy and configure the new Nexus Dashboard cluster and the MSO application, which you will use to restore your previous configuration.

Before you begin

You must have the following completed:

- Existing configuration backed up as described in Back Up Existing Cluster Configuration.

- Deployed Nexus Dashboard cluster and installed Multi-Site Orchestrator application as described in Prepare New Cluster.

Procedure

Step 1

Disconnect the existing Multi-Site Orchestrator cluster.

You must disconnect or bring down the existing Multi-Site Orchestrator cluster so it does not communicate with the Cloud APIC sites during migration.

If you deployed a brand new Nexus Dashboard cluster rather than upgrade an existing cluster, we recommend preserving the existing MSO cluster until the new cluster is deployed and configuration is restored.

Step 2

Ensure that the new Nexus dashboard cluster is up and running and the MSO application is installed.

The MSO application must be a fresh install with no configuration changes to the sites or policies.

Step 3

Log in to your Nexus Dashboard GUI.

Step 4

Ensure that all the sites are on-boarded to Nexus Dashboard.

When you restore the backup, MSO will validate that every site in the backup is present in the Nexus Dashboard with matching site name and type. If validation is unsuccessful, for example if a site is not on-boarded in Nexus Dashboard, configuration restore will fail and you will need to on-board the site before retrying. On-boarding sites is described in Adding Cisco ACI Sites and Adding Cisco DCNM Sites.

Step 5

Import the backup file to your new Orchestrator cluster deployed on the Nexus Dashboard.

If you saved the backup locally, simply import the file:

1. Open your new Multi-Site Orchestrator application.

2. From the left navigation pane, select Operations > Backups & Restore.

3. In the main window, click Import.

4. In the Import from file window that opens, click Select File and choose the backup file you want to import.

Importing a backup will add it to the list of the backups displayed the Backups page.

Step 6

Restore the configuration.

1. From the left navigation menu, select Admin > Backups.

2. In the main window, click the actions () icon next to the backup you want to restore and select Rollback to this backup.

3. Click Yes to confirm that you want to restore the backup you selected.

When the configuration is restored, any sites previously managed by MSO and on-boarded to the Nexus Dashboard will be enabled for MSO management in the GUI. If the configuration backup contains sites that are not on-boarded to your Nexus Dashboard, backup restore will fail with a Pre-restore check failed error and you will need to repeat the procedure after on-boarding any missing sites.

After the configuration is imported and restored, a number of services will be restarted.

Step 7

Update the password.

Due to CSDL (Cisco Secure Development Lifecycle) requirements, you will be required to update the admin user password after configuration restore is completed.

Step 8

Verify that backup was imported successfully and all objects and configurations are present.

1. In the Sites page, verify that all sites are listed as Managed.

2. In the Tenants and Schemas pages, confirm that all tenants and schemas from your previous MSO cluster are present.

3. Navigate to Infrastructure > Infra Configuration > Configure Infra and confirm that intersite connectivity is intact.

In the Connectivity Overview screen, verify that the existing /30 tunnels are up and connectivity was not interrupted.

In the General Settings screen, confirm that the External Subnet Pools previously configured in Cloud APIC have been imported from the cloud sites:

These subnets are used to address the IPsec tunnel interfaces and loopbacks of the Cloud Routers used for on-premises connectivity and had to be configured directly in the Cloud APIC in earlier Multi-Site Orchestrator releases.

Note

You must not make any changes or deploy any configurations at this stage until the cloud sites are upgraded to Cloud APIC release 5.2(1) as described in following sections.

Upgrade Cloud Sites

After Multi-Site Orchestrator is migrated to the 3.3(1) release, you must upgrade any Cloud APIC sites managed by the MSO to release 5.2(1). While existing intersite connectivity will remain intact, you will not be able to change or deploy any cloud site Infra configurations to sites running Cloud APIC releases prior to release 5.2(1).

Before you begin

You must have the following completed:

- Deployed Nexus Dashboard cluster and installed Multi-Site Orchestrator application as described in Prepare New Cluster.

- Existing configuration backup restored to the new cluster as described in Restore Configuration in the New Cluster.

Procedure

Step 1

Upgrade cloud sites.

For each cloud site, you must upgrade its Cloud APIC and then its CSRs before proceed to upgrading the next site. After a site is upgraded and healthy, you can repeat the same steps to upgrade any additional sites.

1. Upgrade a site's Cloud APIC.

You can upgrade Cloud APIC as you typically would using the process detailed in the "Performing a System Upgrade, Downgrade or Recovery" chapters of Cisco Cloud APIC for Azure Installation Guide or Cisco Cloud APIC for AWS Installation Guide.

Note that after the Cloud APIC upgrade, any existing public IP tunnels will remain intact and intersite connectivity via public IPsec will not be interrupted .

2. Upgrade that site's CSR.

Starting with Cloud APIC release 5.2(1) , CSRs upgrade does not happen automatically as it used to in earlier releases, so you must manually trigger CSR upgrade after Cloud APIC is upgraded. You must upgrade the site's CSRs before moving on to upgrading the next site.

You can upgrade Cloud APIC CSRs using the process detailed in the "Performing a System Upgrade, Downgrade or Recovery" chapters of Cisco Cloud APIC for Azure Installation Guide or Cisco Cloud APIC for AWS Installation Guide.

As you upgrade CSRs in each site, the following will occur:

- As each CSR is upgraded, its existing /30 tunnels will be recreated and the traffic will continue to flow.

- Tunnel-management and all Infra configuration changes from Multi-Site Orchestrator are disabled for as long as any of the cloud sites are still running any Cloud APIC or CSR releases prior to 5.2(1).

- If the last site you upgrade is an AWS cloud site, the following will occur for that site's CSRs only:

- The last cloud site's tunnel endpoints will be deleted by Cloud APIC and MSO will delete the corresponding tunnels that use the endpoint

- MSO will delete the tunnels originating from CSRs in the last cloud site

- New hcloudInterCloudSiteTunnel MO will be created and Multi-Site Orchestrator's tunnel management will allocate /31 addresses for the new tunnels

- The CSRs in this site and the CSRs in another cloud site peering with it will establish /31 tunnels.

If the last upgraded site is an Azure site, the same /30 tunnel will be created on the CSRs and the above four bullet points are not relevant.

For any CSRs you add or any underlay configuration changes to existing CSRs after the migration process is completed, all new tunnels created by MSO will be /31 tunnel.

Note

If you do not see BGP sessions within 5 minutes of CSRs upgrade finishing and CSRs coming up, refresh the site's infra connectivity in the Multi-Site Orchestrator Infra Configuration screen.

3. Repeat this step for each cloud site one at a time.

Step 2

Verify Cloud APIC and CSR upgrades have completed.

1. In each site's Cloud APIC, check that the hcloudReconcileDone MO shows reconcileState=steadyState.

You can check the MO by navigating to https://<cloud-apic-ip>/visore.html and searching for hcloudReconcileDone in the Class or DN or URL field.

2. In Multi-Site Orchestrator, verify that intersite connectivity is intact.

You can view the status by navigating to Infrastructure > Infra Configuration > Configure Infra > Connectivity Overview and checking Overlay Status and and Underlay Status tabs:

3. In Multi-Site Orchestrator, confirm that the External Subnet Pools previously configured in Cloud APIC have been imported and are present.

You can view the external pools by navigating to Infrastructure > Infra Configuration > Configure Infra > General Settings:

4. In Multi-Site Orchestrator, confirm that underlay connectivity using public IPs is preserved for existing sites.

You can check existing intersite connectivity by navigating to Infrastructure > Infra Configuration > Configure Infra, then select a specific cloud site from the left sidebar and the Underlay Connectivity tab:

Update MSO Infra Configuration

In order to make subsequent changes to Infra configuration, you must first provide the following information immediately after the cloud sites are upgraded to Cloud APIC release 5.2(1):

- OSPF area ID

- IPN configuration

Before you begin

You must have the following completed:

- Deployed Nexus Dashboard cluster and installed Multi-Site Orchestrator application as described in Prepare New Cluster.

- Existing configuration backup restored to the new cluster as described in Restore Configuration in the New Cluster.

- Upgraded cloud sites as described in Upgrade Cloud Sites.

Procedure

Step 1

Log in to your new Multi-Site Orchestrator.

Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left sidebar, select General Settings.

Step 5

Provide the OSPF Area ID field.

This is OSPF area ID used by cloud sites for on-premises ISN peering, which you previously configured in the Cloud APIC for inter-site connectivity in earlier Multi-Site Orchestrator releases.

Step 6

Add IPN Devices information.

1. Select the IPN Devices tab.

2. Click Add IPN Device.

3. Provide the Name and the IP Address of the on-premises IPN devices.

You must provide the IP addresses of the devices in your on-premises sites that are used as the tunnel peer address from the Cloud APIC's CSRs, not the IPN device's management IP address.

4. Click the check mark icon to save the device information.

5. Repeat this step for any additional IPN devices you want to add.

Step 7

Update Underlay Configuration for inter-site connectivity between on-premises and cloud sites.

For each on-premises site that connects to cloud sites, you need to provide at least one IPN device IP address from the ones you added in the previous step, to which the Cloud APIC's CSRs establish a tunnel.

1. In the left pane, under Sites, select the on-premises site.

2. In the right <Site> Settings pane, select the Underlay Configuration tab.

3. Click +Add IPN Device to specify an IPN device.

4. From the dropdown, select one of the IPN devices you defined previously.

The IPN devices must be already defined in the General Settings > IPN Devices list, as described in the previous step.

Step 8

From the dropdown at the top of the screen, select Deploy to re-deploy the Infra configuration.

Resolve Configuration Drifts

Any time Multi-Site Orchestrator adds support for managing object properties that previously had to be managed directly in the APIC, it sets those properties to some default values for existing objects in MSO Schemas, but does not push them to sites. Typically, when upgrading Multi-Site Orchestrator, you may need to resolve any configuration drifts in the object properties that are newly managed by MSO where the default values picked by MSO differ from the custom values set directly in the fabrics' controllers.

Note

Deploying any templates at this point would push the default values and overwrite the existing values for these properties in the fabrics.

In addition, when first migrating to Release 3.3(1), every template will explicitly indicate a configuration drift in order to force a re-deployment of all templates required to rebuild the information in the databases. In this case, you can use the information presented during template re-deployment to determine if a real configuration drift exists and take an appropriate action (such as re-importing the objects and their properties from the sites), or simply re-deploy the configuration that hasn't changed.

Before you begin

You must have the following completed:

- Deployed Nexus Dashboard cluster and installed Multi-Site Orchestrator application as described in Prepare New Cluster.

- Existing configuration backup restored to the new cluster as described in Restore Configuration in the New Cluster.

- Upgraded cloud sites as described in Upgrade Cloud Sites.

- Updated Multi-Site Orchestrator Infra configuration for the cloud sites as described in Update MSO Infra Configuration.

Procedure

Step 1

Log in to your Nexus Dashboard GUI.

Step 2

Select the Schema you want to verify.

Step 3

In the Schema view, check if the deployment status indicated a configuration drift.

Step 4

Click Deploy to bring up the configuration comparison screen to check which objects contain configuration drifts.

The configuration diff screen will indicate which objects have changed since last deployment. Note down the objects that indicate a Config Drift:

Step 5

If configuration drift is real, resolve the conflicts.

1. Cancel the deployment process to return to the Schema view.

2. Re-import all the objects that contained a configuration drift to sync the site-local properties to MSO.

3. Re-deploy the template.

After you resolve all configuration drift caused by the newly managed object properties, re-deploy the Schema to sync its deployment status across MSO and the fabrics.

Step 6

If no changes are shown in the comparison, simply re-deploy the template.

Trademarks

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS REFERENCED IN THIS DOCUMENTATION ARE SUBJECT TO CHANGE WITHOUT NOTICE. EXCEPT AS MAY OTHERWISE BE AGREED BY CISCO IN WRITING, ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS DOCUMENTATION ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

The Cisco End User License Agreement and any supplemental license terms govern your use of any Cisco software, including this product documentation, and are located at: http://www.cisco.com/go/softwareterms.Cisco product warranty information is available at http://www.cisco.com/go/warranty. US Federal Communications Commission Notices are found here http://www.cisco.com/c/en/us/products/us-fcc-notice.html.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any products and features described herein as in development or available at a future date remain in varying stages of development and will be offered on a when-and if-available basis. Any such product or feature roadmaps are subject to change at the sole discretion of Cisco and Cisco will have no liability for delay in the delivery or failure to deliver any products or feature roadmap items that may be set forth in this document.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com go trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R)