41 – Interconnecting Traditional DCs with VXLAN EVPN Multi-site using DCNM

Good day team,

The same question arises often about how to leverage DCNM to deploy a VXLAN EVPN Multi-site between traditional Data Centers. To clarify, DCNM can definitively help interconnecting two or multiple Classical Ethernet-based DC networks, in a very short time, thanks to its automation engine.

Just a reminder (post 37) , VXLAN EVPN Multi-site overlay is initiated from the Border Gateway nodes (BGW). The BGW function is the key element of the EVPN Multi-site solution offering the extension of the Layer 2 and Layer 3 connectivity across distant sites. The BGW nodes can run in two different modes; either in Anycast BGW mode (Up to 6 BGW devices per site), traditionally deployed to interconnect VXLAN EVPN based fabrics, or in vPC BGW mode (up to 2 vPC BGW devices per site), essentially designed to interconnect traditional data centers, but not limited to; indeed, vPC BGW devices can also be leveraged to locally dual-attached endpoints at layer 2, even though the infrastructure supports VXLAN EVPN from end-to-end.

In this demo, I used several designations to describe the Classic Ethernet-based data center networks such as Traditional or legacy data center network. All these terms mean that these networks are non VXLAN EVPN-based fabrics. In this use-case, the vPC BGW node is agnostic about data center network types, models or switches that construct the data center network infrastructure, as long as it offers Layer 2 (dot1Q) connectivity for layer 2 extension, and if required, external routed traffic (VRF lite). The traditional data center can be organized with any Cisco Nexus series (N9k/N7k/N5k/N2k) or Catalyst or simply non-Cisco switches. Any enterprises with two or more data centers built with a hierarchical multi-tier architecture (Core, Aggregation, Access) running a simple classic Ethernet based Layer 2 network, with or without centralized routing block can deploy a VXLAN EVPN Multi-site solution using DCNM 11 to interconnect their legacy data centers.

DCNM 11 can discover and import the network devices belonging to the traditional data center using CDP (Cisco) or LLDP (Vendor-neutral). In case of NX-OS platforms, DCNM 11 can also be leveraged to push traditional network configuration and monitor these Nexus devices inside an “External Fabric”.

The figure below depicts the lab used to demonstrate step by step how DCNM will auto-provision the underlay and overlay networks between DC-1 to DC-2. The lab consists of two Classic Ethernet-based data center networks. On each location, a pair of BGW switches (Nexus 9000-EX series) has been grafted to each aggregation layer using a virtual port-channel. Regarding the physical interconnection between sites, a complex routed network has been set up. Nonetheless, it is possible to directly interconnect the two pairs of BGW nodes in a back-to-back fashion using direct fibers for example, either in square fashion or in full-mesh, making its deployment again faster. However, the goal for adding a Layer 3 core between the distant data centers for this video, is to make the demo more comprehensive, explaining thereby how DCNM can also auto-provision the layer 3 underlay for the Inter-fabric connectivity (IFC) designed with a complex routed network. Last but not least, a Layer 3 Core network becomes very helpful if you have more than 2 data centers to interconnect.

For an end-to-end architecture with a back-to-back connectivity between BGW nodes, you can skip the step 2 [ Creation of one “External Fabric” that contains the complex Layer 3 network ] discussed during the video. Indeed, DCNM will automatically provision the underlay network required between the distant BGW nodes.

The deployment for this DCI architecture can be organized in 2 main stages. The first stage consists of creating the infrastructure with multiple building blocks called “fabrics”, one fabric per network function.

The following demo describes the 1st stage organized in four building blocks:

1. Creation of two “External Fabric” comprising the traditional data centers networks.
   • Each brownfield data center network is respectively imported in “monitor” mode. This means that DCNM discovers the switches including the connectivity between the devices. The existing configuration of the legacy network is imported without any changes.
   • Notice this External fabric for each data center is optional for the purpose of Multi-site. However, it brings the visual topology of the traditional DC network. Furthermore, DCNM can configure the Classic Ethernet (CE) switches if needed and it provides device health and link utilization in real time.
2. Creation of one “External Fabric” that contains the complex Layer 3 network.
   • The Layer 3 is imported with its current configuration, however, this time it is configured in “managed” mode. The reason is that, as demonstrated in this video, we want DCNM to also auto-provision the network connectivity between the BGW nodes and the Core routers. As a result, DCNM will discover and configure the routed interfaces on both sides of the IFC, from the BGW nodes and the Core routers. To achieve this automation, the user needs to uncheck the monitor mode and configure the role of the Core routers.
3. Creation of two “Easy Fabric” used to support the function of BGW Multi-site.
   • The BGW devices are discovered by DCNM and imported without preserving the configuration. Actually, DCNM is going to deploy the whole configuration reuired for EVPN Multi-site. All parameters will be used from the Easy Fabric settings.
   • You can leave all default parameters, or you can change any of them accordingly to your specific needs or habits. DCNM policies that automates the deployment rely on Cisco best practices. Nevertheless, during this demo, some of the parameters in the resources tab are updated, particularly for the loopback interfaces address ranges in order to appear differently between the two sites – however, changing the parameters remains optional, you can leave all default values if you wish.
4. Creation of the Multi-Site Domain (MSD)
   • In this particular use-case, you import the two “Easy Fabric” created for the management of the BGW nodes and the “External Fabric” comprising the Layer 3 core.
   • MSD is responsible to auto-deploy the Underlay and Overlay networks required for the VXLAN EVPN Multi-site established between the two data centers.

During the deployment of the configurations, different colors will inform you about the status of the devices. Green means that the intent matches the running config and that configuration is in sync between the device of interest and DCNM. When the deployment is pending on a switch, the switch is displayed in blue color.

The multi-site network infrastructure interconnecting the two traditional data centers being deployed, you want now to provision the networks and VRF required for the Day-2 operations.

In this scenario, you have several endpoints spread across the traditional data centers belonging to two different networks. In this example, VLAN 2100 and 2101 under the same Tenant 1 must be extended between the two data centers. You want to allow these two networks across the 2 port-channel interfaces “22” facing the traditional data centers, in order to extend the networks of interest from the traditional data centers across the Multi-site infrastructure. Notice that one physical host is connected to the BGW 1 in single attachment via the interface e1/36 in which the concerned VLANs must also be allowed.

The following video describes how to deploy the networks and VRFs. This is the same deployment described in post 39 with end-to-end VXLAN EVPN fabrics, except that, in this particular scenario the auto deployment concerns only the BGW nodes. The Dot1Q Layer 2 traffic uses the traditional network transport across the legacy data centers to hit the destined endpoints.

Finally, a short series of tests demonstrating the successful multi-site deployment and network connectivity extended from DC-1 to DC-2 concludes this demo.

After the Nexus9000-EX have been connected to the aggregation layer of the traditional data center, it took less than 30 mins to fully deploy the Multi-site infrastructure between the two traditional data centers, including the extension of the Layer 2 and Layer 3 networks.

The BGW function requires a particular hardware offering tunnel stitching and hardware-based rate limiters for the tunnel transporting the BUM traffic. As a result, the BGW nodes must be part of the CloudScale ASIC family, starting from the Nexus9000-EX series to any new Nexus 9000 platforms. However, the network devices inside the traditional data centers can be any hardware platforms, and network operating software, including non-Cisco switches.

I ran through an exhaustive configuration showing further integration with legacy data centers, and a complex Layer 3 Core network, however, if you have direct fibers that interconnect two data centers, and want to make the deployment faster, you just need to run through the steps 3 (easy Fabric for BGWs) and 4 (MSD), and then deploy network a_la_demand.

Hope you enjoyed, thank you !