Good day every one,

As you certainly know, we are now rebranding DCNM 11 to Nexus Dashboard Fabric Controller, consequently, NDFC 12 is the new acronym to automate and operate NX-OS based fabrics

It is not just about changing the name, there are several significant changes with NDFC 12.

Just to list few of them:

One of the key evolution in regard to NDFC is that it now joins the ecosystem of services that runs natively on top of the common Nexus dashboard platform.

One of the key advantages for the network operators with this evolution, is that they just need a single experience, it doesn’t really matter what application you run, from the installation process to the common Web GUI. All applications or services look with the same logic behind any requests.

You have a single pane of glass which is the Nexus dashboard platform through which the end-user can consume different applications for the daily operations. The goal is to simplify and align all Apps into the same platform to make the day 2 ops easier.

For those of you who’ve seen other services like Nexus Dashboard Insights (NDI) or the Nexus Dashboard Orchestrator (NDO), you can now see Nexus dashboard Fabric Controller with the same look and feel, such as the icons, the meaning of the colors, the filters, the navigation panel etc, to just list few. It’s fully unified with the same components for the Web UI.

Thus, for the end-user perspective, you get the same look and feel regardless of what service interfaces your use.

I’ve compiled this short video below to give you a quick overview of the new Nexus Dashboard Fabric Controller.

Enjoy and feel free to comment or ask questions

Thank you

Good day everyone,

I have recorded 6 different modules aimed at giving you a good understanding of the key functions that DCNM 11 offers to manage, automate and operate your data center network.

While waiting for the new version 12 coming soon, this series of modules covers the Essentials of Data Center Network Manager  (DCNM™) version 11.

By “Essentials” the mean is that it covers the most features needed for all enterprises, from small legacy Data Center Network to multiple modern Fabrics or a mix of. This essential series of modules covering DCNM 11 emphases the Classic LAN as well as the VXLAN EVPN Fabric environments. Indeed, any network team can benefit from the advantages of leveraging the same and unique platform to operate a classic based data center network or a modern VXLAN EVPN based fabric or a mixture of the both.

I have developed the content for the lectures on DCNM 11 and 10.1(n) environment.

If you are not yet familiar with DCNM™, you can read previous posts, otherwise, DCNM is the leading Intent-Based Networking solution for Standalone NX-OS based Data Center. It encompasses the comprehensive management solution for all NX-OS network deployments spanning LAN fabrics across all Cisco Nexus Series Switches deployed in Standalone mode using Cisco NX-OS Software, SAN fabrics that rely on Cisco MDS Family, and IP Fabric for Media (IPFM) networking for Broadcaster requirements in the data center. This series focuses on LAN Fabrics.

By listening all of these video modules (preferably, to be viewed in ascending order), you will learn how DCNM 11 offers device management for a large number of equipment’s while proposing a ready-to-use control from a zero-touch deployment to the network fabric control, monitoring and automation capabilities for Cisco Nexus based VXLAN EVPN as well as control and automation for Cisco Nexus Classic LAN infrastructures.

In addition to the 6 modules focusing on DCNM 11, I recorded an extra module 7 giving you a quick overview of the evolution to operate Multi-Site, Multi-Domain and Multi-Cloud in the context of the Standalone NX-OS network environment.

Please enjoy the videos. I hope it will help you to better capture all awesome features that you can leverage using DCNM 11.

• Module 1 covers the Cisco Intent-Based Networking solution, the evolution, and the requirements with a focus on the network centric approach. What is DCNM 11 and Why DCNM? How DCNM can help the enterprises to address their growing requirements, hence, how it helps to increase their revenue? This module comprises a short description of the DCNM 11 platform including its administration points, and a brief reminder on licensing. Finally, it embraces a generic overview of the DCNM Graphical User Interface to get a little familiar with DCNM 11.

Module 1 includes also a subsidiary section which provides a brief update on VXLAN EVPN allowing everyone to better track different deployments and use cases discussed in all subsequent modules. Indeed DCNM often refers to terminologies used by VXLAN. If you are already familiar with VXLAN EVPN, you can skip this section.

• Module 2 describes the concept fabric automation embedded with DCNM 11 Fabric. During this module, you will learn how to automate in few clicks the creation and the deployment from ground up of a VXLAN EVPN Fabric, how to import and operate a Classic LAN in a mixed environment, and how to deploy quickly a solid VXLAN EVPN Multi-site. It comprises the deployment of Networks and VRF, as well as service nodes insertion.

• Module 3 depicts the extension for traditional segmented Layer 2 and Layer 3 networks to the external side of the VXLAN EVPN fabric.

• Module 4 highlights the Policy based templates used by DCNM 11 to automate the deployment of the end-user intent. It includes a series of sections that covers the Freeform Template, the Config Compliance process, Synch interface options as well as the software image management.

• Module 5 covers the fabric programming approaches using Restful API. In addition to the DCNM 11 GUI, the fabric can be fully manageable via the DCNM Restful API using your preferred API management tools. It also comprises the other DCNM interactive API documentation that lets you try out the API calls directly in your web browser.

• Module 6 demonstrates the operation tools embedded with DCNM such as health monitoring, alarming, EndPoint locators, backup and restore, and more.

• Finally, Module 7 discusses about the Evolution to operate Multi-domain and Multi-cloud in the context of the Standalone NX-OS network environment. It gives an overview of Cisco’s strategy offering a common platform supporting day 2 operation tools as well as the orchestrator platform managing Multi-sites, Multi-domain and Multi-cloud environment.​​​​​​​

Refresh on Endpoint Locator in the context of DCNM

The Endpoint Locator (EPL) feature allows DCNM to monitor in real-time the endpoints (EPs), the switches (location) where the EPs have been learnt, networks & VRF associated to the EPs, and this analysis is supported for up to 4 VXLAN EVPN fabrics with a maximum of 100 thousand endpoints across all fabrics. The analysis results can be displayed in a single pane of glass combining the whole infrastructure, either from the datacenter scope or from the Multi-Site Domain (MSD) scope.

The state information for each element is displayed in real time and recorded between two synch processes providing the network life history of an endpoint, as well as the time of connectivity of a particular IP and/or MAC address or the moves associated to endpoints across different locations.

An endpoint can be a virtual machine (VM), a container, a bare-metal server, a network service appliance and so on. In other terms, any element that offers network reachability information learnt by the VXLAN BGP EVPN fabric (EVPN Route-type 2 advertisements).

EPL relies on the MP-BGP EVPN Address Family updates to pinpoint endpoint information. In order to get a full visibility of the EPs and their locations, DCNM needs to peer with the BGP Route-Reflector(s) (a functionality usually deployed on the Spine nodes) to get these updates. For that purposes DCNM offers a dedicated interface Eth-2, also known as In-Band interface to peer with the RR(s).

DCNM Cluster and Use of Compute Nodes

Some applications like “Network insights for Resources” (NIR) require more CPU and more memory resources. Consequently, DCNM must be deployed in “Cluster Mode”, meaning with additional Compute nodes in order to distribute the workload and replicate database information across different service resources.

The Compute Nodes expand therefore the resources for the DCNM framework to scale-out according to the end-user requirements. Also, additional Compute Nodes are required, traditionally 3 nodes, if you need to manage a large number of nodes (typically more than 80).

When the cluster mode is deployed, in the context of the EPL tracking service, the network interface used to peer between the RR and the DCNM framework is automatically provisioned by DCNM in a container created in one the available Compute nodes. One In-Band Management IP address is automatically created for each Fabric.

Notice for NIR application, DCNM will also automatically provision additional In-Band interfaces across the Compute nodes cluster for all network devices solicitated for telemetry and data analytics, as discussed in the chapter “Network Insight in the context of DCNM” below.

Figure 1 depicts the DCNM HA cluster with 3 Compute Nodes with the In-Band Management network (in red) and the two IP addresses provisioned by DCNM on Compute Node 1 and 2, respectively (33.33.33.97 and 33.33.33.98). Each IP address is automatically created during the EPL configuration for each VXLAN EVPN Fabric. In our case studies, we have two VXLAN EVPN Fabrics, as presented in the figure 4 below.

The IP address assigned to the compute nodes for In-band management purposes can be retrieved directly from the EPL configuration page (Control  Endpoint Locator  Configure). As shown in Figure 2, you can click on the “i” icon on the top right corner to display the BGP neighbor configuration of the Route-Reflector, highlighting the peering with the DCNM In-band Mgmt address provisioned on one of the Compute nodes.

Repeat the same for the EPL configuration for the other Fabric to get the second In-band IP address for Fabric-2.

In addition to the BGP configurations, DCNM will configure automatically the interfaces eth2 for In-Band Management in the Compute nodes.

show In-Band Interface eth2 from one DCNM Compute Node

$ ifconfig eth2
 eth2: flags=4163  mtu 1500
         inet 33.33.33.97  netmask 255.255.255.0  broadcast 33.33.33.255
         inet6 fe80::250:56ff:fe98:138d  prefixlen 64  scopeid 0x20
         ether 00:50:56:98:13:8d  txqueuelen 1000  (Ethernet)
         RX packets 59136853  bytes 48581283056 (45.2 GiB)
         RX errors 0  dropped 371  overruns 0  frame 0
         TX packets 29383282  bytes 2195885777 (2.0 GiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

DCNM relies on a dedicated policy template called “epl_evpn_bgp_rr_neighbor” to push the BGP configurations toward each Route-Reflector (Spine nodes) that belongs to a VXLAN EVPN fabric from where EPL needs to retrieve the endpoints’ information. The Policy template can be retrieved by selecting the “View/Edit policies” for every RR Spine node of interest. The BGP AS number and the BGP IPv4 Neighbor address are automatically configured by DCNM. This template is non-editable.

As depicted in Figure 2 and Figure 3, the BGP configuration CLI is pushed to the RRs as shown in this sample below for the RR in Fabric-2.

 router bgp 65512
    neighbor 33.33.33.98
      remote-as 65512
      address-family l2vpn evpn
        send-community
        send-community extended
        route-reflector-client 

Notice how the “update-source loopback0” command is not included in the BGP neighbor configuration of the Spine nodes. Currently, DCNM doesn’t mandate to receive data packet only originated from each configured RR (Spine node) interface loopback 0 (BGP Router-ID).

Figure 4 below shows the BGP peerings that must be established between the Route-Reflectors of each VXLAN EVPN Fabric and the In-band management interfaces of the compute nodes part of the DCNM cluster.

Adding redistribution of BGP to OSPF (and vice-versa) between the two Fabrics

As depicted in Figure 4, routing information to reach the Route-Reflectors deployed in the remote site must be exchanged across sites to establish the MP-BGP EVPN adjacencies required to track the endpoints localized in Fabric-2. Those adjacencies must be established between the compute nodes In-band IP addresses and the loopback0 interfaces deployed on the remote RRs.

When deploying VXLAN Multi-Site, only the minimum required ‘underlay’ IP prefixes are exchanged between sites (just to ensure the establishment of control plane and data plane connectivity between the Border Gateway nodes). This means that the In-band subnet (33.33.33.0/24 in our example) used to address the containers on the compute nodes and the loopback addresses of the remote RRs (addressed out of the original TEP pool assigned by DCNM during the fabric bring up process) are not exchanged by default between fabrics.

As a result, it is therefore required to explicitly create specific route-maps to redistribute the In-band network 33.33.33.0/24 from Fabric-1 to Fabric-2 and the remote RRs Loopback0 IP addresses to the local leaf nodes where the DCNM cluster nodes are attached (actually, we will redistribute the remote RR reachability information to the whole Fabric-1).

Hence, we need first to create a FreeForm Template and apply it to the pair of BGWs located in Fabric-1 as follow:

1. Go to Control => Fabric Builder => select Fabric-1
2. Select Tabular View
3. Check the 2 BGWs
4. Select View/Edit Policies
5. Add a new Policy (press + button)
   • In the policy name field, type “free”, and select “switch_freeform”
6. A pop-up window appears
   • Add a description
   • Enter the CLI as shown in Figure 5

Note: by selecting the two BGW devices, the freeform template will be created for each selected node. This is possible only if the CLI configuration to be applied is strictly identical for both devices.

Example of Route-map for Fabric-1

The configuration sample below shows how to exchange the required routing information in and out of Fabric 1.

 ip prefix-list EPL-RR-FAB2 seq 5 permit 10.22.0.5/32
 ip prefix-list EPL-NWK seq 5 permit 33.33.33.0/24
 !
 route-map EPL-BGP-to-OSPF permit 10
   match ip address prefix-list EPL-RR-FAB2
 route-map EPL-OSPF-to-BGP permit 10
   match ip address prefix-list EPL-NWK
 !
 router ospf UNDERLAY
   redistribute bgp 65510 route-map EPL-BGP-to-OSPF
 !
 router bgp 65510
   address-family ipv4 unicast
     redistribute ospf UNDERLAY route-map EPL-OSPF-to-BGP 

• 10.22.0.5/32 associated with the prefix-list EPL-RR-FAB2 is the Router-ID for the Route-Reflector based in Fabric-2 (Loopback0).
• 33.33.33.0/24 associated with the prefix-list EPL-NWK is the In-Band Management network used for the services that are created on the compute nodes.
• Under the router OSPF process, the route-map EPL-BGP-to-OSPF permits the redistribution of 10.22.0.5/32 learnt from the BGP EVPN adjacencies with the remote BGW nodes into the OSPF (underlay) protocol used in Fabric 1.
• Under the router BGP process, the route-map EPL-OSPF-to-BGP permits the redistribution of the subnet 33.33.33.0/24 into the BGP EVPN Control Plane, to be able to advertise it to the remote BGW nodes.

Example of Route-map for Fabric-2

Likewise, you need to create another FreeForm Template to redistribute the proper routes in and out of Fabric 2, as listed below and depicted in Figure 7.

 ip prefix-list EPL-NWK seq 5 permit 33.33.33.0/24
 ip prefix-list EPL-RR-FAB2 seq 5 permit 10.22.0.5/32
 !
 route-map EPL-BGP-to-OSPF permit 10
   match ip address prefix-list EPL-NWK
 route-map EPL-OSPF-to-BGP permit 10
   match ip address prefix-list EPL-RR-FAB2
 !
 router ospf UNDERLAY
   redistribute bgp 65512 route-map EPL-BGP-to-OSPF
 !
 router bgp 65512
   address-family ipv4 unicast
     redistribute ospf UNDERLAY route-map EPL-OSPF-to-BGP

When the freeform templates have been saved and deployed, you can check the In-band reachability from the Route-reflector in Fabric-2:

Fabric-2-Spine-2# show bgp l2vpn evpn neighbors 33.33.33.98
 BGP neighbor is 33.33.33.98, remote AS 65512, ibgp link, Peer index 7
   BGP version 4, remote router ID 33.33.33.1
   Neighbor previous state = OpenConfirm
   BGP state = Established, up for 10:01:07
   Neighbor vrf: default
   Last read 00:00:07, hold time = 90, keepalive interval is 30 seconds
 ..//..
   Forwarding state preserved by peer for:
   Restart time advertised to peer: 120 seconds
   Stale time for routes advertised by peer: 300 seconds
   Extended Next Hop Encoding Capability: advertised
   Message statistics:
                               Sent               Rcvd
   Opens:                         3                  2
   Notifications:                 1                  0
   Updates:                     369                  0
   Keepalives:                 1647               1659
   Route Refresh:                 0                  0
   Capability:                    0                  0
   Total:                      1684               1661
   Total bytes:               95164              31483
   Bytes in queue:                0                  0
   For address family: L2VPN EVPN
   BGP table version 9805, neighbor version 9805
 ..//..
   Local host: 10.22.0.5, Local port: 179
   Foreign host: 33.33.33.98, Foreign port: 42354
   fd = 79

Network Insights in the context of DCNM

Network Insights offers network resource utilization, rate of change, trends, environmental and resource anomalies over time for operational, configuration and hardware resources for all the VXLAN EVPN nodes supporting these functions. In order to get those telemetry statistics and flow analytics information, each concerned network device must be configured by NI in order to push the required information to the Network Insight database supported by the DCNM Cluster nodes.

Even if Network Insights is not strictly related to EPL, they both have one common requirement when it comes to establishing inter-fabrics connectivity.

When configuring the telemetry parameters, different groups are created for exporting from the leaf and spine nodes the Software telemetry information, as well as the Flow telemetry data. This information is sent to different IP addresses belonging to the In-Band management network which are automatically created by DCNM using micro-services in containers distributed across the 3 Compute nodes, as shown in the few examples below:

..//.. Software Telemetry

destination-profile
     use-vrf default
     source-interface loopback0
   destination-group 500
     ip address 33.33.33.103 port 57500 protocol gRPC encoding GPB
     use-chunking size 4096

..//.. Flow Telemetry

flow exporter telemetryExp_0
   destination 33.33.33.101 
   transport udp 30000
   source loopback0
   dscp 44

As a result, all leaf nodes, including the devices that belong to the remote Fabrics, must be able to peer with the In-band management network in the VXLAN EVPN Fabric where DCNM Compute Nodes are located (Fabric-1 in our example).

To achieve the peering between all nodes where the telemetry and analytic functionalities are enabled, it suffices to slightly modify the prefix-list already shown for the EPL use case and replace the specific RR router-ID(s) with a less specific prefix matching the subnet used to assign the Loopback0 addresses to all the nodes of the remote VXLAN EVPN Fabric:

Example of FreeForm Template deployed in the BGWs located in Fabric-1

ip prefix-list TELEMETRY-FAB2 seq 10 permit 10.22.0.0/24 le 32
 ip prefix-list IB-NWK seq 5 permit 33.33.33.0/24
 !
 route-map TELEMETRY-BGP-to-OSPF permit 10
   match ip address prefix-list TELEMETRY-FAB2
 route-map IB-OSPF-to-BGP permit 10
   match ip address prefix-list IB-NWK
 !
 router ospf UNDERLAY
   redistribute bgp 65510 route-map TELEMETRY-BGP-to-OSPF
 !
 router bgp 65510
   address-family ipv4 unicast
     redistribute ospf UNDERLAY route-map IB-OSPF-to-BGP

• 10.22.0.0/24 associated with the prefix-list TELEMETRY-FAB2 includes all the Router-ID for the VXLAN EVPN Network devices based in Fabric-2 (Loopback0).
• 33.33.33.0/24 associated with the prefix-list IB-NWK is the In-Band Management network used for the services that are created on the DCNM compute nodes.
• Under the router OSPF process, the route-map TELEMETRY-BGP-to-OSPF permits the redistribution of any endpoints that belongs to 10.22.0.0/24 learnt from the BGP EVPN adjacencies with the remote BGW nodes into the OSPF (underlay) protocol used in Fabric 1.
• Under the router BGP process, the route-map IB-OSPF-to-BGP permits the redistribution of the subnet 33.33.33.0/24 into the BGP EVPN Control Plane, to be able to advertise it to the remote BGW nodes.

Example of FreeForm Template deployed in the BGWs located in Fabric-2

 ip prefix-list IB-NWK seq 5 permit 33.33.33.0/24
 ip prefix-list TELEMETRY-FAB2 seq 10 permit 10.22.0.0/24 le 32
 !
 route-map IB-BGP-to-OSPF permit 10
   match ip address prefix-list IB-NWK
 route-map TELEMETRY-OSPF-to-BGP permit 10
   match ip address prefix-list TELEMETRY-FAB2
 !
 router ospf UNDERLAY
   redistribute bgp 65512 route-map IB-BGP-to-OSPF
 !
 router bgp 65512
   address-family ipv4 unicast
     redistribute ospf UNDERLAY route-map TELEMETRY-OSPF-to-BGP

Good day Network experts

It has been a great pleasure and an honor working with Tarique Shakil and Vinit Jain on this book below, deep-diving on this amazing LISP protocol.

I would like also to take this opportunity to thank Max Ardica, Victor Moreno and Marc Portoles Comeras for their invaluable help. I wrote the section on LISP Mobility deployment with traditional and modern data center fabrics (VXLAN EVPN based as well as ACI Multi-Pod/Multi-Site), however this could not have been done without the amazing support of these guys.

Available from Cisco Press
or from Safari book Online

Implement flexible, efficient LISP-based overlays for cloud, data center, and enterprise

The LISP overlay network helps organizations provide seamless connectivity to devices and workloads wherever they move, enabling open and highly scalable networks with unprecedented flexibility and agility.

LISP Network Deployment and Troubleshooting is the definitive resource for all network engineers who want to understand, configure, and troubleshoot LISP on Cisco IOS-XE, IOS-XR and NX-OS platforms. It brings together comprehensive coverage of how LISP works, how it integrates with leading Cisco platforms, how to configure it for maximum efficiency, and how to address key issues such as scalability and convergence.

Focusing on design and deployment in real production environments, three leading Cisco LISP engineers present authoritative coverage of deploying LISP, verifying its operation, and optimizing its performance in widely diverse environments. Drawing on their unsurpassed experience supporting LISP deployments, they share detailed configuration examples, templates, and best practices designed to help you succeed with LISP no matter how you intend to use it.

This book is the Cisco authoritative guide to LISP protocol and is intended for network architects, engineers, and consultants responsible for implementing and troubleshooting LISP network infrastructures. It includes extensive configuration examples with troubleshooting tips for network engineers who want to improve optimization, performance, reliability, and scalability.

This book covers all applications of LISP across various environments including DC, Enterprise, and SP.

• Review the problems LISP solves, its current use cases, and powerful emerging applications
• Gain in-depth knowledge of LISP’s core architecture and components, including xTRs, PxTRs, MR/MS, ALT, and control plane message exchange
• Understand LISP software architecture on Cisco platforms
• Master LISP IPv4 unicast routing, LISP IPv6 routing, and the fundamentals of LISP multicast routing
• Implement LISP mobility in traditional data center fabrics, and LISP IP mobility in modern data center fabrics
• Plan for and deliver LISP network virtualization and support multitenancy
• Explore LISP in the Enterprise multihome Internet/WAN edge solutions
• Systematically secure LISP environments
• Troubleshoot LISP performance, reliability, and scalability

Good day team,

The same question arises often about how to leverage DCNM to deploy a VXLAN EVPN Multi-site between traditional Data Centers. To clarify, DCNM can definitively help interconnecting two or multiple Classical Ethernet-based DC networks, in a very short time, thanks to its automation engine.

Just a reminder (post 37) , VXLAN EVPN Multi-site overlay is initiated from the Border Gateway nodes (BGW). The BGW function is the key element of the EVPN Multi-site solution offering the extension of the Layer 2 and Layer 3 connectivity across distant sites. The BGW nodes can run in two different modes; either in Anycast BGW mode (Up to 6 BGW devices per site), traditionally deployed to interconnect VXLAN EVPN based fabrics, or in vPC BGW mode (up to 2 vPC BGW devices per site), essentially designed to interconnect traditional data centers, but not limited to; indeed, vPC BGW devices can also be leveraged to locally dual-attached endpoints at layer 2, even though the infrastructure supports VXLAN EVPN from end-to-end.

In this demo, I used several designations to describe the Classic Ethernet-based data center networks such as Traditional or legacy data center network. All these terms mean that these networks are non VXLAN EVPN-based fabrics. In this use-case, the vPC BGW node is agnostic about data center network types, models or switches that construct the data center network infrastructure, as long as it offers Layer 2 (dot1Q) connectivity for layer 2 extension, and if required, external routed traffic (VRF lite). The traditional data center can be organized with any Cisco Nexus series (N9k/N7k/N5k/N2k) or Catalyst or simply non-Cisco switches. Any enterprises with two or more data centers built with a hierarchical multi-tier architecture (Core, Aggregation, Access) running a simple classic Ethernet based Layer 2 network, with or without centralized routing block can deploy a VXLAN EVPN Multi-site solution using DCNM 11 to interconnect their legacy data centers.

DCNM 11 can discover and import the network devices belonging to the traditional data center using CDP (Cisco) or LLDP (Vendor-neutral). In case of NX-OS platforms, DCNM 11 can also be leveraged to push traditional network configuration and monitor these Nexus devices inside an “External Fabric”.

The figure below depicts the lab used to demonstrate step by step how DCNM will auto-provision the underlay and overlay networks between DC-1 to DC-2. The lab consists of two Classic Ethernet-based data center networks. On each location, a pair of BGW switches (Nexus 9000-EX series) has been grafted to each aggregation layer using a virtual port-channel. Regarding the physical interconnection between sites, a complex routed network has been set up. Nonetheless, it is possible to directly interconnect the two pairs of BGW nodes in a back-to-back fashion using direct fibers for example, either in square fashion or in full-mesh, making its deployment again faster. However, the goal for adding a Layer 3 core between the distant data centers for this video, is to make the demo more comprehensive, explaining thereby how DCNM can also auto-provision the layer 3 underlay for the Inter-fabric connectivity (IFC) designed with a complex routed network. Last but not least, a Layer 3 Core network becomes very helpful if you have more than 2 data centers to interconnect.

For an end-to-end architecture with a back-to-back connectivity between BGW nodes, you can skip the step 2 [ Creation of one “External Fabric” that contains the complex Layer 3 network ] discussed during the video. Indeed, DCNM will automatically provision the underlay network required between the distant BGW nodes.

The deployment for this DCI architecture can be organized in 2 main stages. The first stage consists of creating the infrastructure with multiple building blocks called “fabrics”, one fabric per network function.

The following demo describes the 1st stage organized in four building blocks:

1. Creation of two “External Fabric” comprising the traditional data centers networks.
   • Each brownfield data center network is respectively imported in “monitor” mode. This means that DCNM discovers the switches including the connectivity between the devices. The existing configuration of the legacy network is imported without any changes.
   • Notice this External fabric for each data center is optional for the purpose of Multi-site. However, it brings the visual topology of the traditional DC network. Furthermore, DCNM can configure the Classic Ethernet (CE) switches if needed and it provides device health and link utilization in real time.
2. Creation of one “External Fabric” that contains the complex Layer 3 network.
   • The Layer 3 is imported with its current configuration, however, this time it is configured in “managed” mode. The reason is that, as demonstrated in this video, we want DCNM to also auto-provision the network connectivity between the BGW nodes and the Core routers. As a result, DCNM will discover and configure the routed interfaces on both sides of the IFC, from the BGW nodes and the Core routers. To achieve this automation, the user needs to uncheck the monitor mode and configure the role of the Core routers.
3. Creation of two “Easy Fabric” used to support the function of BGW Multi-site.
   • The BGW devices are discovered by DCNM and imported without preserving the configuration. Actually, DCNM is going to deploy the whole configuration reuired for EVPN Multi-site. All parameters will be used from the Easy Fabric settings.
   • You can leave all default parameters, or you can change any of them accordingly to your specific needs or habits. DCNM policies that automates the deployment rely on Cisco best practices. Nevertheless, during this demo, some of the parameters in the resources tab are updated, particularly for the loopback interfaces address ranges in order to appear differently between the two sites – however, changing the parameters remains optional, you can leave all default values if you wish.
4. Creation of the Multi-Site Domain (MSD)
   • In this particular use-case, you import the two “Easy Fabric” created for the management of the BGW nodes and the “External Fabric” comprising the Layer 3 core.
   • MSD is responsible to auto-deploy the Underlay and Overlay networks required for the VXLAN EVPN Multi-site established between the two data centers.

During the deployment of the configurations, different colors will inform you about the status of the devices. Green means that the intent matches the running config and that configuration is in sync between the device of interest and DCNM. When the deployment is pending on a switch, the switch is displayed in blue color.

The multi-site network infrastructure interconnecting the two traditional data centers being deployed, you want now to provision the networks and VRF required for the Day-2 operations.

In this scenario, you have several endpoints spread across the traditional data centers belonging to two different networks. In this example, VLAN 2100 and 2101 under the same Tenant 1 must be extended between the two data centers. You want to allow these two networks across the 2 port-channel interfaces “22” facing the traditional data centers, in order to extend the networks of interest from the traditional data centers across the Multi-site infrastructure. Notice that one physical host is connected to the BGW 1 in single attachment via the interface e1/36 in which the concerned VLANs must also be allowed.

The following video describes how to deploy the networks and VRFs. This is the same deployment described in post 39 with end-to-end VXLAN EVPN fabrics, except that, in this particular scenario the auto deployment concerns only the BGW nodes. The Dot1Q Layer 2 traffic uses the traditional network transport across the legacy data centers to hit the destined endpoints.

Finally, a short series of tests demonstrating the successful multi-site deployment and network connectivity extended from DC-1 to DC-2 concludes this demo.

After the Nexus9000-EX have been connected to the aggregation layer of the traditional data center, it took less than 30 mins to fully deploy the Multi-site infrastructure between the two traditional data centers, including the extension of the Layer 2 and Layer 3 networks.

The BGW function requires a particular hardware offering tunnel stitching and hardware-based rate limiters for the tunnel transporting the BUM traffic. As a result, the BGW nodes must be part of the CloudScale ASIC family, starting from the Nexus9000-EX series to any new Nexus 9000 platforms. However, the network devices inside the traditional data centers can be any hardware platforms, and network operating software, including non-Cisco switches.

I ran through an exhaustive configuration showing further integration with legacy data centers, and a complex Layer 3 Core network, however, if you have direct fibers that interconnect two data centers, and want to make the deployment faster, you just need to run through the steps 3 (easy Fabric for BGWs) and 4 (MSD), and then deploy network a_la_demand.

Hope you enjoyed, thank you !

Another great feature supported by DCNM 11 concerns the extension of Layer 3 network connections from a VXLAN EVPN fabric across an external Layer 3 network using VRF-Lite hand-off from the Border leaf node toward the external Edge router.

There are different options to deploy a VRF-Lite connection to the outside of the VXLAN fabric. Either using a manual deployment or leveraging the auto-configuration process that will configure automatically the VRF-lite on a Border Leaf node toward an external Layer 3 network. VRF Lite is supported on border devices with the role of either Border Leaf node, Border Spine node or Border Gateway node. Thus, the later providing both functions, Layer 3 connectivity from a VXLAN EVPN Fabric to outside a Layer 3 network, as well as VXLAN EVPN Multi-site service offering hierarchical and integrated VXLAN EVPN to VXLAN EVPN fabric interconnection.

Prior to deploy the external VRF-lite, an External Fabric must exist, and if not, it must be created, in which the concerned external router (DCI-WAN-1) should be imported.

Manual VRF Lite Configuration

One of the key reasons for configuring the interfaces manually could be for example when the Layer 3 network is managed by an external service entity, thus the Network team has no control on the configuration which is traditionally managed by a different organization, internal or external such as a Layer 3 service operator. The first demo follows this scenario and illustrates an end-to-end manual configuration of VRF-Lite connections from the Border leaf node to an external Edge router. Actually, this manual mode comes with the default Fabric settings.

The Border leaf nodes, BW1 and BW2, being vPC peer devices, it is a best practice to configure a routed interface per device connecting the external Layer 3 network. As a result, one physical link per Border Gateway connects toward the WAN Edge router as depicted in the figure above, and demo’ed.

For that particular scenario, the external fabric management can be left to Monitor mode (default mode under Fabric setup) as we don’t need DCNM to push any configuration to the Edge router

The role of the targeted external router must be set to “Edge Router” in order to extend the VRF-Lite network. As elaborated in the previous demo, the role of “Core Router” is used for the Multi-site deployment.

The interface E1/3 of the Edge router (DCI-WAN-1) connecting the Border Leaf node BW1 of the VXLAN Fabric is manually configured with the associated sub-interface for the VRF Tenant-1. Notice the choice for the Dot1Q tag being “2” in order to be aligned with the 1st sub-interface Dot1Q from the pool (see figure below):

interface Ethernet1/3.2
   mtu 9216
   encapsulation dot1q 2
   vrf member Tenant-1
   ip address 172.16.1.2/30

Finally, for connectivity and testing purposes, a Loop-back 100 is configured for the VRF Tenant-1 in the external Edge router:

interface loopback100
   vrf member Tenant-1
   ip address 101.101.101.101/3

Under the Fabric builder topology, it is crucial to check and set the role of the Border leaf node to “Border”.

From the VXLAN Fabric setup, the deployment for the VRF Lite is left to “Manual” (default).

When both, VXLAN and External Fabrics are ready, VRF-Lite can be deployed as illustrated in the video below.

Automatic VRF Lite Configuration

The second demo illustrates the automatic configuration for an external network connection using VRF Lite. We use the Border Leaf node, BW3, from the VXLAN EVPN Fabric 2 that connects the Edge Router DCI-WAN-2 belonging an External Fabric.

DCNM allows the Network Manager to automatically detect and create the external links. In order to succeed the automatic configuration for the VRF Lite IFC (Inter-Fabric Connection) stage, each physical interface should belong to a Border Leaf node set with the role “Border”, and should be connected to a device with the Edge Router role belonging to an External Fabric. Under the VXLAN Fabric setting, the VRF Lite deployment must be configured to “To External Only”. This mode gives automatically a subnet IP address range for the VRF lite interfaces (Point-to-Point).

On the External Fabric, we need to allow DCNM to manage the Edge Router. By default the External Fabric comes in “Monitor” only mode. It is crucial to uncheck the Monitor mode under the Fabric setting, thus DCNM can push the configuration to the Edge router.

As soon as the roles has been attributed for each device, Border Leaf node and Edge router) and both Fabrics has been configured accordingly, DCNM 11 will detect and provision automatically the external Links (IFC) for each interface of the Border Leaf node(s) that connects a Edge router role device. DCNM allocates the VRF Policy setup automatically to those external links using the policy called “ext_fabric_setup”.

The following demo covers two different stages. First of all, it shows the deployment of the VRF Lite with the automatic configurations for the Border Leaf node. Secondly, it demonstrates how DCNM can manage an external router using the Interface Control windows to configure a Loopback interface for network continuity testing purposes, a Sub-interface for the extension of the VRF Lite network, as well as how to push configuration using line commands (CLI) directly from DCNM via a “freeform config” template. In this example, we configure BGP setup for the VRF Tenant-1 network as the neighbor information.

Dear Network experts,

It took a while to post this update on DCNM 11.1 due to other priorities, but I should admit it’s a pity due to all the great features that came with DCNM 11.1. As mentioned in the previous post, DCNM 11.1 brings a lot of great improvements.

Hereafter is a summary of the top LAN fabric enhancements for VXLAN Multi-Site deployment that comes with DCNM 11.1 for LAN Fabric. Notice that many other new features and improvements come with 11.1. Please feel free to look at the Release-note for an exhaustive list of New Features and Enhancements in Cisco DCNM, Release 11.1(1).

Instead of describing in details all those new features, I think that a series of videos can be more efficient to understand those functionalities through specific demos. Please notice that I’m not covering all the features of DCNM 11.1 such as analytics, telemetry, compute monitoring, backup and restore, etc.; the main focus for this post is around VXLAN EVPN Multi-Site. I will provide a demo on VRF-Lite connection to the outside the the fabric in a next post.

The series of following demos rely on the same VXLAN EVPN Multi-Site infrastructure that I used for the previous post 38. Hence, you can can also compare and see the evolution of DCNM 11.1 for those particular functions.

Please feel free to make any comment or ask any questions if one of the following sections is not clear enough, and I will try to provide you with the needed information.

Brownfield Migration

• Transition an existing VXLAN fabric management into DCNM.

The network manager can now import directly to DCNM 11.1 an existing VXLAN EVPN Fabric while the DC network continues to provide connectivity to production endpoints. Typically, the original VXLAN fabric has been previously created and managed through manual CLI configurations or custom automation scripts. With this new software release, the Network manager can import an existing VXLAN network and start managing the fabric through DCNM. All configurations, as well as logical Networks and VRFs  are preserved. The migration happens without interruption. After the migration, the initial configurations of the VXLAN Fabric underlay and overlay networks will be managed by DCNM and new devices, Networks and VRFs can be added afterward seamlessly.

The 1st demo shows how to import an existing Brownfield VXLAN EVPN Fabric while maintaining the Networks and VRFs

Fabric Builder, fabric devices and fabric underlay networks

• Configuration Compliance displays side-by-side the current and pending configuration before deployment.

• vPC support for VXLAN Multi-Site Border Gateway (BGWs) and standalone fabrics have been added.

The second demo shows how to create a new Greenfield VXLAN fabric from the ground up. The process is quite similar to what illustrated in the previous post for DCNM 11.0; however, several improvements have been added such as the configuration compliance that allows to compare side by side the current setup and the expected configuration. Also, new roles can be assigned to Spine and Border Leaf nodes extending the support for additional topologies, including:

• Multi-Site Border Gateways deployed on the Spine nodes (Border Spines).
• The new vPC Border Gateway mode, allowing to locally connect endpoints (single- or dual-homed attached) at Layer 2 to the same vPC pair of switches where the VXLAN Multi-site is initiated (coexistence of Border Gateway and Compute leaf roles).

External Fabric

The external fabric can be created to include network devices (switches and routers) that offer Layer 3 connectivity services to the fabric (for both north-south and east-west communication).

• Inter-Fabric Connections (IFCs) can be automatically created to/from those external devices and the fabric border switches.

• The external fabric devices can have two roles: Edge Router (used to establish VRF-Lite connectivity required for north-south communication) and Core Router (representing the inter-site network devices allowing to establish east-west VXLAN data-plane communication between sites).

The following demo  illustrates how to import an External Layer 3 Network interconnecting the VXLAN EVPN Fabrics. Routers, link connectivity as well as the topology are automatically discovered and imported into DCNM.

Notice that the devices belonging to the external fabric can be managed by DCNM (this applies to Cisco Nexus devices only).

VXLAN EVPN Multi-Site Domain

The fourth video shows how DCNM 11.1 allows to quickly deploy a VXLAN EVPN Multi-Site infrastructure by moving the existing VXLAN Fabrics into an active Multi-site Domain. The underlay and overlay network configurations for the Inter-Fabric Connection (BGWs <=> Core-routers) is made automatically by DCNM, either using a peer to peer link establishment for the underlay and a full-mesh establishment for the overlay network or using a Router Server for eBGP (functioning like a traditional Route Reflector normally used for iBGP peering) located in the Layer 3 core network.

Overlay Network/VRF provisioning & Interfaces

• Port-channels, vPCs, sub-interfaces, and loopback interfaces can be added and modified with an external fabric devices.
• Cisco DCNM 11.1(1) specific template enhancements are made for interfaces.

• Networks and VRFs deployment can be deployed automatically at the Multi-site Domain level from Site to Site in one single action.

Finally, the last video shows how DCNM can simplify the deployment for Network and VRF extension across both sites, in few clicks.

Just amazing 🙂

Hope you will enjoy these few demos

Thank you, yves

Hot networks served chilled, DCNM style

================================================================

*** Important Note ****

Since I ran the following demos using the first version of DCNM 11, a new maintenance release 11.1 came out in December 2018. As a consequence, several videos given underneath are not anymore topical. However, I leave them available for those who are interested, knowing that some demos are still valid such as the DCNM installation, or the deployment of a Green-Field Fabric (with some slight differences). But for the Multi-site demo, just switch to the next post http://yves-louis.com/DCI/?p=1744

================================================================

When I started this blog for Data Center Interconnection purposes some time ago, I was not planning to talk about network management tools. Nevertheless, I recently tested DCNM 11.0 to deploy an end-to-end VXLAN EVPN Multi-site architecture, hence, I thought about sharing with you my recent experience with this software engine. What pushed me to publish this post is that I’ve been surprisingly impressed with how efficient and time-saving DCNM 11.0 is in deploying a complex VXLAN EVPN fabric-based infrastructure, including the multi-site interconnection, while greatly reducing the risk of human errors caused by several hundred required CLI commands. Hence, I sought to demonstrate the power of this fabric management tool using a little series of tiny videos, even though I’m usually not a fan of GUI tools.

To cut a long story short, if you are not familiar with DCNM (Data Center Network Manager), DCNM is a software management platform that can run from a vCenter VM, a KVM machine, or a Bare metal server.  DCNM is a general purpose Network Management Software (NMS) and Operational Support Software (OSS) product targeted at NX-OS networking equipment. It focuses on Cisco Data Center infrastructure, supporting a large set of devices, services, and architecture solutions. It covers multiple types of Data Center Fabrics; from the Storage fabric management, where it comes from originally, with MDS 9000 and Nexus platforms, to the IP-based fabric, to the traditional Classical STP network or even legacy FabricPath domains, and recently Media networking. But more importantly, and relevant to this blog, it also supports modern VXLAN EVPN Fabrics. It enables the network admin organization to provision, monitor, and troubleshoot the data center infrastructure (FCAPS*). DCNM relies on multiple pre-configured templates used to automate the deployment of many network services and transport such as routing, switching, VXLAN MP-BGP EVPN, Multicast deployment, etc. We can also modify or create our own templates, which makes DCNM very flexible, going beyond a limited hard-coded configuration. And last but not least, DCNM offers a Power-On Auto Provisioning (POAP) service process using the fabric bootstrap option, allowing network devices to be automatically provisioned with their configuration of interest at Powered on, without entering any CLI command. Finally DCNM contributes to the Day0 (Racking, Cabling, power-on – though zero bootstrap configuration), Day1 (POAP, VXLAN EVPN Fabric builder, Underlay, Overlay) and DayN (Network and host deployment, firmware updates, monitoring, fault-mgmt) operations of Network Fabrics.

* FCAPS is a network management framework created by the International Organization for Standardization (ISO). FCAPS groups network management operational purposes into five levels: (F) fault-management, (C) configuration, (A) Accounting, (P) performance, and (S) security.

For this article, I will just cover a small section of DCNM 11.0, focusing on VXLAN EVPN and Multi-site deployment. And, because of the Graphical User Interface, I thought that a series of videos could be more visual and easier to understand than a long technical post to read.

However, if you need further technical details about DCNM 11.0, there is a very good User Guide available from this public DCNM repository

I have divided the deployment of a VXLAN EVPN Multi-site infrastructure into five main stages. I’m not certain this is the initial thought of the DCNM Dev team, but this is how I saw it when using DCNM 11.0 for the first time.

1. 1. • Installation of DCNM 11.0
      • Building multiple VXLAN EVPN Fabrics
      • Extending the VXLAN Fabrics using VXLAN EVPN Multi-site
      • Deploying Network’s and VRF’s
      • Attaching Hosts to the VXLAN Fabric

DCNM 11.0 installation

The installation of DCNM 11.0 is very fast, however it requires two different steps: The OVF template deployment and the DCNM Web Installer. Notice that instead of the OVA installation, it is also possible to install the ISO Virtual appliance if you wish to install DCNM on KVM or Bare Metal computer. Let’s talk about on the OVA installation, but if needed, all installation and upgrade details are available at cisco.com.

The first action for installing the OVA is to download the dcnm.ova file from CCO. You can try DCNM 11 for free for a certain period of time. Yes, there is a 30 day full Advanced Feature Trial license!

If you are not yet familiar with VXLAN EVPN, with or without VXLAN EVPN Multi-site, or if you are a bit anxious about risks of configuration errors, you will definitely enjoy this software platform.

A few comments on the installation. Firstly, the OVF template comes with 3 networks:

• One interface network for the DCNM management itself
• A second interface for the Out-Band-Management of the Fabric
• A third one that provides In-Band connection to the fabric. It will be used for communicating with the control plane of the fabric, for example to locate the end-points within the VXLAN fabric.

If you are running the vCenter web client, at the end of the OVA installation, a pop-up window will propose to start the DCNM Web Installer automatically. Otherwise, you can start the Web Installer using the management IP address, associated with port 2080 (http://a.b.c.d:2080) as shown in this 1st video below.

Secondly, the DCNM Installer will inquire to choose the installation mode corresponding to the type of network architecture and technology you are planning to deploy. Consequently, select the “Easy Fabric” mode for VXLAN EVPN fabric deployment and automation. It will then ask for a few traditional network parameters such as NTP servers, DNS server, Fully Qualified Host Name, so be prepared with these details.

Subsequently, you need to configure the network interfaces.

• dcnm-mgmt network:

This network provides connectivity (SSH, SCP, HTTP, HTTPS) to the Cisco DCNM Appliance. Associate this network with the port group that corresponds to the subnet that is associated with the DCNM Management network.

This one should be already configured from the previous OVA installation.

• enhanced-fabric-mgmt

This network provides enhanced fabric management of Nexus switches. You must associate this network with port group that corresponds to management network of leaf and spine switches.

• enhanced-fabric-inband

This network provides in-band connection to the fabric. You must associate this network with port group that corresponds to a fabric in-band connection.

This network will be essentially used to communicate with the Control plane of the VXLAN Fabric in order to exchange crucial endpoint reachability and location information. Actually, we don’t need this interface for the deployment of the VXLAN EVPN fabric per se, but it will be useful later, for example, for a feature called Endpoint Locator. I didn’t configure this network interface for the purpose of this demo.

I will try to add more videos to demonstrate further DCNM features such as Endpoint Locator in a fabric or future across Multi-sites. In addition VXLAN OAM (NGOAM) provides the reachability information to the hosts and the VTEPs in a VXLAN network. NGOAM can be enabled from DCNM to show the physical path and devices taken by the overlay network (L2 and L3 VNI). I will configure the In-bound network interface when appropriate.

The 1st video below is optional, it describes how to install the OVA file and show the configuration of the DCNM Web installer. If you are already familiar with the OVF template deployment, you can skip this video.

Video 1: DCNM 11 OVA Installation and WEB Installer

Building multiple VXLAN EVPN Fabrics

This second video demonstrates how DCNM 11.0 can be leverage to deploy VXLAN EVPN- Fabric in the context of multiple sites interconnected using a routed Layer 3 WAN.

The first action is to create the network underlay for each Fabric. There are several profiles available to build the DC network of your choice. The one we need to select for deploying a VXAN EVPN Fabric is called “Easy Fabric”, which is the default architecture we previously configured during the Web Installer initialization. Most of network parameters are already provisioned by default. The only values to manually enter are the AS Number and the Site number for each VXLAN domain. The rest can be left by default. Nonetheless, DCNM offers the flexibility to change any parameters by the ones that the network manager wishes to practice. The video demonstrates the usage of particular ranges of IP addresses for the underlay components. Additional options are available if needed, for example, to enable Tenant Routed Multicast in order to optimize the routed c-multicast traffic across the VXLAN EVPN fabric. Another great added value of DCNM is to automate the deployment of multiple fabric devices using POAP bootstrap configured from the centralized DCNM platform. As a result, there is no need to connect slowly switch-by-switch using each physical console port in order to configure the manageability access via the CLI language. On the contrary, all the switches automatically discovered by DCNM and identified by the network manager as part of the VXLAN Fabric, will download their configuration automatically during their power cycle.

Even the respective role of each device (Spine, Leaf, Border Leaf) is automatically recognized, although it can be modified on the fly if needed, as shown in the video for the Multi-site Border Gateway function.

Therefore, the only requirement prior to start the DCNM configuration, is to collect the serial numbers of the switch devices, so they can be identified and selected accordingly to be assigned with their corresponding network management identifiers. Another slight TIP that can help prior to the creation of the fabric, is to draw a high-level topology figure with the physical interfaces used for the uplinks as well as connections toward the WAN edge devices crossed-out for inter-fabric network establishment.

For Network managers who want to check the outline prior to deploy the configuration toward the switches, it is possible to “preview” the configuration given in a CLI format, hence he knows exactly what configuration is pushed toward the switch. When a configuration is deployed, it is also possible to check the status of it and check, in case of error, where the config stops and why. In addition, if the  DCNM configuration is different than the one expected (for example, if someone changed one parameter directly using the CLI), DCNM will highlight the differences with the non-expected CLI  from a DCNM point of view.

This video also includes how to connect each fabric to the WAN. To achieve this inter-site Layer 3 connectivity, we need to create an External Fabric that acts for the L3 WAN/MAN. By electing a seed unit from the L3 network and defining the appropriate number of hops from there, the Layer 3 topology with all existing routers of interest will be automatically discovered.

Video 2: Building Multiple VXLAN EVPN Fabrics

Video3: Fabric interconnection with VXLAN EVPN Multi-site

With the previous video 2, we have seen how the external Layer 3 Core connects both VXLAN EVPN fabrics. This following stage illustrates the initialization of the VXLAN EVPN Multi-site feature in order to interconnect these 2 VXLAN EVPN fabrics over the top of the Layer 3 Core via their respective Border Gateways.

VXLAN EVPN Multi-Site uses the same physical border gateways to terminate and initiate the overlay network inside the fabric, as well as to interconnect distant VXLAN EVPN domains using integrated VXLAN EVPN based network overlay.

VXLAN EVPN Multi-Site is a hierarchical solution that relies on the networking design principles to offer an efficient solution extending layer 2 and layer 3 segments across multiple EVPN-based Fabrics. For additional reading on VXLAN EVPN Multi-site, I recommend you to look at the previous post (37) as well as the public white paper that covers deeply VXLAN EVPN Multi-site.

Currently, DCNM 11.0(1) requires you to configure the underlay and overlay network establishments between each device (From each BGW to its distant peers), but, the Dev Engineers are already working on a new version that simplifies further the automation of the Multi-site creation. I will post a new video as soon as I can test it.

Video 3: Fabrics interconnection with VXLAN EVPN Multi-site

Video 4: Deploying Network and VRF

This phase shows the creation of the layer 2 and layer 3 network segmentation and how they are pushed to the Border Gateways. The order of this stage might be different and the creation of Network and VRF can be initiated after the attachment of the compute nodes. The concept to build a network overlay is quite simple. We can leave the proposed VXLAN Network ID which is consumed from the DCNM pool. We can provide a new name for that network or leave as is, etc. What is required is to map the concerned VLAN ID for that specific Network. Finally, we need to associate the network with its Layer 3 segment (VRF). Either the VRF can be selected from a list of existing VRF’s previously created or we have to build a new one from the same window. Then, the default gateway (SVI) for that particular network can be configured under Network profile. All the configuration ready to be deployed takes few seconds.

When all desired Network have been finally created, from the topology view under control network tab, we must select the devices with the same role and deploy the Networks of choice. During this video, I only selected the Border Gateway devices. Indeed, the deployment of networks for the compute leaf nodes are discussed in the next and last video, after the vPC port-channel has been created.

Notice that prior to deploy any configuration to a switch, it is always possible to preview the configuration before it is pushed to the targeted devices, if you wish.

Video 4: Deploying Networks and VRF’s

Video 5: Host Interface deployment and Endpoint discovery

This last video concludes the demonstration of deploying an End-to-End VXLAN EVPN Multi-site infrastructure, with ultimately the configuration of the vPC port-channels where hosts are locally attached to the fabric and how to allow the concerned networks into those vPC port-channels.

It also demonstrates how DCNM 11.0 can integrate a VMware topology onto its dynamic topology views, discovering automatically the vCenter that controls the host-based networking on the fabric to show how virtual machines, hosts, and virtual switches are interconnected.

In this video, when the configuration of the port-channels is completed, the intra-subnet and inter-subnet communication is verified using a series of tests, inside the fabric and across the Multi-site extension.

Video 5: Host Interface deployment, vCenter discovery and Verification

Conclusion

After building the physical deployment of the fabric (racking, cabling, powering all gears), and after collecting the serial numbers and drawing the uplink interfaces with their respective identifiers into a high-level topology diagram, it will take only a couple of hours to build the whole multi-site VXLAN EVPN infrastructure, ready to bridge and to route traffic across multiple VXAN EVPN Fabrics in a multi-tenancy environment.

I only recorded each video once because all first attempts to use DCNM to deploy the VXLAN EVPN multi-site were a success. The communication from site to site worked immediately. BTW, I only found the DCNM 11.0 User Guide to add the pointer in this post after I ran all the DCNM configurations. It’s a rumor, please don’t repeat, but the Engineers working on DCNM said that several crucial improvements will come with the major release (MR) coming soon. I am very eager to test what they have done :). I will demo these improvements here as soon as I can get a solid beta version to try.

I’ll keep you posted. However, in the meantime, please feel free to bring your comments.

Thank you for reading and watching

Dear readers,

Since I wrote this article last spring discussing about future integrated DCI features, VXLAN EVPN Multi-site has become real. It is available since NX-OS 7.0(3)I7(1).

There is also a new White paper that discusses deeper about VXLAN EVPN Multi-site Design and deployment

Good read !

++++++++++++++++++++++  Updated Jan 2018  +++++++++++++++++++++++++

DCI is dead, long live to DCI

Some may find the title a bit strange, but, actually, it’s not 100% wrong. It just depends on what the acronym “DCI” stands for. And, actually, a new definition for DCI came out recently with VXLAN EVPN Multi-site, disrupting the way we used to interconnect multiple Data Centres together.

For many years, the DCI acronym has conventionally stood for Data Centre Interconnect.

The “Data Centre Interconnect” naming convention may essentially be used to describe solutions for interconnecting traditional-based DC network solutions, which have been used for many years. I am not denigrating any traditional DCI solutions per se, but the Data Centre networking is evolving very quickly from the traditional hierarchical network architecture to the emerging VXLAN-based fabric model gaining momentum in enterprise adopting it to optimize modern applications, computing resources, save costs and gain operational benefits. Consequently, if these independent DCI technologies may continue to be deployed for extending Layer 2 networks between traditional DC networks, they will tend to be replaced by modern solution allowing a smooth migration to VXLAN EVPN-based fabrics. Nonetheless, for the interconnection of modern VXLAN EVPN standalone (1) Fabrics, a new innovative solution called “VXLAN EVPN Multi-site” – which integrates in a single device the extension of the Layer 2 and Layer 3 services across multiple sites – has been created for a long life. As a consequence, DCI also stands for Data Centre Integration when interconnecting multiple fabrics using VXLAN EVPN Multi-site, thus this strange title.

(1) The term “standalone Fabric” describes a Network-based Fabric. The concept of Multi-Pod, Multi-Fabric and Multi-site discussed in this article doesn’t concerned the enhanced SDN-based Fabric, also known as ACI Fabric. If both rely on VXLAN as the network overlay transport within the Fabric, they both use different solutions for interconnecting multiple Fabrics together.

ACI already offers a more sophisticated and solid integration of Layer 2 and Layer 3 services with its Multi-Pod solution since ACI 2.0 as well as ACI Multi-site since ACI 3.0 (Q3CY2017).

A bit of history:

Cisco brought the MP-BGP EVPN based control-plane for VXLAN in beginning of 2015. After a long process of qualification and testing in 2015, this article has been elaborated to describe the approach of the VXLAN EVPN Multi-Pod. At the time, it offered the best design considerations and recommendations for interconnecting multiple VXLAN EVPN Pods. Afterwards, we ran a similar process for validating the architecture for interconnecting multiple independent VXLAN EVPN Fabrics, of which post 36 with its sub-sections has been published. This second set of technical information and design considerations has mainly been elaborated to clarify the network services and network design required for interconnecting thru an independent DCI technology, multiple VXLAN EVPN Fabrics in conjunction with the Layer 3 Anycast Gateway (1) functions.

(1) The Layer 3 Anycast gateway is distributed among all compute and service leaf nodes and across all VXLAN EVPN Fabrics offering the default gateway for any endpoints at the closest Top of Rack.

Although the sturdiness of this solution of VXLAN EVPN Multi-Fabric is fully reached, this methodical validation design demonstrates the critical software and hardware components required when interconnecting VXLAN EVPN Fabrics with the Layer 3 Anycast Gateway, with manual connectivity’s for independent and dedicated Layer 2 and Layer 3 DCI services, making the whole traditional connectivity a bit more complex than usual.

To summarise, as of August 2017, we had 2 choices when interconnecting multiple VXLAN EVPN Pods or Fabrics:

VXLAN EVPN Multi-PoD

The first option aims to create a single logical VXLAN EVPN Fabric in which multiple Pods are dispersed to different locations using a Layer 3 underlay to interconnect the Pods. Also known as stretched Fabric, this option is more commonly called VXLAN EVPN Multi-Pod Fabric. This model goes without a DCI solution, from a VXLAN overlay point of view, there is no demarcation between a Pod and another. The key benefit of this design is that the same architecture model can be used to interconnect multiple Pods within a building or campus area as well as across metropolitan areas.

Figure 1: VXLAN EVPN Multi-PoD

If this solution is simple and easy to deploy, and thanks to the flexibility of VXLAN EVPN, it’s lack of sturdiness with the same Data-Plane and Control-Plane shared across all geographically dispersed Pod.

Although with the VXLAN Multi-Pod methodology, the Underlay and Overlay Control-Planes have been optimised – separating the Control-Planes for the Overlay network (iBGP ASN) using eBGP for the external site network – to offer a more robust End-to-End architecture than a basic stretched Fabric, the same VXLAN EVPN Fabric is overextended across different locations. Indeed, the reachability information for all VXLAN Tunnel endpoints (VTEPs) as well as that of endpoints is extended across all VXLAN EVPN Pod (throughout the whole VXLAN Domain).

If this VXLAN EVPN Multi-pod solution simplifies its deployment, the failure domain is kept extended accros all the PoD, and the scalability remains the same as it is in a single VXLAN fabric.

VXLAN EVPN Multi-Fabric

The second option when interconnecting multiple VXLAN EVPN Fabrics in a more solid fashion, is by inserting an autonomous DCI solution, extending Layer 2 and Layer 3 connectivity services between independent VXLAN Fabrics, while maintaining segmentation for the multi-tenancy purposes. Compared to the VXLAN EVPN Multi-Pod Fabric design, the VXLAN EVPN Multi-Fabric architecture offers a higher level of separation for each of the Data Centre Fabrics, from a Data-Plane and Control-Plane point of view. The reachability information for all VXLAN Tunnel endpoints (VTEPs) is contained within each single VXLAN EVPN Fabric. As a result, VXLAN tunnels are initiated and terminated inside each Data Centre Fabric (same location).

The standalone VXLAN EVPN Multi-Fabric model also offers higher scalability than the VXLAN Multi-Pod approach, supporting a larger total number of leaf nodes and endpoints across all Fabrics.

Figure 2: VXLAN EVPN Multi-Fabric

However, the sensitive aspect of this solution is that it requires complex and manual configurations for a dedicated Layer 2 and Layer 3 extension between each VXLAN EVPN Fabric. In addition, the original protocol responsible for Layer 2 and Layer 3 segmentations must handoff at the egress side of the border leaf nodes (respectively, VRF-Lite handoff and Dot1Q handoff). Last but not least, in addition to the dual-box design for dedicated Layer 2 and Layer 3 DCI services, endpoints cannot be locally attached to them.

Although the additional DCI layer has been a key added-value, maintaining the failure domain inside each Data Centre with a physical boundary between DC and DCI components, it has been a choice balanced between sturdiness and CAPEX, sometimes impacting the final choice to deploy a solid DCI solution.

On the same article part 5, Inbound and Outbound Path Optimisation solutions, that can be leveraged for different Data Centre deployments, have also been elaborated. And although VXLAN Multi-Pod and Multi-Fabric solutions may become soon “outdated” for interconnecting modern VXLAN-based Fabrics (see next section), the technical content of path optimisation is still valid for VXLAN Multi-Site or just when trying to better apprehend the full learning and distribution processes for the endpoint reachability information within a VXLAN EVPN Fabric. Hence, don’t ignore them yet, they may still be very useful for education purposes.

If this VXLAN EVPN Multi-fabric solution improves the strength of the end to end solution, there is no integration per se between each VXLAN fabric and the DCI, making its deployment complex to operate and manage.

VXLAN EVPN Multi-site

The above short refresh on the traditional solutions when interconnecting geographically dispersed modern DC Fabrics being said, and while some network vendors (hardware and software/virtual platforms) are trying to offer a suboptimal method of the options discussed previously, actually, these two traditional solutions are obsolete when interconnecting standalone VXLAN EVPN-based Fabrics. Indeed, a new architecture called VXLAN EVPN Multi-Site has become available since August 2017 with the NX-OS software release 7.0(3)I7(1). A related IETF draft is available for further details (https://tools.ietf.org/html/draft-sharma-Multi-Site-evpn) if needed.

For guidelines, please visit the Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 7.x

The concept of VXLAN EVPN Multi-Site integrates the Layer 2 and Layer 3 services so that they can be extended within the same box. This model relies on Layer 2 VNI stitching. A new function known as Border Gateway (BGW) is responsible for that integrated extension. As a result, an Intra-Site VXLAN tunnel terminates at a Border Gateway which, from the same VTEP, re-initiates a new VXLAN tunnel toward the remote sites (Inter-Sites). The extension of the L2 and L3 services are transported using the same protocol. This new innovation combines the simplicity and flexibility of the VXLAN Multi-Pod with the sturdiness of the VXLAN Multi-Fabric.

The role of Border Gateway is a unique function offered by the hardware at line rate the because of Cisco ASIC technology (Cloud Scale ASIC). Yet another feature that will never come with a whitebox with Merchant Silicon.

It is not mandatory that the non-Border Gateway network devices within the VXLAN EVPN Fabric support the function of Multi-Site, neither in hardware nor in software.  As a result, any VXLAN EVPN Fabric based on the first-generation of N9k in production today will be able to deploy Multi-Site by simply adding the pair of required devices(1) that support the Border Gateway function. This will both integrate and extend the Layer 2 and Layer 3 services.

(1) VXLAN Multi-Site is supported on –EX and –FX Nexus 9k series platforms (ToR and Line-cards) as well as on the Nexus 9364C device.

Nevertheless, in addition, this new approach to the interconnection of multiple sites brings several added values.

VXLAN EVPN Multi-Site offers an “All-In-One” box solution with, if desired, the possibility to locally attach endpoints for other purposes (computes, firewalls, load balancers, WAN edge routers, etc).

Reminder:

For VXLAN EVPN Multi-PoD, in order to attach endpoints to the transit node, it is required that the hardware support the bud-node mode. A bud node is a device that is a VXLAN VTEP device and at the same time an IP transit device for the same multicast group used for VXLAN VNIs.

For VXLAN EVPN Multi-Fabric, a pair of additional boxes must be dedicated for DCI purposes only, with no SVI and no attached endpoint.

This new integrated solution offers a full separation of Data-Plane and Control-Plane per site, maintaining the failure domain to its smallest diameter.

VXLAN EVPN Multi-site overview

VXLAN EVPN Multi-site Border Gateway

The stitching of the Layer 2 and Layer 3 VNIs together (intra-Fabric tunnel ⇔ inter-Fabric tunnel) happens in the same Border node (Leaf or Spine). This Border node role is known as Border Gateway or BGW.

Figure 3: VXLAN EVPN Multi-Site Overview

Multiple designs for VXLAN EVPN Multi-site

The approach of VXLAN EVPN Multi-Site is to offer a level of simplicity and flexibility never-before-reached. Indeed, the Border Gateway function can be initiated from existing vPC-based Border leaf nodes, with a maximum of the two usual vPC peer devices, offering seamless migration from traditional to Multi-Site architectures. This vPC-based option allows for locally attached dual-homed endpoints at Layer 2.

However, it is also possible to deploy up to four independent border leaf nodes per site, relying on BGP delivering the function of Border Gateway in a cluster fashion, all active with the same VIP address. With this BGP-based option, Layer 3 services (FW, router, SLB, etc..) can be leveraged on the same switches.

Finally, it is also possible to initiate the Multi-Site function with its Border Gateway role directly from the Spine layer (up to 4 Spine nodes), as depicted in Figure 4.

Figure 4: VXLAN EVPN Multi-Site Flexibility

When the Border Gateway is deployed, however the cluster is built – with 2 or 4 nodes – it will share the same VIP (Virtual IP address) across the associated members. The same Border Gateway VIP is used for the virtual networks initiated inside the VXLAN Fabric as well as for the VXLAN tunnels established with the remote sites.

The architecture of VXLAN EVPN Multi-Site is transport agnostic. It can be established between two sites connected with direct fibre links in a back-to-back fashion, or across an external native Layer 3 WAN/MAN or Layer 3 VPN Core (MPLS VPN), interconnecting multiple Data Centres across any distances. This new solution offers higher reliability and resiliency for the end-to-end design with all redundant devices delivering fast convergence in case of link or device failure.

From an operational, admin and network management point of view, VXLAN OAM tools can be leveraged to understand the physical path taken by the VXLAN tunnel, inside and between the Fabrics, to provide information on path reachability; Overlay to Underlay correlation; interface load and error statistic in the path; latency, etc. – a data lake of information which is usually “hidden” in a traditional network overlay deployment.

VXLAN EVPN Multi-site – Layer 2 Tunnel Stitching

Because the  VNI to VNI stitching happens in the same VTEP, there is no requirement to handoff the VRF-Lite, nor the Dot1Q frame. Hence, theoretically, Multi-Site allows for going beyond the 4K Layer 2 segments imposed by traditional DCI solution.

Figure 5: Traditional DCI design with VRF-lite and Dot1Q Hand-off

Figure 6: VXLAN EVPN Multi-Site with L2 VNI stitching

As depicted in Figure 6, the border leaf offers multiple functions, including local endpoint attachment and Border Gateway for integrated Layer 2 and Layer 3 service extension.

vPC must be used for any locally dual-homed endpoints at layer 2. This picture depicts a BGP-based deployment of the Border Gateways with Layer 3 connections toward an external Layer 3 network.

VXLAN EVPN Multi-site with Legacy Data Centers

The VXLAN EVPN Multi-Site solution is not limited to interconnecting modern Greenfield Data Centres. It offers two additional flavours for seamless network connectivity with Brownfield Data Centres.

The first use-case is with a legacy DC alone devoid of any DCI service. For this option, if the enterprise wants to build two new Greenfield DCs and keep the original one, with all sites interconnected together, it is possible to deploy a pair of Border Gateways, dual-homed toward the aggregation layer. This function of Border Gateway mapping Dot1Q to VXLAN is also known as Pseudo BGW. The VLANs to be extended are extended using VXLAN EVPN toward the brand-new VXLAN Multi-Site, as shown in Figure 7.

Figure 7: Integration with traditional Data Centre

The second use-case concerns traditional Data Centre networks to interconnect, however these DC are planning to migrate smoothly to VXLAN EVPN fabric. With that migration in mind, it is possible to leverage the Pseudo BGW fonction, leveraging VXLAN EVPN Multi-Site as shown in Figure 8.

Figure 8: Integration with traditional DC Network

Figure 8 illustrates two legacy Data Centre infrastructures interconnected using the Pseudo BGW feature. This is an alternative to OTV when there is a migration planned to a VXLAN EVPN-based fabric. As a result, this option can be leveraged a pair of Pseudo-BGWs inserted in each legacy site to extend Layer-2 and Layer-3 connectivity between sites that will be reused afterward for full Multi-site functions. It offers a slowly phase out the legacy networks replacing them with VXLAN EVPN fabrics. It allows integration with existing legacy default gateway (HSRP, VRRP, etc.) with a smooth migration for a more efficient and integrated Layer 3 Anycast Gateway offering the default gateways for all Endpoints.

Finally this option offers a more solid and integrated solutions than a simple VXLAN EVPN-based solution, with features natively embedded with the Border Gateway functions in order to improve the sturdiness from end-to-end, such as Split Horizon, Designated Forwarded for BUM traffic, Traffic Control of BUM (Disable or Rate-based), Control Plane with Selective Advertisement for L2 & L3 segments.

VXLAN EVPN Multi-site with outside Layer 3 connectivity

Connecting the VXLAN EVPN fabric in conjunction with Multi-site to the outside of the world.

Every DC fabric needs external connectivity to the campus or core network (WAN/MAN).

The VXLAN EVPN fabric can be connected across Layer 3 boundaries using MPLS L3VPN, Virtual Routing and Forwarding (VRF) IP routing (VRF-lite), or Locator/ID Separation Protocol (LISP).

Integration can occur for the Layer 3 segmentation toward a WAN/MAN, native or segmented Layer 3 network. If required, each Tenant from the VXLAN EVPN Fabric can connect to a specific L3 VPN network, like achieved previously with the Multi-Pod and Multi-Fabric models. In this case, in addition to the Multi-Site service, either VRF-Lite are handed off at the Border Leaf nodes to be bound with their external segmentations while the L3 segmentation is also maintained with the remote Fabric, as illustrated in Figure 9.

Figure 9: VXLAN EVPN Multi-site and VRF-Lite

In deployments with a very large number of extended VRF instances, the number of eBGP sessions or IGP neighbors can cause scalability problems and configuration complexity and usually require a two-box solution.

It is now possible to merge the Layer 3 segmentation with the external Layer 3 connectivity (Provider Edge Router) with a new single-device solution.

Figure 10 depicts a full integration of the Layer 3 segmentation provided by the VXLAN EVPN fabric, extended for external connectivity from the BGW toward an MPLS L3VPN network (or any type of Layer 3 network, including LISP). This model requires that the WAN edge device is Border Provider Edge (BorderPE) feature capable. This feature allow to terminate the VXLAN tunnel (VTEP) in order the stitch the L3 VNI from the VXLAN fabric to a Layer3 VPN network. This feature is supported with the Nexus 7k/M3, ASR1k, and ASR9k.

Figure 10: VXLAN EVPN Multi-site integrated Border PE

VXLAN EVPN Multi-site and BUM traffic

Finally, the technology used to replicate BUM traffic is independent per site. One site can run PIM ASM Multicast to replicate the BUM traffic inside the same VXLAN domain regardless what the replication mode used on other sites. The BUM traffic is carried across the Multi-Site overlay using Ingress Replication, while other sites can run whatever is desired by the Network team, Multicast or Ingress Replication. Although Ingress Replication allows BUM transport across any WAN network, VXLAN Multi-Site will support Multicast replication later in a next software release.

To prevent any risks of loop with BUM traffic, a Designated-Forwarder is elected dynamically on a per Layer 2 segment basis. The DF uses its Physical IP address as a unique destination from the BGW cluster.

VXLAN EVPN Multi-site and Rate Limiters

The decapsulation and encapsulation of the respective tunnels happen inside the same VTEP. This method allows for the enabling of traffic policers per tenant, controlling the rate of Storm independently for Broadcast, Unknown Unicast or Multicast traffic, or other policers limiting the bandwidth per tunnel. Therefore, failure containment is improved through the granular control of BUM traffic, protecting remote sites from any broadcast storms.

Figure 12: VXLAN EVPN Multi-Site and Broadcast Storm control (BUM Control)

DC Interconnection versus DC Integration

With VXLAN EVPN Multi-Site coming shortly, we will consider 3 main use cases for interconnecting multiple DC network:

• Interconnecting Legacy to Legacy DC

For Brownfield to Brownfield DC Interconnection, if OTV previously existed for DC interconnection requirements, then, it’s likely to continue using the same DCI approach. There is no technical reason to change to a different DCI solution if the OTV Edge devices are already available. Indeed, OTV offers an easy way to add new DC into its Layer 2 Overlay network.

If a new project to interconnect multiple Brownfield DCs (Classical Ethernet-based DC), it is worth to understand if there is a requirement to migrate in the near future the existing legacy network to a modern fabric infrastructure, as depicted with Figure 8.

Nonetheless, we never want to extend all VLANs from site to site, hence, even if not often elaborated in the DCI documentation, it is important to take the routed traffic inter-site into consideration. The new function of Border Gateway integrated with VXLAN EVPN Multi-site offers a great alternative for interconnecting Data Centre for Layer 2 and Layer 3 requirements, maintaining the in-dependency network transport of each DC, while reducing the failure domain to a single site with a built-in very granular Storm controller for Broadcast, Unknown Unicast or Multicast traffic.

Although OTV 2.5 relies on VXLAN as the encapsulation method, VXLAN continues to evolve with the necessary embedded intelligence leveraging the open standards MP-BGP EVPN control plane to provide a viable, efficient and solid DCI solution comparable with OTV. VXLAN EVPN Multi-site architecture represents this evolution. It offers a single integrated solution for interconnecting multiple traditional Data Centres with embedded and distributed Layer 3 Anycast gateway from the same Border Gateway devices, extending the classical Ethernet frame and routed packets via its Overlay network.

• Interconnecting Legacy DC to modern Fabric

Figure 13: Brownfield to Greenfield Data Centre Interconnection

For the second, as depicted in Figure 13, VXLAN EVPN Multi-site can be leveraged to interconnect the Brownfield DC by adding a new pair of N93xx-EX (or FX) series running VXLAN Multi-site with the Pseudo Border Gateway function for controlling the VLAN to VNI transport. On the remote site, the Greenfield DC leverages the Multi-site feature that integrates the Layer 2 and Layer 3 services toward the Brownfield DC, all functions offered in a single-box model.

• Interconnecting modern Fabrics

For Standalone VXLAN EVPN fabrics, the solution is VXLAN EVPN Multi-Site. There is no technical reason to add a dedicated DCI solution inside a modern Fabrics when Layer 2 and Layer 3 services are integrated in the same Border leaf or Border Spine layer, reducing the failure domain to its smallest diameter, that is to say, the link between the host and its Top of Rack switch.

Naming Convention: Pod, Multi-Pod, Multi-Fabric, Multi-Site

The same naming convention for Pod, Multi-Pod, Multi-Fabric and Multi-Site has been elaborated for both VXLAN EVPN Standalone and ACI.

Nonetheless, it is important to specify that the three models discussed in the previous sections differ when comparing VXLAN Standalone with ACI Fabric.

In a VXLAN context deployment, a Pod is seen as a “Fault Domain” area.

The Pod is usually represented by a Leaf-Spine Network Fabric sharing a common Control-plane (MP-BGP) across its whole physical connectivity plant.

A Fabric represents the VXLAN Domain deployed in the same location (same building or Campus) or Geographically Dispersed (Metro distances) sharing a common Control-plane and Data-plane. A single or multiple Pods form therefore a single VXLAN fabric or VXLAN domain.

VXLAN Multi-Pod – Single Availability Zone

In VXLAN Stand-alone the Multi-Pod architecture is organized with multiple PoD sharing a common Control-plane (MP-BGP) and Data-Plane across the whole physical connectivity plant across the same VXLAN Domain. The VXLAN Multi-Pod represents a single Availability Zones (VXLAN Fabric) within a Single Region – it is strongly recommended to limit the distances for a single Availability Zone within a Metro area, but technically nothing prevent to deploy a VXLAN Multi-Pod across intercontinental distances. The same VXLAN domain is stretched across multiple locations. The term of Multi-Pod differs from the VXLAN EVPN stretched fabric because the strength of the whole solution has been improved as discussed above.

ACI already offers a more sophisticated and solid integration of Layer 2 and Layer 3 services with its Multi-Pod solution since CY2016.

VXLAN Multi-Fabric – Multiple Availability Zones

VXLAN Multi-Fabric deployment offers Multiple Availability Zone (Multiple VXLAN Fabrics) designs, one per VXLAN Domain managed by its own isolated Control-plane with an independent Layer 2 and Layer 3 DCI services. There is no concept of Regions per se, as each VXLAN Fabric runs autonomously.

VXLAN Multi-site – Multiple Availability Zones

VXLAN Multi-Site offers the same concept as Multi-Fabric with independent VXLAN Fabric but with the extended Layer 2 and Layer 3 services integrated and fully controlled. As a result, Multiple autonomous VXLAN domains with integrated L2 & L3 services form the VXLAN Region.

Good day,

In the meantime that this long series of sub-posts becomes a white paper, there is a new document available on CCO written by Lukas Krattiger that covers the Layer 2 and Layer 3 interconnection of multiple VXLAN fabrics. I feel this document is complementary to this series of Post 36, describing from a different angle (using a human language approach) how to achieve VXLAN EVPN Multi-Fabric design in conjunction with OTV.

Optimizing Layer 2 DCI with OTV between Multiple VXLAN EVPN Fabrics (Multifabric) White Paper

Good reading, yves

Ingress/Egress Traffic Path Optimization

In the VXLAN Multi-fabric design discussed in this post, each data center normally represents a separate BGP autonomous system (AS) and is assigned a unique BGP autonomous system number (ASN).

Three types of BGP peering are usually established as part of the VXLAN Multi-fabric solution:

• MP internal BGP (MP-iBGP) EVPN peering sessions are established in each VXLAN EVPN fabric between all the deployed leaf nodes. As previously discussed, EVPN is the intrafabric control plane used to exchange reachability information for all the endpoints connected to the fabric and for external destinations.
• Layer 3 peering sessions are established between the border nodes of separate fabrics to exchange IP reachability information (host routes) for the endpoints connected to the different VXLAN fabrics and the IP subnets that are not stretched (east-west communication). Often, a dedicated Layer 3 DCI network connection is used for this purpose. In a multitenant VXLAN fabric deployment, a separate Layer 3 logical connection is required for each VRF instance defined in the fabric (VRF-Lite model). Although either eBGP or IGP routing protocols can be used to establish interfabric Layer 3 connectivity, the eBGP scenario is the most common and is the one discussed in this post.
• Per-VRF eBGP peering sessions are frequently used for WAN connectivity to exchange IP reachability information with the external Layer 3 network domain (north-south communication). A common best practice and the recommended approach is to deploy eBGP for this purpose.

When extending IP subnets across separate VXLAN fabrics, you need to consider the paths used for ingress and egress communication. You particularly should avoid establishing an asymmetric path such as one shown in Figure 1 (virtual machines VM1 and VM3 are part of the same extended IP subnet that is advertised in the MAN and WAN), which would cause communication failure when independent stateful network services are deployed across sites.

Figure 1: Undesired Asymmetric Flow Establishment

The solution presented in this article avoids establishing asymmetric traffic paths by following three main design principles, illustrated in Figure 2:

• Traffic originating from the external Layer 3 domain and destined for endpoints connected to a specific VXLAN fabric should always come inbound through the site’s local border nodes (ingress traffic-path optimization).
• Traffic originating from data center endpoints and destined for the external Layer 3 domain should always prefer the outbound path through the local border nodes (outbound traffic-path optimization).
• All east-west routed communication between endpoints that are part of different data center sites should always prefer the dedicated Layer 3 DCI connection if it is available.

Figure 2: Ingress/Egress Traffic Path Optimization and East-West Communication

Ingress Traffic Path Optimization

This article proposes the use of host-route advertisement from the border nodes to the local WAN edge routers to influence and optimize ingress traffic flows. After the WAN edge routers receive the host routes, two approaches are possible:

• You can inject specific host-route information from each VXLAN fabric into the MAN or WAN so that incoming traffic can be optimally steered to the destination. This method is usually applicable when a Layer 3 VPN hand-off is deployed to the WAN, so that host routes can be announced in each specific Layer 3 VPN service. Before adopting this approach, be sure to assess the scalability implications of the solution for consumption of local resources on the data center WAN edge and remote routers and within the WAN provider network.
• In designs in which advertisement of host routes in the MAN or WAN is not desirable because of scalability concerns or is not possible (as, for example, in many cases in which the MAN or WAN is managed by a service provider), you can deploy Cisco Location Identifier Separation Protocol (LISP) as an IP-based hand-off technology. LISP deployment is beyond the scope of this paper. For more information about LISP and LISP mobility, see http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Data_Center/DCI/5-0/LISPmobility/DCI_LISP_Host_Mobility.html.

The rest of this section focuses on the deployment model using a Layer 3 VPN hand-off to the MAN or WAN. The goal is to help ensure that the border nodes in each VXLAN fabric always advertise to the local WAN edge routers only host routes for endpoints connected to the local fabric.

As shown in Figure 3, the border nodes at a given site receive host route information for endpoints connected to remote fabrics through the route peering established over the dedicated Layer 3 DCI connection. You therefore must help ensure that these host routes advertised from the Layer 3 DCI are never sent to the local WAN edge routers. Host routing from the WAN always should steer the traffic to the fabric to which those destinations are connected, and a less specific route should be used only in specific WAN isolation scenarios discussed later in this section.

Figure 3: Border Nodes advertising only Host Routes for Locally Connected Endpoints

Many different approaches can be used to achieve the behavior shown in Figure 3. The example shown here simply proposes configuring the border nodes in a given fabric so that they do not announce to the local WAN edge routers host route information received through the Layer 3 DCI connection.

ip as-path access-list 1 permit "_65200_”
!
ip prefix-list MATCH-HOST-ROUTES seq 5 permit 0.0.0.0/0 eq 32
!
ip access-list ANY
  10 permit ip any any
!
route-map DENY-HOST-ROUTES-FROM-REMOTE-DCs deny 10
  match as-path 1
  match ip address prefix-list MATCH-HOST-ROUTES
!
route-map DENY-HOST-ROUTES-FROM-REMOTE-DCs permit 20
  match ip address ANY 
!
router bgp 65100
  vrf Tenant-1
    neighbor 10.0.1.1
      description Local WAN Edge Device
      remote-as 65300             
      address-family ipv4 unicast
        route-map DENY-HOST-ROUTES-FROM-REMOTE-DCs in

Note: The same configuration must be applied to all the border nodes facing the WAN edge and deployed across VXLAN fabrics.

Note that other prefixes received from remote sites are still accepted, as long as they are not host routes. This behavior is required mainly to handle the potential WAN isolation scenario in which a given fabric loses connectivity to the MAN or WAN (as a result of a dual failure of the WAN edge routers or a WAN or MAN outage). In that case, traffic originating from the external routed domain and destined for an endpoint connected to the WAN-isolated fabric should be steered to a different VXLAN fabric following a less specific route (usually the IP prefix for the subnet to which the destination endpoint is connected). This scenario is shown in Figure 4.

Figure 4: Inbound Traffic in a WAN Isolation Scenario

Note: The same considerations apply if VXLAN fabric 1 experiences a WAN isolation scenario.

When virtual machine VM1 migrates to data center DC2, the host route information is updated across the VXLAN fabrics, as previously described in the part 4 “Host Mobility Across Fabrics.” As a consequence, local border nodes BL3 and BL4 will start advertising VM1’s host route to the fabric 2 WAN edge router with AS 65200 of fabric 2, and border nodes BL1 and BL2 will stop sending the same host route information to the fabric 1 WAN edge router. As a consequence, traffic destined for VM1 and originating from the Layer 3 MAN or WAN will be steered directly to fabric 2.

Egress Traffic Path Optimization

After you have optimized the ingress traffic, you usually should do the same for the egress traffic to maintain symmetry for the communications with the external Layer 3 domain. As previously mentioned, this optimization is mandatory for deployment across sites with independent stateful network services such as firewalls.

To force the egress traffic to prefer the local WAN connection, you can modify the local-preference value for the prefixes learned through peering with the local WAN edge routers. The local preference is an attribute that routers exchange in the same autonomous system and that tells the autonomous system which path to prefer to reach destinations that are external to it. A path with a higher local-preference value is preferred. The default local-preference value is 100.

In the configuration sample shown here, one of the border nodes in VXLAN EVPN fabric 1 is configured to assign a higher local preference (200) to all the prefixes received from the WAN edge routers in fabric 1 in data center DC1 using the route map INCREASE-LOCAL-PREF-FOR-WAN-ROUTES.

ip access-list ANY
  10 permit ip any any
!
route-map INCREASE-LOCAL-PREF-FOR-WAN-ROUTES permit 10
  match ip address ANY
  set local-preference 200
!
router bgp 65100
  vrf Tenant-1
    neighbor 10.0.1.1
      remote-as 65300
      description eBGP Peering with WAN Edge router in DC1
      address-family ipv4 unicast
        route-map INCREASE-LOCAL-PREF-FOR-WAN-ROUTES in

The same external prefixes received through the Layer 3 DCI connection would instead have the default local-preference value of 100, so the local path will always be preferred (steered with the local preference 200), as shown in Figure 5.

Figure 5: Local Preferences for Outbound traffic

In the WAN isolation scenario previously considered, the border nodes in isolated VXLAN fabric 2 would start using the external prefixes received on the Layer 3 DCI connection from the border nodes in VXLAN fabric 1. This behavior still helps ensure that inbound and outbound communication between VM2 and the WAN remain symmetrical, which you can verify by comparing previous Figure 4 to Figure 6.

Figure 6: Outbound Traffic in a WAN Isolation Scenario

Keeping Inter-Fabric Routing via Layer 3 DCI

The last requirement is to help ensure that all communications between endpoints belonging to different VXLAN fabrics preferably are established using the dedicated Layer 3 DCI connection. This route is desirable because this connectivity usually is characterized by lower latency and higher bandwidth than the path through the MAN or WAN.

To meet this requirement, you must help ensure that even if a prefix (host route or IP subnet) belonging to fabric 2 is received in fabric 1 through the WAN edge router, this latter information is considered less preferable than the information received through the Layer 3 DCI connection.

Recall the configuration previously discussed to optimize the egress traffic flows. In this case, all the routes received by the border nodes from the WAN edge routers are characterized by a local-preference value of 200. You then must add a route map (INCREASE-LOCAL-PREF-FOR-REMOTE-HOST-ROUTES) to the routing updates received from the remote border nodes to help ensure that all the IP prefixes belonging to the remote fabric (that is, received by eBGP updates originating from the autonomous system of the remote data center) have a higher local-preference value (300 in the example in Figure 7).

Figure 7: Increase Local-Preference to Keep Inter-Fabric Routing via Layer 3 DCI

The following sample shows the required configuration.

ip as-path access-list 2 permit "^65200$”
!
ip access-list ANY
  10 permit ip any any
!
route-map INCREASE-LOCAL-PREF-FOR-REMOTE-FABRIC-ROUTES permit 10
  match as-path 2
  set local-preference 300
route-map INCREASE-LOCAL-PREF-FOR-REMOTE-FABRIC-ROUTES permit 20
  match ip address ANY
!
router bgp 65100
  vrf Tenant-1
    neighbor 172.16.1.1
      remote-as 65200
      description eBGP Peering with BL Node 1 in DC2
      address-family ipv4 unicast
        route-map INCREASE-LOCAL-PREF-FOR-REMOTE-FABRIC-ROUTES in

As a result of this configuration, all the intersite communication stays on the Layer 3 DCI connection and will use the MAN or WAN path only if this connection completely fails.

Conclusions

VXLAN EVPN Multi-Fabric is a hierarchical network design comprising individual Fabrics interconnected together. The design described in this article focuses on the individuality of the data center domains, allowing independent scale and, more important, independent failure domains. The connectivity between the individual fabric domains is independent of the choice that is being used within the data center, and thus a natural separation is achieved.

Overlay Transport Virtualization (OTV) is the recommended technology to provide Layer 2 extension while maintaining failure containment. With this ability and the additional attributes OTV offers for Data Center Interconnect (DCI), modern data center fabrics can be extended in an optimized fashion.

Different solutions can also be adopted to extend multi-tenant Layer 3 connectivity across Fabrics, mainly depending on the nature of the transport network interconnecting them.

Finally, specific deployment considerations and configuration can be used to symmetrize inbound and outbound traffic flows. This is always desirable to optimize access to Data Center resources that can be spread across different Fabrics and becomes mandatory when independent sets of stateful network services (as firewalls for example) are deployed in separate Fabrics.

Host Mobility across Fabrics

This section discusses support for host mobility when a distributed Layer 3 Anycast gateway is configured across multiple VXLAN EVPN fabrics.

In this scenario, VM1 belonging to VLAN 100 (subnet_100) is hosted by H2 in fabric 1, and VM2 on VLAN 200 (subnet_200) initially is hosted by H3 in the same fabric 1. Destination IP subnet_100 and subnet_200 are locally configured on leaf nodes L12 and L13 as well as on L14 and L15.

This example assumes that the virtual machines (endpoints) have been previously discovered, and that Layer 2 and 3 reachability information has been announced across both sites as discussed in the previous sections.

Figure 1 highlights the content of the forwarding tables on different leaf nodes in both fabrics before virtual machine VM2 is migrated to fabric 2.

Figure 1: Content of Forwarding Tables Before Host Migration

The following steps show the process for maintaining communication between the virtual machines in a host mobility scenario, as depicted in Figure 2

Figure 2: VXLAN EVPN Multifabric and Host Mobility

1. For operational purposes, virtual machine VM2 moves to host H4 located in fabric 2 and connected to leaf nodes L21 and L22.
2. After the migration process is completed, assuming that VMware ESXi is the hypervisor used, the virtual switch generates a RARP frame with VM2’s MAC address information.

Note: With other hypervisors, such as Microsoft Hyper-V or Citrix Xen, a GARP request is sent instead, which includes the source IP address of the sender in the payload. As a consequence, the procedure will be slightly different than the one described here.

3. Leaf L22 in this example receives the RARP frame and learns the MAC address of VM2 as locally connected. Because the RARP message can’t be used to learn VM2’s IP address, the forwarding table of the devices in fabric 2 still points to border nodes BL3 and BL4 (that is, VM2’s IP address is still known as connected to fabric 1). Leaf L22 also sends an MP-BGP EVPN route-type-2 update in fabric 2 with VM2’s MAC address information. When doing so, it increases the sequence number associated with this specific entry and specifies as the next hop the anycast VTEP address of leaf nodes L21 and L22. The receiving devices update their forwarding tables with this new information.
4. On the data plane, the RARP broadcast frame is also flooded in fabric 2 and reaches border nodes BL3 and BL4, which forward it to the local OTV devices.
5. The OTV AED in fabric 2 forwards the RARP frame across the Layer 2 DCI overlay network to reach the remote OTV devices in fabric 1. The OTV AED device in fabric 1 forwards the frame to the local border nodes.
6. Border nodes BL1 and BL2 learn the MAC address of VM2 from the reception of the RARP frame as locally attached to their Layer 2 interfaces connecting to the OTV AED device. As a consequence, one of the border nodes advertises VM2’s MAC address information in fabric 1 with a route-type-2 BGP update using a new sequence number (higher than the previous number).
7. The forwarding tables for all relevant local leaf nodes in fabric 1 are updated with the information that VM2’s MAC address is now reachable through the anycast VTEP address of border nodes BL1 and BL2.

At this point, all the devices in fabrics 1 and 2 have properly updated their forwarding tables with the new VM2’s MAC address reachability information. This process implies that intrasubnet communication to VM2 is now fully reestablished. However, VM2’s IP address still is known in both fabrics as connected to the old location (that is, to leaf nodes L14 and L15 in fabric 1), so communications still cannot be routed to VM2. Figure 3 shows the additional steps required to update the forwarding tables of the devices in fabrics 1 and 2 with the new reachability information for VM2’S IP address.

Figure 3: Propagation of VM2’s Reachability Information toward Fabric 1

8.   The reception of the route-type-2 MAC address advertisement on leaf nodes L14 and L15 triggers a verification process to help ensure that VM2 is not locally connected anymore. Note that ARP requests to VM2 are locally sent out the local interface to which VM2 was originally connected as well as to fabric 1 and subsequently to fabric 2 through the Layer 2 DCI connection. The ARP request reaches VM2, which responds, allowing leaf nodes L21 and L22 to update the local ARP table and trigger the consequent control-plane updates discussed previously and shown in Figure 16.

9.   After verification that VM2 has indeed moved away from leaf nodes L14 and L15, one of the leaf nodes withdraws VM2’s IP reachability information from local fabric 1, sending an MP-BGP EVPN update. This procedure helps ensure that this information can be cleared from the forwarding tables of all the devices in fabric 1.

10.  Because border nodes BL1 and BL2 also receive the withdrawal of VM2’s IP address, they update the border nodes in the remote fabric to indicate that this information is not reachable anymore through the Layer 3 DCI connection.

11.  As a consequence, border nodes BL3 and BL4 also withdraw this information from remote VXLAN EVPN fabric 2, allowing all the local devices to clear this information from their tables.

The end result is the proper update of VM2’s IP address information in the forwarding tables of all the nodes in both fabrics, as shown in Figure 4.

Figure 4: End State of the Forwarding Tables for Nodes in Fabrics 1 and 2

At this point, Layer 2 and 3 communication with VM2 can be fully reestablished.

Notice

I recommend you to read part 1 and part 2 if you missed them 🙂

Thank you, yves

VXLAN EVPN Multi-Fabric with Distributed Anycast Layer 3 Gateway

Layer 2 and Layer 3 DCI interconnecting multiple VXLAN EVPN Fabrics

A distributed anycast Layer 3 gateway provides significant added value to VXLAN EVPN deployments for several reasons:

• It offers the same default gateway to all edge switches. Each endpoint can use its local VTEP as a default gateway to route traffic outside its IP subnet. The endpoints can do so, not only within a fabric but across independent VXLAN EVPN fabrics (even when fabrics are geographically dispersed), removing suboptimal interfabric traffic paths. Additionally, routed flows between endpoints connected to the same leaf node can be directly routed at the local leaf layer.
• In conjunction with ARP suppression, it reduces the flooding domain to its smallest diameter (the leaf or edge device), and consequently confines the failure domain to that switch.
• It allows transparent host mobility, with the virtual machines continuing to use their respective default gateways (on the local VTEP), within each VXLAN EVPN fabric and across multiple VXLAN EVPN fabrics.
• It does not require you to create any interfabric FHRP filtering, because no protocol exchange is required between Layer 3 anycast gateways.
• It allows better distribution of state (ARP, etc.) across multiple devices.

In the VXLAN EVPN Multi-Fabric with Distributed Anycast Layer 3 Gateway scenario the Border nodes can perform 3 main functions:

1. VLAN and VRF-Lite hand-off to DCI.
2. MAN/WAN connectivity to the external Layer 3 network domain.
3. Connectivity to Network Services.

For IP subnets that are extended between multiple fabrics, instantiation of the distributed IP anycast gateway on the border nodes is not supported. It is not supported because, with the instantiation of the distributed IP anycast gateway on the border nodes that also extend the Layer 2 network, the same MAC and IP addresses become visible on the Layer 2 extension on both fabrics, and unpredictable learning and forwarding can occur. Even if the Layer 2 network is not extended between fabrics, because it may potentially later be extended, the use of a distributed IP anycast gateway on the border node is not recommended (Figure 1).

Figure 1: No Distributed Anycast Gateway Functionality for Endpoints on Border Nodes

The next sections provide more detailed packet walk descriptions. But before proceeding, keep in mind that MP-BGP EVPN can transport Layer 2 information such as MAC addresses as well as Layer 3 information such as host IP addresses (host routes) and IP subnets. For this purpose, it uses two forms of routing advertisement:

• Route type 2: Used to announce host MAC and IP address information for the endpoint directly connected to the VXLAN fabric and also carrying extended community attributes (such as route-target values, router MAC addresses, and sequence numbers).
• Route type 5: Advertises IP subnet prefixes or host routes (associated, for example, with locally defined loopback interfaces) and also carrying extended community attributes (such as route-target values and router MAC addresses).

Learning Process for Endpoint Reachability Information

Endpoint learning occurs on the VXLAN EVPN switch, usually on the edge devices (leaf nodes) to which the endpoints are directly connected. The information (MAC and IP addresses) for locally connected endpoints is then programmed into the local forwarding tables.

This article assumes that a distributed anycast gateway is deployed on all the leaf nodes of the VXLAN fabric. Therefore, there are two main mechanisms for endpoint discovery:

• MAC address information is learned on the leaf node when it receives traffic from the locally connected endpoints. The usual data-plane learning function performed by every Layer 2 switch allows this information to be programmed into the local Layer 2 forwarding tables.

You can display the Layer 2 information in the MAC address table by using the show mac-address table command. You can display the content of the EVPN table (Layer 2 routing information base [L2RIB]) populated by BGP updates by using the show l2route evpn mac command.

• IP addresses of locally connected endpoints are instead learned on the leaf node by intercepting control-plane protocols used for address resolution (ARP, Gratuitous ARP [GARP], and IPv6 neighbor discovery messages). This information is also programmed in the local L2RIB together with the host route information received through route-type-2 EVPN updates and can be viewed by using the show l2route evpn mac-ip command at the command-line interface (CLI).

Note: Reverse ARP (RARP) frames do not contain any local host IP information in the payload, so they can be used only to learn Layer 2 endpoint information and not the IP addresses.

After learning and registering information (MAC and IP addresses) for its locally discovered endpoints, the edge device announces this information to the MP-BGP EVPN control plane using an EVPN route-type-2 advertisement sent to all other edge devices that belong to the same VXLAN EVPN fabric. As a consequence, all the devices learn endpoint information that belongs to their respective VNIs and can then import it into their local forwarding tables.

Intra-Subnet Communication Across Fabrics

Communication between two endpoints located in different fabrics within the same stretched Layer 2 segment (IP subnet) can be established by using a combination of VXLAN bridging (inside each fabric) and OTV Layer 2 extension services.

To better understand the way that endpoint information (MAC and IP addresses) are learned and distributed across fabrics, start by assuming that source and destination endpoints have not been discovered yet inside each fabric.

The control-plane and data-plane steps required to establish cross-fabric Layer 2 connectivity are highlighted in Figure 2 and described in this section.

Figure 2: VXLAN EVPN Multi-Fabric – ARP Request Propagation across L2 DCI

The various control and data plane steps required to establish cross Fabrics L2 connectivity are highlighted in Figure 2 and described below.

1. Host H1 connected to leaf L11 in fabric 1 initiates intrasubnet communication with host H6, which belongs to remote fabric 2. As a first action, H1 generates a Layer 2 broadcast ARP request to resolve the MAC and IP address mapping for H6 that belongs to the same Layer 2 segment.
2. Leaf L11 receives the ARP packet on local VLAN 100 and maps it to the locally configured L2VNI 10100. Because the distributed anycast gateway is enabled for that Layer 2 segment, leaf L11 learns H1’s MAC and IP address information and distributes it inside the fabric through an EVPN route-type-2 update. This process allows all the leaf nodes with the same L2VNI locally defined to receive and import this information in their forwarding tables (if they are properly configured to do so).
   • Note: The reception of the route-type-2 update triggers also a BGP update for H1’s host route from border nodes BL1 and BL2 to remote border nodes BL3 and BL4 through the Layer 3 DCI connection.
   • This process is not shown in Figure 2, because it is not relevant for intrasubnet communication.On the data plane, leaf L11 floods the ARP broadcast request across the Layer 2 domain identified by L2VNI 10100 (locally mapped to VLAN 100). The ARP request reaches all the local leaf nodes in fabric 1 that host that L2VNI, including border nodes BL1 and BL2. The replication of multidestination traffic can use either the underlay multicast functions of the fabric or the unicast ingress replication capabilities of the leaf nodes. The desired behavior can be independently tuned on a per-L2VNI basis within each fabric.
   • Note: The ARP flooding described in the preceding steps occurs regardless of whether ARP suppression is enabled in the L2VNI, because of the initial assumption that H6 has not been discovered yet.
3. The border nodes decapsulate the received VXLAN frame and use the L2VNI value in the VXLAN header to determine the bridge domain to which to flood the ARP request. They then send the request out the Ethernet interfaces that carry local VLAN 100 mapped to L2VNI 10100. The broadcast packet is sent to both remote OTV end devices. The OTV authoritative edge device (AED) responsible for extending that particular Layer 2 segment (VLAN 100) receives the broadcast packet, performs a lookup in its ARP cache table, and finds no entry for H6. As a result, it encapsulates the ARP request and floods it across the OTV overlay network to its remote OTV peers. The remote OTV edge device that is authoritative for that Layer 2 segment opens the OTV header and bridges the ARP request to its internal interface that carries VLAN 100.
4. Border nodes BL3 and BL4 receive the ARP request from the AED through the connected OTV inside interface and learn on that interface through the data plane the MAC address of H1. As a result, the border nodes announce H1’s MAC address reachability information to the local fabric EVPN control plane using a route-type-2 update. Finally, the ARP request is flooded across L2VNI 10100, which is locally mapped to VLAN 100.
5. All the edge devices on which L2VNI 10100 is defined receive the encapsulated ARP packet, including leaf nodes L23 and L24, which are part of a vPC domain. One of the two leaf nodes is designated to decapsulate the ARP request and bridge it to its local interfaces configured in VLAN 100 and locally mapped to L2VNI 10100, so H6 receives the ARP request.

At this point, the forwarding tables of all the devices are properly populated to allow the unicast ARP reply to be sent from H6 back to H1, as shown in Figure 3.

Continue reading →

Notice

I recommend you to read part 1 if you missed it 🙂

thank you, yves

VXLAN EVPN Multi-Fabric with External Active/Active Gateways

The first use case is simple. Each VXLAN fabric behaves like a traditional Layer 2 network with a centralized routing block. External devices (such as routers and firewalls) provide default gateway functions, as shown in Figure 1.

Figure 1: External Routing Block IP Gateway for VXLAN/EVPN Extended VLAN

In the Layer 2–based VXLAN EVPN fabric deployment, the external routing block is used to perform routing functions between Layer 2 segments. The same routing block can be connected to the WAN advertising the public networks from each data center to the outside and to propagate external routes to each fabric.

The routing block consists of a “router-on-a-stick” design (from the fabric’s point of view) built with a pair of traditional routers, Layer 3 switches, or firewalls that serve as the IP gateway. These IP gateways are attached to a pair of vPC border nodes that initiate and terminate the VXLAN EVPN tunnels.

Connectivity between the IP gateways and the border nodes is achieved through a Layer 2 trunk carrying all the VLANs that require routing services.

To improve performance with active default gateways in each data center, reducing the hairpinning of east-west traffic for server-to-server communication between sites, and depending on the IP gateway platform of choice, the routing block can be duplicated with the same virtual IP and MAC addresses for all relevant SVIs on both sides. Hence, to use active-active gateways on both fabrics, you must filter communications between gateways that belong to the same First-Hop Routing Protocol (FHRP) group. With OTV as the DCI solution, the FHRP filter will be applied to the OTV control plane.

Figure 2 shows this scenario.

Note: Although VLANs are locally significant per edge device (or even per port), and the Layer 2 virtual network identifier (VNI) is locally significant in each VXLAN EVPN fabric, the following examples assume that the same Layer 2 VNIs (L2VNIs) are reused on both fabrics. The same VLAN ID was also reused on each leaf node and on the border switches. This approach was used to simplify the diagrams and packet walk. In a real production network, the network manager can use different network identifiers for the Layer 2 and 3 VNIs deployed in the individual fabrics.

Figure 2: VXLAN EVPN Layer 2 Fabric with External Routing Block

Figure 2 shows the following:

Note: The basic assumption is that H1 and H2 have already populated their ARP tables with the default gateway information, for example because they previously sent ARP requests targeted to the gateway. As a consequence, the gateway also has H1 and H2 information in its local ARP table.

1. Host H1 connected to leaf L11 in fabric 1 needs to send a data packet to host H2 connected in vPC mode to a pair of leaf nodes, L12 and L13, in the same fabric 1. Because H1 and H2 are part of different IP subnets, H1 sends the traffic to the MAC address of its default gateway, which is deployed on an external Layer 3 device connected to the same fabric 1. The communication between H1 and the default gateway uses the VXLAN fabric as a pure Layer 2 overlay service.
2. Traffic from H1 that belongs to VLAN 100 is VXLAN encapsulated by leaf L11 with L2VNI 10100 (locally mapped to VLAN 100) and sent to the egress anycast VTEP address defined in border nodes 1 and 2. This address represents the next hop to reach the MAC address of the default gateway. Layer 3 equal-cost multipath (ECMP) is used in the fabric to load-balance the traffic destined for the egress anycast VTEP between the two border nodes. In the example in Figure 9, border node BL1 is selected as the destination.
3. BL1 decapsulates the VXLAN frame and bridges the original frames destined for the local default gateway onto VLAN 100 (locally mapped to L2VNI 10100).
4. The default gateway receives the frame destined for H2, performs a Layer 3 lookup, and subsequently forwards the packet to the Layer 2 segment on which H2 resides.
5. The Layer 2 flow reaches one of the vPC-connected border nodes (BL2 in this example). BL2 uses the received IEEE 802.1q tag (VLAN 200) to identify the locally mapped L2VNI, 10200, to be used for VXLAN to encapsulates the frame. BL2 then forwards the data packet to the anycast VTEP address defined on the vPC pair of leaf nodes, L12 and L13, on which the destination H2 is connected.
6. One of the receiving leaf nodes is designated to decapsulate the VXLAN frame and send the original data packet to H2.
7. Routed communications between endpoints located at the remote site are kept local within fabric 2. This behavior is possible only because FHRP filtering is enabled on the OTV edge devices. In this example, H4 sends traffic destined for H6 using its local default gateway active in data center DC2. East-west routed traffic can consequently be localized within each fabric, eliminating unnecessary interfabric traffic hairpinning.

Notices

 

Since this article was published 2 years ago, for 1 year we have now a integrated and hierarchical solution to interconnect multiple VXLAN EVPN fabrics called Multi-site (post 37) that offers Layer 2 and Layer 3 extension in a much more efficient, more robust and easier way to deploy and maintain the extension of layer 2 and layer 3 networks.

However I want to keep this article still live there because there are several sections such as the packet walk that can help to better understand the learning and distribution handshake mechanisms used by EVPN. An additional reason is path optimisation as the key solutions described in part#5 is still valid with Multis-site.

With my friend and respectful colleague Max Ardica, we have tested and qualified the current solution to interconnect multiple VXLAN EVPN fabrics. We have elaborated this technical support to clarify the network design requirements when the function Layer 3 Anycast gateways is distributed among all server node platforms and all VXLAN EVPN Fabrics. The  whole article is organised in 5 different posts.

• This 1st part below elaborates the design considerations to interconnect two VXLAN EVPN based fabrics.
• The 2nd post discusses the Layer 2 DCI requirements interconnecting Layer-2-based VXLAN EVPN fabrics deployed in conjunction with Active/Active external routing blocks.
• The 3rd section covers the Layer 2 and Layer 3 DCI requirement interconnecting VXLAN EVPN  fabrics deployed in conjunction with distributed Layer 3 Anycast Gateway.
• The 4th post examines host mobility across two VXLAN EVPN Fabrics with Layer 3 Anycast Gateway.
• Finally the last section develops inbound and outbound path optimization with VXLAN EVPN fabrics geographically dispersed.

Introduction

Recently, fabric architecture has become a common and popular design option for building new-generation data center networks. Virtual Extensible LAN (VXLAN) with Multiprotocol Border Gateway Protocol (MP-BGP) Ethernet VPN (EVPN) is essentially becoming the standard technology used for deploying network virtualization overlays in data center fabrics.

Data center networks usually require the interconnection of separate network fabrics, which may also be deployed across geographically dispersed sites. Consequently, organizations need to consider the possible deployment alternatives for extending Layer 2 and 3 connectivity between these fabrics and the differences between them.

The main goal of this article is to present one of these deployment options, using Layer 2 and 3 data center interconnect (DCI) technology between two or more independent VXLAN EVPN fabrics. This design is usually referred to as VXLAN multi-fabrics.

To best understand the design presented in this article, the reader should be familiar with VXLAN EVPN, including the way it works in conjunction with the Layer 3 anycast gateway and its design for operation in a single site. For more information, see the following Cisco® VXLAN EVPN documents:

• VXLAN Overview: Cisco Nexus 9000 Series Switches
• VXLAN Network with MP-BGP EVPN Control Plane Design Guide
• VXLAN Design with Cisco Nexus 9300 Platform Switches
• VXLAN Configuration Guide

To extend Layer 2 as well as Layer 3 segmentation across multiple geographically dispersed VXLAN-based fabrics, you can consider two main architectural approaches: VXLAN EVPN Multi-pods fabric and VXLAN EVPN Multi-fabrics.

Option 1: VXLAN EVPN Multi-Pod Fabric

You can create a single logical VXLAN EVPN fabric in which multiple pods are dispersed to different locations using a Layer 3 underlay to interconnect the pods. Also known as stretched fabric, this option is now more commonly called VXLAN EVPN Multi-pods fabric. The main benefit of this design is that the same architecture model can be used to interconnect multiple pods within a building or campus area as well as across metropolitan-area distances.

The VXLAN EVPN Multipod design is thoroughly discussed in this post 32 – VXLAN Multipod stretched across geographically dispersed datacenters

Option 2: VXLAN EVPN Multi-Fabric

You can also interconnect multiple VXLAN EVPN fabrics using a DCI solution that provides multitenant Layer 2 and Layer 3 connectivity services across fabrics. This article focuses on this option.

Compared to the VXLAN EVPN Multi-pod fabric design, the VXLAN EVPN Multi-fabric architecture offers greater independence of the data center fabrics. Reachability information for VXLAN tunnel endpoints (VTEPs) is contained in each single VXLAN EVPN fabric. As a result, VXLAN tunnels are initiated and terminated within the same fabric (in the same location). This approach helps ensure that policies, such as storm control, are applied specifically at the Layer 2 interface level between each fabric and the DCI network to limit or eliminate storm and fault propagation across fabrics.

The Multi-fabric deployment model also offers greater scalability than the Multi-pod approach, supporting a greater total number of leaf nodes across fabrics. The total number of leaf devices supported equals the maximum number of leaf nodes supported in a fabric (256 at the time of the writing of this post) multiplied by the number of interconnected fabrics. A Multi-fabric deployment can also support an overall greater number of endpoints than a multipod deployment; commonly, only a subset of Layer 2 segments is extended across separate fabrics, so most MAC address entries can be contained within each fabric. Additionally, host route advertisement can be filtered across separate fabrics for the IP subnets that are defined only locally within each fabric, increasing the overall number of supported IP addresses.

Separate VXLAN fabrics can be interconnected using Layer 2 and 3 functions, as shown in Figure 1:

• VLAN hand-off to DCI for Layer 2 extension
• Virtual Routing and Forwarding Lite (VRF-Lite) hand-off to DCI for multitenant Layer 3 extension

Figure 1: VLAN and VRF-Lite Hand-Off

The maximum distance between separate VXLAN EVPN fabrics is determined mainly by the application software framework requirements (maximum tolerated latency between two active members) or by the mode of disaster recovery required by the enterprise (hot, warm, or cold migration).

After Layer 2 and 3 traffic is sent out of each fabric through the border nodes, several DCI solutions are available for extending Layer 2 and 3 connectivity across fabrics while maintaining end-to-end logical isolation.

• Layer 2 DCI:

Layer 2 connectivity can be provided in several ways. It can be provided through a dedicated Layer 2 dual-sided virtual port channel (vPC) for metropolitan-area distances using fiber or dense wavelength-division multiplexing (DWDM) links between two sites. For any distance, it can be provided any valid Layer 2 overlay technology over a Layer 3 transport mechanism such as Overlay Transport Virtualization (OTV), Multiprotocol Label Switching (MPLS) EVPN, VXLAN EVPN, or Virtual Private LAN Service (VPLS).

• Layer 3 DCI:

The Layer 3 DCI connection has two main purposes:

1. It advertises between sites the network prefixes for local hosts and IP subnets.
2. It propagates to the remote sites host routes and subnet prefixes for stretched IP subnets.

The DCI solutions selected for extending multitenant Layer 2 and 3 connectivity across VXLAN EVPN fabrics usually depend on the type of service available in the transport network connecting the fabrics:

Scenario 1: Enterprise-owned direct links (dark fibers or DWDM circuits)

• vPC or OTV for Layer 2 extension
• Back-to-back VRF-Lite subinterfaces for Layer 3 extension

Scenario 2: Enterprise-owned or service provider–managed multitenant Layer 3 WAN service

• OTV for Layer 2 extension through a dedicated VRF-Lite subinterface
• VRF-Lite subinterfaces to each WAN Layer 3 VPN service

Scenario 3: Enterprise-owned or service provider–managed single-tenant Layer 3 WAN service

• OTV for Layer 2 extension across native Layer 3
• VRF-Lite over OTV for Layer 3 segmentation

Note: In this article, OTV is the DCI solution of choice used to extend Layer 2 connectivity between VXLAN fabrics. OTV is an IP-based DCI technology designed purposely to provide Layer 2 extension capabilities over any transport infrastructure. OTV provides an overlay that enables Layer 2 connectivity between separate Layer 2 domains while keeping these domains independent and preserving the fault-isolation, resiliency, and load-balancing benefits of an IP-based interconnection. For more information about OTV as a DCI technology and associated deployment considerations, refer to the document at  :

http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Data_Center/DCI/whitepaper/DCI3_OTV_Intro/DCI_1.html.

Scenario 1: In a metropolitan area network (MAN), you can use a direct DWDM transport for Layer 2 and 3 extension. Therefore, dedicated links for Layer 2 and 3 segmentation are established using different DWDM circuits (or direct fiber when the network is owned by the enterprise). Figure 2 shows two separate physical interfaces. One is used for Layer 3 segmentation, and the other is used for Layer 2 extension (classic Ethernet or overlay). A dedicated Layer 3 connection in conjunction with vPC is a best practice, and it is usually required for platforms that do not support dynamic routing over vPC. However, using border node platforms (such as Cisco Nexus® 7000 or 9000 Series Switches) and software releases, you also can use Layer 3 over a vPC dual-sided connection (not covered in this article).

Note: Dynamic routing over vPC is supported on Cisco Nexus 9000 Series Switches starting from Cisco NX-OS Software Release 7.0(3)I5(1).

Figure 2- Layer 2 and Layer 3 segmented across a DWDM network

Nonetheless, the recommended approach is to deploy an overlay solution to provide Layer 2 DCI services such as OTV, especially if more than two sites need to be interconnected (a scenario not documented here). OTV inherently offers multipoint Layer 2 DCI services while helping ensure robust protection against the creation of end-to-end Layer 2 loops.

Note: At this stage, the OTV encapsulation can be built with a generic routing encapsulation (GRE) header or with a VXLAN and User Datagram Protocol (UDP) header. The latter option is preferred because it allows better load balancing of OTV traffic across the Layer 3 network that interconnects the VXLAN fabrics. However, it requires the use of F3 or M3 line cards and NX-OS Release 7.2 or later.

• OTV edge devices are connected to the border nodes with a Layer 3 and a L2 connection as shown in Figure 3, a deployment model referred to as “on a stick”. The OTV overlay is carried within a dedicated Layer 3 subinterface and VRF instance. This subinterface is carved on the same physical interface that is used to provide cross-fabric Layer 3 connectivity for all the tenants (discussed later in this section) and can belong to the default VRF instance or to a dedicated OTV VRF instance.

Note: Dual-sided vPC can be used as a Layer 2 DCI alternative for VXLAN dual-fabric deployments, though it is less preferable due to the lack of various failure isolation functions natively offered by OTV.

• Dedicated back-to-back subinterfaces carry OTV encapsulated traffic and provide Layer 3 Tenant connectivity.
  • OTV: Default VRF = E1/1.999
  • Tenant 1: VRF T1 = E1/1.101
  • Tenant 2: VRF T2 = E1/1.102
  • Tenant 3: VRF T3 = E1/1.103
  • etc..

Figure 3 : Physical View with OTV “on the stick” to carry Intra-subnet communication

Scenario 2: The second scenario applies to deployments in which a Layer 3 multitenant transport service is available across separate fabrics.

This deployment model is similar to the one in scenario 1: OTV edge devices “on a stick” are used to extend Layer 2 connectivity across sites. The OTV traffic is sent out of the border nodes on dedicated subinterfaces that are mapped to a specific Layer 3 VPN service on the WAN edge router acting as a provider edge device.

At the same time, independent Layer 3 subinterfaces are deployed for each tenant for end-to-end Layer 3 communication. Each subinterface is then mapped to a dedicated Layer 3 VPN service.

Figures 4 and 5 show this scenario.

Figure 4: OTV Overlay “on a stick” and VRF-Lite sessions run independently

Figure 5 : Physical View with OTV “on a stick” to carry Intra-subnet communication – Layer 3 VPN / MPLS Core

Scenario 3: In this scenario, the WAN and MAN transport network offers only single-tenant Layer 3 service (it does not support MPLS or VRF-Lite).

In this case, the Layer 2 DCI services offered by an overlay technology such as OTV can be used to establish cross-fabric Layer 2 and 3 multitenant connectivity. This connection is achieved in two steps:

• First, a Layer 2 DCI overlay service is established over the native Layer 3 WAN and MAN core.
• Next, per-tenant Layer 3 peerings are established across the fabrics over this Layer 2 overlay transport. The dedicated tenant Layer 3 interfaces to connect to the remote sites can be deployed as Layer 3 subinterfaces or as switch virtual interfaces (SVIs). Your choice has an impact on the physical connectivity between the border nodes and the OTV edge devices:
  • Use of subinterfaces for tenant-routed communication: In this case, you must deploy two separate internal interfaces to connect the border nodes to the OTV devices. As shown in Figure 6, the first interface is configured with subinterfaces to carry tenant traffic (a unique VLAN tag is associated with each tenant), and the second interface is a Layer 2 trunk used to extend Layer 2 domains between sites.

Figure 6: Layer 3 VPN peering over the Layer 2 Overlay Transport

Because Layer 3 routing peerings are established for each tenant to the remote border nodes across the OTV overlay service, you must enable bidirectional forwarding detection (BFD) on each Layer 3 subinterface for indirect failure detection, helping ensure faster convergence in those specific cases.

• Use of SVIs for tenant-routed communication: In this case, shown in Figure 7, the same Layer 2 trunk interface can be used to carry the VLANs associated with each tenant Layer 3 interface and the VLANs extending the Layer 2 domains across sites. Note that these two sets of VLANs must be unique, and that SVIs are not defined for the VLANs that extend the Layer 2 domains across sites.

In this specific scenario, the join interfaces of the OTV edge devices can be connected directly to the Layer 3 core, as shown in Figure 7.

Figure 7 – OTV in line to carry data VLANs + VLANs for Tenants Layer 3 communication

Deployment Considerations

The following sections present two use cases in which independent VXLAN EVPN fabrics are interconnected:

• The first use case is a more traditional deployment in which each VXLAN EVPN fabric runs in Layer 2 mode only and uses a centralized external routing block for intersubnet communication. This scenario can be a requirement when the default gateway function is deployed on external devices such as firewalls. This use case is discussed in the sections VXLAN EVPN Multi-Fabrics with External Routing Block (part 2)
• The second use case deploys the distributed functions of anycast Layer 3 gateways across both sites, reducing hairpinning workflow across remote network fabrics. This enhanced scenario can address a variety of requirements:
  1. For business-continuance purposes, it is a common practice to allow transparent “hot” mobility of virtual machines from site to site without any service interruption. To reduce application latency, the same default gateways are active at both sites, reducing hairpinning across long distances for local routing purposes as well as for egress path optimization. Nonetheless, to reduce hairpinning for east-west workflows and to offer north-south traffic optimization, the same MAC and IP addresses for the default gateways must be replicated on all active routers at both sites. With the anycast Layer 3 gateway, the default gateway function is performed by all computing leaf nodes. This scenario is the main focus of this article and is discussed in greater detail in the section VXLAN EVPN Multi-fabric with Distributed Anycast Layer 3 Gateway (part 3).
  2. For disaster-recovery purposes and for operational cost containment, enterprises may want to move endpoints using a “cold” migration process only. Consequently, only the IP address of the gateway will be replicated to simplify operation management after machines have been moved (for example, to maintain the same IP address schema after the relocation of the servers). The MAC address of the default gateway can be different on each fabric, and for a “cold” migration process, the endpoint will apply Address Resolution Protocol (ARP) on its default gateway when restarting its processes and will get the new MAC address accordingly. This use case is not covered in this post.

East-West Communication Intra and Inter-sites

For the following scenario, subnets are stretched across multiple locations using a Layer 2 DCI solution. There are several use cases that require LAN extension between multiple sites, such as Live migration, Health-check probing for HA cluster (heartbeat), Operational Cost containment such as migration of Mainframes, etc.  It is assumed that due to long distances between sites, the network services are duplicated and active on each of the sites. This option allows the use of local network services such as default gateways, load balancer’s and security engines distributed across each location, helps reduce server to server communication latency (East-West work flows).

Traditionally, an IP address uses a unique identifier assigned to a specific network entity such as physical system, virtual machine or firewall, default gateway, etc. The routed WAN uses the identifier to also determine the network entity’s location in the IP subnet. When a Virtual Machine migrates from one data center to another, the traditional IP address schema retains its original unique identifier and location, although the physical location has actually changed. As a result, the extended VLAN must share the same subnet so that the TCP/IP parameters of the VM remain the same from site to site, which is necessary to maintain active sessions for migrated applications.

When deploying a logical data center architecture stretched across multiple locations, with duplicated active network services, a couple of critical behaviors have to be taken into consideration. First, it is extremely important to maintain transparently with no interruption the same level of security within and across sites, before, during and after live migration of any Endpoints. Secondly, while machines follow a stateful migration process (hot live migration) from site to site, each security and network service that owns current active sessions should not reject any sessions due to an asymmetric workflow. It is mandatory that the round-trip data packet flow always reach the stateful device that owns the current session, otherwise the session is broken by the stateful device. To achieve the required symmetric workflow, it is fundamental that each IP Endpoint is localized dynamically and any movement of virtual machines are notified in real time to update all relevant routing tables.

The scenario discussed in the following lines covers security appliances in routed mode, however, the same behavior applies with any stateful devices such as SLB engines, physical appliances or virtual services.

In the following use-case, two physical data centers are interconnected using a Layer 2 DCI solution offering a single logical data center from a multi-tiered application framework point of view. Using OTV it is possible to address intercontinental distances between the two sites in a simple and robust fashion. However, this solution is agnostic to the DCI transport (OTV, VPLS, PBB EVPN, MPLS EVPN, VXLAN EVPN) as long as a method to filter FHRP handshake between datacenter protocol is supported.

The figure below depicts a generic deployment of multiple sites interconnected with a Layer 2 overlay network extended over the Layer 3 connection. The two datacenter tightly close by the Layer 2 extension are organized to support multitenancy with in this example, Tenant Red and Tenant Green elaborated latter.

DC Multi-sites – High Level Physical Architecture

The goal is to utilize the required services from local stateful devices, reducing latency for server to server communication, in a multi-tenant environment, whilst traffic flows are kept symmetric from end-to-end.

In the following example, when host R1 from tenant Red communicates with host G1 from tenant Green, both located in DC-1, the traffic is inspected by the local FW in DC-1. On the other side, when host R2 communicates with hosts G2, the traffic is inspected by the local FW in DC2.

DC Multi-sites – Localized E-W traffic

If this behavior sounds obvious for independent data centers, with geographically dispersed data centers it becomes a real concern when the broadcast domains are extended across multiple sites, with duplicate active stateful devices.

The following logical view depicts the issue. The traffic from VLAN 10 (subnet Red) destined to VLAN 20 (subnet Green) is routed via each local active Firewall. As a consequence, when host R1 (VLAN 10) in DC-1 needs to communicate with host G2 (VLAN 20) in DC-2, its data packet is routed toward its local FW in DC-1, which in turn, routes the traffic destined to G2 (subnet Green) extended to the remote site. It is therefore needed to extend the L2 for those Subnets to reach the endpoints wherever they are located. The routed traffic is therefore forwarded toward host G2 across the extended Subnet Green (Layer 2 DCI connectivity established across the OTV overlay). G2 receives the data packets and replies as expected to R1 using its local default gateway, which in turn, routes toward its local FW (DC-2) the response destined to R1 (subnet Red). In this design, by default the local FW on each site receives the preferred path for Red and Green Subnet from its local fabric.  Hence, routing between endpoints belonging to those two IP subnets is always kept local to a site. As a result, this behavior affects the workflow that becomes asymmetrical and consequently the FW in DC2 terminates that current session for security reason.

Asymmetric flow with Stateful device not allowed

Two solutions exist based on host-routing to provide the physical location of the IP Endpoint, with and without Layer 2 segments extended between multiple locations. The concept relies on more specific routes (/32) propagated to multiple host databases (DC-1 and DC-2). Having a dynamic database of endpoints associated with their physical location allows to redirect the traffic destined to IP machines of interest over a specific Layer 3 path.

 Host Route Injection

The following section describes a sub-function of LISP that consists injecting the host-route toward the IGP protocol in conjunction with extended subnets across distant locations; this is known as LISP IGP Assist ESM (Extended Subnet Mode). LISP IGP Assist is agnostic to the DCI technology deployed, however because IGP Assist uses a Multicast group to transport host route notifications from site to site, it is important that IP Multicast traffic can be routed across the DCI network.

Note the latter doesn’t mean that the WAN must be IP Multicast capable, it means that Multicast data packet can be carried across all remote locations, thus the choice to OTV in this DCI design, which supports and optimizes MCAST traffic using a Head-end Replication technique (aka OTV Adjacency Server) to transport data Multicast packets over an non-Multicast capable WAN.

LISP IGP Assist can be leveraged to trigger dynamically an update in its Endpoint Identifiers (EID) database for each detection of new machine that belong to a selected subnet. As soon as an Endpoint is powered-on or has moved to a new physical host, it is automatically detected by its Default gateway (LISP First Hop Router function), the LISP process running on the relevant switch (DG) registers the new host reachability information (host route) to its EID database with its location and notifies all remote First Hop Routers accordingly using a dedicated Multicast group across the L2 DCI. Meanwhile, the FHR redistributes the Host route into its IGP routing table, adding the /32 Host route for each local EID.

Based on the above, the key concept is to propagate the host routes dynamically to the remote site using a dedicated Layer 3 DCI network. As a consequence, a more specific route (/32) is announced dynamically to attract the traffic to the EID of interest using a specific path. This L3 DCI connectivity is depicted in the next figure as the Secure L3 DCI connection for inter-site inter-VRF routed traffic.

• R1 and G1 Host routes are propagated toward DC-2 over the Layer 3 DCI.
• R2 and G2 Host routes are propagated toward DC-1 over the Layer 3 DCI.

The following diagram provides further details that will be used for the following test-bed.

Physical Architecture of DC Multi-sites with dedicated L3 DCI

Site to site Inter-VRF secure communication

To summarize the logical view above:

• Hosts R1 and R2 belong to the same VLAN 10 within Tenant Red, while G1 and G2 share the same VLAN 20 that belongs to Tenant Green.
• VLAN 10 and VLAN 20 are extended across the L2 DCI connection established using OTV.
• R1 and G1 are located in DC-1, R2 and G2 are located in DC-2.
• Communications within and between IP subnets belonging to a given DMZ or tenant happens freely, while inter-tenant packet flows must be enforced through the Firewall. As a result, a L3 segmentation (VRF) is performed between tenants to force the routed traffic to use the Firewall. VLAN 10 belongs to VRF Red and VLAN 20 to VRF Green.
• FHRP filter is enabled in conjunction with OTV, so that the same SVI can be active on both sites.
• LISP First Hop Router (FHR) must be configured as the Default Gateways for all hosts of interest as it uses ARP Messages to trigger the host route notification for the EIDs.
• An active Firewall on each site is used to secure the routed traffic between VRF Red and VRF Green.

It is assumed (but not discussed in this document) that each Network service such as SVI, LISP, FW, OTV, etc… are fully redundant within each data center (usually Active/Standby mode per site).

We can consider 3 types of data flows:

• Intra-Subnet communication via L2 DCI: Bridged traffic destined for a remote host within the same broadcast domain uses the Layer 2 DCI connectivity (OTV). Note that, additional security services in transparent mode and/or encryption can be leveraged if needed to secure the Layer 2 DCI connectivity without impacting this scenario (not discussed in this post).

Intra-Subnet communication via L2 DCI

• Local routed traffic inter-Tenant intra-DC: Each  Firewall within a data center is first of all used to secure the local traffic between the two VRF’s. Firewalls are configured using dynamic routing.

Local Inter-VRF communication

• Routed traffic inter-Tenant inter-DC: Routed traffic destined for a remote machine that belongs to a different DMZ, uses both Firewalls from each site. The traffic is routed via a dedicated Layer 3 DCI connection (depicted in the following figure as Secured L3 DCI link), preventing asymmetric traffic. Indeed, when receiving a host route notification update from the remote LISP FHR, the local gateway records the concerned Endpoint with a /32 entry. As a result, the route to reach that host is now being more specific via the remote Firewall as the next hop. As a consequence, the relevant traffic inter-VRF will be transported across the dedicated Layer 3 DCI path, toward the remote Firewall.

Site to site Inter-VRF communication across both FWs

Configuration

The full configuration used for this testbed can be downloaded here:

DC-1 OTV

OTV has been tested in Multicast mode and in Adjacency server mode (Head End Replication) carrying the multicast group used to exchange the LISP EID notification across sites. HSRP filtering is initiated in the OTV edge device.

interface Overlay1
 otv join-interface Ethernet1/10
 otv control-group 239.1.1.100
 otv data-group 232.10.1.0/24
 otv extend-vlan 10, 20
 no shutdown
 otv-isis default
 vpn Overlay1
 redistribute filter route-map stop-HSRP
 otv site-identifier 0001.0001.0001

interface Ethernet1/9
 description OTV Internal interface
 switchport
 switchport mode trunk
 switchport trunk allowed vlan 10,20,210

interface Ethernet1/10
 description OTV Join Interface
 ip address 192.168.1.2/24
 ip ospf network point-to-point
 ip router ospf 1 area 0.0.0.0
 ip igmp version 3
 no shutdown

DC-1 Firewall

For the tests, ICMP (to check host reachability), and SSH (to validate that routed traffic remains symmetric from site to site) are permitted from any source to any destination.

DC-1 LISP IGP Assist

The minimum configuration required for LISP IGP Assist is quite simple.

The first step is to enable PIM to distribute the host route notification using a multicast group. All interfaces of interest must be configured with PIM sparse-mode. In this setup, a single RP is configured in DC-1 using the same loopback address used for the LISP locator.

interface loopback10
 description LISP Loopback
 ip address 10.10.10.10/32
 ip router ospf 1 area 0.0.0.0
 ip pim sparse-mode

ip pim rp-address 10.10.10.10 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8

Then, it is required to configure the route-map to advertise the host-routes toward the distant LISP FHR. The command described here is “global” for any subnets but more specific prefixes can be used. The route-map is redistributed via the OSPF process per VRF.

ip prefix-list HOST-ROUTES seq 5 permit 0.0.0.0/0 eq 32
route-map ADV-HOST-ROUTES deny 5
 match interface Null0
route-map ADV-HOST-ROUTES permit 10
 match ip address prefix-list HOST-ROUTES

router ospf 100
 router-id 1.1.1.2
 vrf GREEN
 redistribute lisp route-map ADV-HOST-ROUTES
 vrf RED
 redistribute lisp route-map ADV-HOST-ROUTES

Because it’s a multi-tenant environment, LISP IGP Assist is configured under each relevant VRF. The notification of host routes is achieved using a dedicated multicast group 239.1.1.n per Tenant. A dedicated LISP Instance is also required per Tenant.  The Subnet 20.1.1.0/24 used for the VRF GREEN is added into the “local” mapping database (actually the loop-back address 10.10.10.10 in DC-1). Subnet 10.1.1.0/24 used for the VRF RED is added into the same “local” mapping database. In this testbed, the notifications of EID (host-route) are performed using the multicast group 239.1.1.2.

vrf context GREEN
 ip pim ssm range 232.0.0.0/8
 ip lisp itr-etr
 lisp instance-id 2
 ip lisp locator-vrf default
 lisp dynamic-eid LISP_EXTENDED_SUBNET
 database-mapping 20.1.1.0/24 10.10.10.10 priority 1 weight 50
 map-notify-group 239.1.1.2
 no route-export away-dyn-eid

vrf context RED
 ip pim ssm range 232.0.0.0/8
 ip lisp itr-etr
 lisp instance-id 1
 ip lisp locator-vrf default
 lisp dynamic-eid LISP_EXTENDED_SUBNET
 database-mapping 10.1.1.0/24 10.10.10.10 priority 1 weight 50
 map-notify-group 239.1.1.1
 no route-export away-dyn-eid

All concerned interface VLAN must be configured to use the LISP dynamic EID process

interface Vlan10
 no shutdown
 vrf member RED
 lisp mobility LISP_EXTENDED_SUBNET
 lisp extended-subnet-mode
 ip address 10.1.1.251/24
 ip ospf passive-interface
 ip pim sparse-mode
 hsrp 10
 preempt
 priority 110
 ip 10.1.1.254

interface Vlan20
 no shutdown
 vrf member GREEN
 lisp mobility LISP_EXTENDED_SUBNET
 lisp extended-subnet-mode
 ip address 20.1.1.251/24
 ip ospf passive-interface
 ip pim sparse-mode
 hsrp 20
 preempt
 priority 110
 ip 20.1.1.254

The above configuration is the minimum required with IGP Assist to detect and inject the host-route dynamically toward the upward IGP routing table, offering more specific route for remote Endpoints.

DC-2

The same configuration is duplicated on each site with the relevant changes.

!
interface loopback20
  description LISP Loopback
  ip address 20.20.20.20/24
  ip router ospf 1 area 0.0.0.0
  ip pim sparse-mode
!
ip pim rp-address 10.10.10.10 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
!
ip prefix-list HOST-ROUTES seq 5 permit 0.0.0.0/0 eq 32
  match interface Null0
route-map ADV-HOST-ROUTES deny 5
route-map ADV-HOST-ROUTES permit 10
  match ip address prefix-list HOST-ROUTES
!
vrf context GREEN
  ip pim ssm range 232.0.0.0/8
  ip lisp itr-etr
  lisp instance-id 2
  ip lisp locator-vrf default
  lisp dynamic-eid LISP_EXTENDED_SUBNET
    map-notify-group 239.1.1.2
    database-mapping 20.1.1.0/24 20.20.20.20 priority 1 weight 50
    no route-export away-dyn-eid
!
vrf context RED
  ip pim ssm range 232.0.0.0/8
  ip lisp itr-etr
  lisp instance-id 1
  ip lisp locator-vrf default
  lisp dynamic-eid LISP_EXTENDED_SUBNET
    database-mapping 10.1.1.0/24 20.20.20.20 priority 1 weight 50
    map-notify-group 239.1.1.1
    no route-export away-dyn-eid
!
interface Vlan10
  no shutdown
  vrf member RED
  lisp mobility LISP_EXTENDED_SUBNET
  lisp extended-subnet-mode
  ip address 10.1.1.252/24
  ip ospf passive-interface
  ip pim sparse-mode
  hsrp 10
    preempt
    priority 120
    ip 20.1.1.254
    ip 10.1.1.254
!
interface Vlan20
  no shutdown
  vrf member GREEN
  lisp mobility LISP_EXTENDED_SUBNET
  lisp extended-subnet-mode
  ip address 20.1.1.252/24
  ip ospf passive-interface
  ip pim sparse-mode
  hsrp 20
    preempt
    priority 120
  router-id 1.1.1.2
!
router ospf 100
  vrf GREEN
    redistribute lisp route-map ADV-HOST-ROUTES
  vrf RED
    redistribute lisp route-map ADV-HOST-ROUTES
no system default switchport shutdown

Verification

Check the dynamic-eid database on each site

Note that the output is reduced to make it readable

The EID reachability information is given for each VRF:

• R1 (10.1.1.1)
• R2 (10.1.1.2)
• G1 (20.1.1.1)
• G2 (20.1.1.2)

Data Center 1 (Left)

LISP-IGP_DC-1# sho lisp dynamic-eid summary vrf RED
LISP Dynamic EID Summary for VRF “RED”
LISP_EXTENDED_SUBNET      10.1.1.1    Vlan10 
!
LISP-IGP_DC-1# sho lisp dynamic-eid summary vrf GREEN
LISP Dynamic EID Summary for VRF “GREEN”
LISP_EXTENDED_SUBNET      20.1.1.1    Vlan20

Data Center 2 (Right)

LISP-IGP_DC-2# sho lisp dynamic-eid summary vrf RED
LISP Dynamic EID Summary for VRF "RED"
LISP_EXTENDED_ SUBNET      10.1.1.2    Vlan10
!
LISP-IGP_DC-2# sho lisp dynamic-eid summary vrf GREEN
LISP Dynamic EID Summary for VRF "GREEN"
LISP_EXTENDED_SUBNET       20.1.1.2    Vlan20

Check the routing table (just keeping the relevant EID). From VRF RED, R1 is (10.1.1.1/32) locally attached to VLAN  10. R2 (10.1.1.2) , G1 (20.1.1.1) & G2 (20.1.1.2) are reachable via the next hop router (100.1.1.254).

Data Center 1 (Left)

LISP-IGP_DC-1# sho ip route vrf RED
IP Route Table for VRF "RED"
...
10.1.1.1/32, ubest/mbest: 1/0, attached 
 *via 10.1.1.1, Vlan10, [240/0], 00:31:42, lisp, dyn-eid
10.1.1.2/32, ubest/mbest: 1/0   
 *via 100.1.1.254, Vlan100, [110/1], 00:32:49, ospf-100, type-2
...
20.1.1.1/32, ubest/mbest: 1/0
 *via 100.1.1.254, Vlan100, [110/1], 00:31:40, ospf-100, type-2
20.1.1.2/32, ubest/mbest: 1/0
 *via 100.1.1.254, Vlan100, [110/1], 00:32:47, ospf-100, type-2
...

Firewall in DC-1

FW-DC-1# sho route
..//..
O E2 10.1.1.2 255.255.255.255 [110/1] via 30.1.1.2, 60:53:29, Inter-DMZ  <== via L3 DCI toward DC-2
O E2 10.1.1.1 255.255.255.255 [110/1] via 100.1.1.10, 60:52:56, RED          <== Local routed traffic
..//..

Firewall in DC-2

FW-DC-1# sho route
..//..
O E2     10.1.1.1 255.255.255.255 [110/1] via 30.1.1.1, 2d12h, Inter-DMZ  <== via L3 DCI toward DC
O E2     10.1.1.2 255.255.255.255 [110/1] via 102.1.1.20, 2d12h, RED     <== Local routed traffic
..//..

Data Center 1 (Left)

From VRF GREEN, G1 (20.1.1.1) is locally attached to VLAN 20. R1 (10.1.1.1/32), R2 (10.1.1.2) & G2 (20.1.1.2) are reachable via the next hop router (200.1.1.254).

LISP-IGP_DC-1# # sho ip route vrf green
IP Route Table for VRF "GREEN"
...
10.1.1.1/32, ubest/mbest: 1/0
    *via 200.1.1.254, Vlan200, [110/1], 01:04:23, ospf-100, type-2
10.1.1.2/32, ubest/mbest: 1/0
    *via 200.1.1.254, Vlan200, [110/1], 01:05:30, ospf-100, type-2
...
20.1.1.1/32, ubest/mbest: 1/0, attached
    *via 20.1.1.1, Vlan20, [240/0], 01:04:22, lisp, dyn-eid
20.1.1.2/32, ubest/mbest: 1/0
    *via 200.1.1.254, Vlan200, [110/1], 01:05:28, ospf-100, type-2

Let’s perform a Live migration with R1  moving to DC-2 and R2  moving to DC-1. As soon as the migration is complete, the dynamic EID mapping database is updated accordingly.

Data Center 1 (Left)

LISP-IGP_DC-1# sho lisp dynamic-eid summary vrf red
LISP Dynamic EID Summary for VRF "RED"
LISP_EXTENDED_SUBNET      10.1.1.2    Vlan10 

Data Center 2 (Right)

LISP-IGP_DC-2# sho lisp dyn summary vrf red
LISP Dynamic EID Summary for VRF "RED"
LISP_EXTENDED_ SUBNET      10.1.1.1    Vlan10

Ping and SSH sessions inter-DC between the 2 VRF continue to work with a sub-second interruption.

Configuration (continue)

As previously mentioned, the LISP IGP Assist set-up given above is the minimum configuration required to notify the EID dynamically using the multicast protocol across the L2 DCI links and redistribute the host route into the IGP routing table. As is, it already works like a charm, as long as the multicast group can reach the remote LISP mapping-database using the Layer 2 DCI extension.

It is optionally possible, but recommended, to add an alternative mechanism to notify the EID via a routing protocol established with a LISP Map-Server, in case the primary mechanism fails. For any reasons, if the Multicast transport or L2 extension stops working, the map-server will notify the remote mapping database about the new EID using the routing protocol. Actually, this is the method used for IGP Assist in ASM mode (Across Subnet Mode without any L2 extension), when no extended VLAN exists across data centers to carry the multicast accordingly for each VRF.

The Map-Resolver is responsible to receive map requests from remote ingress Tunnel  Routers (iTRs) to retrieve the mapping between an Endpoint Identifier and its current location (Locator). For the purpose of this specific scenario, there is no inbound path optimization, nor either eTR, iTR or LISP encapsulation. Hence, only the Map-server function is relevant for this solution as an backup mechanism to trigger the Endpoint notification’s. In the context of IGP Assist, the Map-Server system is responsible to exchange EID mapping between all other LISP devices. The M-DB can cohabit on the same device as the LISP FHR multiple mapping databases can be distributed over dedicated hosts. For the purpose of our test, the function of MS runs on the same switch device.

To configure the LISP map-server, follow the next global configuration on each LISP IGP router.

Data Center 1 (Left)

ip lisp itr-etr
ip lisp map-server
lisp site DATA_CENTER
  eid-prefix 10.1.0.0/16 instance-id 1 accept-more-specifics
  eid-prefix 20.1.0.0/16 instance-id 2 accept-more-specifics
  authentication-key 3 9125d59c18a9b015
ip lisp etr map-server 10.10.10.10 key 3 9125d59c18a9b015
ip lisp etr map-server 20.20.20.20 key 3 9125d59c18a9b015

Data Center 2 (Right)

ip lisp itr-etr
ip lisp map-server
lisp site DATA_CENTER
  eid-prefix 10.1.0.0/16 instance-id 1 accept-more-specifics
  eid-prefix 20.1.0.0/16 instance-id 2 accept-more-specifics
  authentication-key 3 9125d59c18a9b015
ip lisp etr map-server 10.10.10.10 key 3 9125d59c18a9b015
ip lisp etr map-server 20.20.20.20 key 3 9125d59c18a9b015

Additional document

http://lisp.cisco.com/docs/LISP_IGP_ASSIST_WHITEPAPER.pdf

Dear Network and DCI Experts !

While this post is a little bit out of the DCI focus, and assuming many of you already know Q-in-Q, the question is, are you yet familiar with Q-in-VNI? For those who are not, I think this topic is a good opportunity to bring Q-in-VNI deployment with VXLAN EVPN for intra- and inter-VXLAN-based Fabrics and understand its added value and how one or multiple Client VLANs from the double encapsulation can be selected for further actions.

Although it’s not an unavoidable rule per-se, some readers already using Dot1Q tunneling may not necessarily fit into the following use-case. Nonetheless, I think it’s safe to say that most of Q-in-Q deployment have been used by Hosting Provider for co-location requirements for multiple Clients, hence, the choice of the Hosting Provider use-case elaborated in this article.

Nonetheless, what is elaborated in the following sections can address as well other requirement, to list just one more. For example, after the acquisition of one or multiple enterprises or other business organisations, merging multiple DC together into a single infrastructure, in order to reduce OPEX. This is another form of multi-tenancy that requires deploying the same level of Layer 2 and Layer 3 segmentation.

For many years now, Hosting Services have physically sheltered thousands of independent clients’ infrastructures within the Provider’s Data Centers. The Hosting Service Provider is responsible for supporting each and every Client’s data network in its shared network infrastructure. This must be achieved without changing any Tenant’s Layer 2 or Layer 3 parameter, and must also be done as quickly as possible. For many years, the co-location business for independent Tenants has been accomplished by using a dedicated physical Provider network for each Client’s infrastructure, which has been costly in terms of network equipment, operational deployment and maintenance, rack space, power consumption, and this method has also been rigid, with limited scalability for growth.

With the evolution of this business and virtualization, there is a strong need to offer additional services at lower cost with more agility for customers, and the requirements for this solution can be summarized as follows:

• To seamlessly append any new Client network to the shared infrastructure owned by the Hosting Service Provider with the same requirement mentioned previously in terms of Layer 2 and Layer 3 segmentation.
• To support multiple Clients sharing the same Top-of-Rack switches.
• The above implies that the shared Provider’s infrastructure must be able to support duplicate Layer 2 VLAN identifiers across all the Tenants, including reuse of the same Layer 3 address space by all of them.
• To provide the flexibility of transparently spreading each Tenant‘s infrastructure throughout the Hosting Provider’s organization.
• To allow for the seamless growth and evolution of each Tenant’s network, for the Client business as well as for the Service Provider production.
• To offer new Service Clouds for each Client.

Figure 1: hosting service provider with service cloud

Figure 1 shows a logical view of different Client infrastructures spread across the Provider’s Network infrastructure with an extension of their respective network segments between local and geographically dispersed Pods or data centers. In addition to traditionally maintaining the Client network segmentation from end-to-end, each Tenant must be able to access both the private and global Service Cloud.

Essentially, besides managing their Tenant’s data network, Hosting Providers should be able to develop their business by proposing connectivity to a new and enhanced model of Service Cloud, to each Tenant. The Service Cloud provides new and exciting service tools for the hosted Clients such as a broad range of XaaS as well as “Network and Security as a Service” (NSaaS) for the Clients’ applications. However, the key technical challenge for the Hosting Provider is to be able to select one or multiple “private” segments from any isolated Client infrastructure in order to provide access to a dedicated private or shared public Service Cloud.

Note: “Network and Security as a Service”: This Cloud Service provides Application Security, Deep Inspection, Optimization, Offloading, and Analytics Services, just to list a few, for each and every client‘s set of multi-tier applications.

Hosting Service Requirements & Solutions

The shared Data Center Network infrastructure from the Hosting Provider must support VLAN overlapping as well as duplicated Layer 3 networks for any Client organization.

To address this requirement, as done for many years, the historical “double VLAN tag” encapsulation method is leveraged in conjunction with VXLAN EVPN as a Layer 3 transport. This Dot1Q tunnel encapsulation for the Layer 2 segmentation is named Q-in-VNI. Besides the Layer 2 overlay network transport, VXLAN EVPN is used for the multi-tenancy Layer 3 segmentation.

The ingress interface connecting the Client network is configured in access mode with a Backbone VLAN (B-VLAN) that encapsulates the original Client VLAN (C-VLAN). B-VLANs belong to the Provider network’s resources and are consumed accordingly. Q-in-VNI maps each Backbone VLAN to a L2 VNI transport which contains all the Client VLAN’s (C-VLAN) carried with the inner Dot1Q. As a consequence, the original C-VLAN’s can be retrieved at the egress VTEP and this transparently provides the extension of all Client Networks across different Top-of-Racks.

Figure 2: Q-in-VNI to transport client-vlans from end-to-end

As a consequence, the VXLAN Fabric is agnostic regarding the original Client VLAN’s. There is no need to configure the “private” C-VLAN’s as such. Only the B-VLANs mapped to a VXLAN tunnel network need to be created as shown in the sample configuration below.

Figure 3: Q-in-VNI sample configuration with vlan 1001 as the dot1q-tunnel vlan

VXLAN EVPN Fabric Multipod

Because all the “private” VLAN’s for each and every Client must be carried from end-to-end throughout the shared network connectivity for multiple sites’ deployment, one option is to maintain the double tag encapsulation across different and distant locations over the Layer 3 network, without any hand-off between the VLAN’s.

Consequently, the first choice for network transport relies on VXLAN EVPN Multipod discussed in post 32

The VXLAN EVPN Multipod architecture pillars can be expressed as follows:

• Encapsulation:
  • Maintains the Tenant encapsulation from Pod-to-Pod and from site-to-site.
  • No requirement for VLAN hand-off at any site boundary, which will break the Dot1Q tunneling (Figure 4).
• Sturdiness
  • Validated for geographically dispersed Pods.
  • Offers a solid Layer 3-based underlay Fabric, including inter-site transport.
  • Contains the failure domain by reducing the amount of flooding frames across the whole Fabric (no Flood & Learn, ARP Suppress).
  • Efficient Endpoint learning with MP-BGP EVPN – Client host’s (MAC) reachability information is discovered and distributed among all Leaf Nodes of interest by the Control Plane.
  • Maintains Layer 2 and Layer 3 segmentation for each Tenant from end-to-end.
• Flexibility
  • Multiple Client infrastructures share the same Top-of-Rack switch.
  • Mix of Q-in-Q transport, Dot1Q trunk and Access port within the same ToR switch. From each Top-of-Rack switch on to which client infrastructures are locally attached, multiple interfaces per Tenant organization can use different encapsulation types (Q-in-Q, Dot1Q) and modes (Trunk, Access, Routed).
  • Per Port VLAN translation for native Dot1Q client-VLAN.
  • Independent Client Layer 2 and Layer 3 network infrastructures (Physical, Virtual, Hybrid).
• Scalability
  • Offers very high scalability, which is required by Service Providers.
  • Reuses duplicate VLAN IDs across all Tenants throughout the whole Fabric.
  • Transit inter-site devices are pure Layer 3 routers that do not require any Layer 2 encapsulation capabilities. Thus they do not dictate the amount of L2 encapsulated frames and tags that can be stretched across the remote sites (note, MTU must be increased accordingly).
  • Transit inter-site devices can also be leveraged to extend or support additional Tenants (Bud node support).

Figure 4: Q-in-VNI with VXLAN EVPN Multipod geographically dispersed

Figure 4 illustrates the requirement of Layer 2 segmentation and VLAN ID overlapping between Clients, across different Leaf nodes spread over multiple locations. The transit transport between Pods being pure Layer 3, the double encapsulation is performed at the edge of each Leaf Node.

Service Cloud Integration

As mentioned above, it is critical that the Hosting Provider offers other Service Clouds to its Clients. Although each Client network infrastructure keeps fully isolated, access to the Service Cloud for each Tenant can be achieved using two different approaches.

Selective Client Service C-VLAN’s from Ingress Access Interface

The first option is an elementary method to provide external access so that each individual Client infrastructure can benefit from the Provider Services. As explained previously, in order to maintain the scalability and segmentation of each and every Client network, the key transport relies on Q-in-VNI (a Per-Client Q-in-Q encapsulation mapped to a L2 VNI). As a result, due to the double tagging induced with the Dot1Q tunneling, as of today, Client VLAN’s cannot be natively selected for any further treatment such as routing, bridging or for security purposes. As described in Figure 2 and Figure 4, Client-VLAN’s are fully isolated from the rest of the Provider’s infrastructure, which is, above all, the foremost expectation from each Client. As a consequence, for that particular requirement, each Tenant’s infrastructure consumes at least one physical interface per Top-of-Rack in order to be expanded to other racks or other Pods using Q-in-VNI.

Additionally, to offer access to the Service Cloud, one or multiple C-VLANs for each individual private Client network must be selected from a different source Interface, where the latter is locally connected. Consequently, a second physical access interface is allocated on the Top-of-Rack for one or more selective C-VLANs to be routed outside the Client’s organization.

Figure 5: Separated Interface for Hosting transport and Services

Figure 5 depicts an extreme scenario in which, firstly each Client infrastructure uses the full range of isolated Dot1Q VLANs, and secondly Client Blue and Client Orange have elected the same VLAN 10 to access a Service Cloud

Note: Let me call these Client-VLANs which are used to access the Service Cloud, “Client Service C-VLANs” or “public C-VLANs”.

To keep it simple with this example, for each Client:

• A first L2 VNI is initiated to transport the double-tagged frames (Q-in-VNI), to carry all the “private” L2 network segments of a particular Client within the VLAN Fabric.
• Another L2 VNI is leveraged to transport the “Client Service VLAN” (public network) used to access one of the Provider L3 services.
• Finally, a L3 VNI is used to maintain the Layer 3 segmentation for that particular Client among the other Tenants.

Note that it is also possible to transport multiple “Client Service VLANs” with their respective L2 VNIs (1:1) for a specific Client infrastructure over the same ingress edge trunk interface of the concerned ToR. It is also critical to maintain the Layer 3 segmentation for each Client extended within the VXLAN Fabric (multiple L2 VNIs with multiple L3 VNIs per Client infrastructure). However, it is important to mention that for Clients to access the Provider’s Service Cloud, each Tenant will consume one Provider VLAN for their L2 segment and another for their L3 segment.

Consequently, this must be used with caution with regard to the consumption of the Provider’s VLAN resources.

VLAN Overlapping for “private” Client VLAN’s

To address the VLAN overlapping such that the 4k C-VLAN’s are spread across the Provider infrastructure, the Dot1Q VLAN tunnel used as the Backbone VLAN is unique (per ToR) for each Client (1001, 1002, 1003, etc..). Each B-VLAN is mapped to a dedicated and unique Layer 2 VNI which also carries within the VXLAN header the original Dot1Q tags of each and every Client segment.

Note: VLAN ID are per Leaf node locally significant, consequently, it is possible to reuse the same VLAN ID for a different Tenant as long as they are locally attached to distinct Top-of-Rack switch and these B-VLAN are mapped to a different and unique VN ID.

As discussed previously, for the Q-in-Q transport, the VXLAN Fabric is not aware of the existence of any C-VLAN per-se, hence there is no C-VLAN to configure onto the Fabric, as illustrated in Figure 6, but the B-VLANs need to be configured. The original “private” C-VLAN’s are kept fully “hidden” from the rest of the network infrastructure.

However, for “public” C-VLAN’s aiming to be routed to the Service Cloud, these particular VLAN’s must be created like any traditional access VLAN as they are not part of the Q-in-Q tunneling encapsulation. As illustrated in the configuration sample below (Figure 6), Client VLAN’s 10, 20 and 30 are created to be used respectively for further forwarding actions controlled by the Fabric itself.

Figure 6: VLAN configuration with mapping to L2 VNI as well as the L3 VNI

“Client Service VLAN’s” can now be routed like any other Native Dot1Q VLAN mapped to an L2 VNI. All VXLAN Layer 2 and Layer 3 services are available for those Client segments.

• Specific “public” C-VLAN’s can be routed to Layer 3 Service Clouds.
• The Layer 3 Anycast Gateway feature can hence be leveraged to each Client for that specific “public” C-VLAN.
• Layer 3 segmentation between Tenants is addressed using the traditional VRF-lite transported over a dedicated L3 VNI as shown in the configuration sample in Figure 4.

VLAN Overlapping for “public” Client VLAN’s

To address the VLAN Overlapping for the “Client Service VLAN’s,” the service, “Per-port VLAN mapping,” is leveraged at the ingress port level to offer a one-to-one VLAN translation feature with a unique Provider VLAN.

Figure 7: Interface VLAN mapping per port VLAN Translation for native C-VLAN

As the result, each “public” C-VLAN (original or translated) can be mapped to its respective Layer 2 VNI, eliminating the risk of any VLAN IDs overlapping with another Client’s infrastructure. Since it’s a one-to-one translation between the original C-VLAN and a unique Provider VLAN, although this feature is very helpful, it must be considered only for a limited number of C-VLAN’s (4k VLAN’s per shared Bridge Domain). In the context of Service Providers in general, with a hundred or a thousand clients, this per-port VLAN mapping solution alone may not be scalable enough for a large number of originated Client VLAN’s. Hence, the need for the parallel transport using Q-in-VNI, leveraged to all the other isolated “private” C-VLAN’s that do not require any Services from the Hosting Provider, as discussed previously. 

Consequently, from each set of “private” C-VLAN’s, each Client must provide at least one C-VLAN ID for accessing the Hosting Provider network. The “Client Service VLAN’s” can then be routed within the Fabric and can, above all, benefit from the dedicated or shared Layer 3 Service Cloud.

Figure 8: Tenant connections to Service-Cloud

Client Access to the Provider Service Cloud

To route the “Client Service VLANs” to a Service Cloud, the corresponding VNIs are extended and terminated at a pair of Border Leaf Nodes that provide direct connectivity with the Service Cloud gateway. On this physical interface connecting the Cloud network, a logical sub-interface is created for each Tenant as depicted in Figure 8.

Each sub-interface is configured with a unique Dot1Q encapsulation established with its peer network. The Dot1Q tag used in this configuration is not directly related to any encapsulated tag from the Access Interfaces (Client VLAN attachment). However, the same Dot1Q tagging is used on the reverse side (Provider Service Cloud Gateway) in order to provide the L3 segmentation peering between each Client network (Tenant) toward their respective private or shared Service Cloud (Figure 9 & 10)

VXLAN Border Leaf Sub-interfaces

Accordingly, the first solution described above requires a second physical interface per Client to be configured for selective C-VLAN’s to be routed. As a result, this solution doubles the number of physical access interfaces consumed from the Provider infrastructure, which may have an impact on the operational expenses. In addition, it is important to remember that from a scalability point-of-view, additional Provider VLAN’s to route these Client VLAN’s outside their private infrastructure are used, which are not unlimited in number.

Selective Client Service C-VLANs using Ethernet Flow Point (EFP)

For keeping the consumption of access interfaces for locally attached Client infrastructures as low as possible, a more sophisticated method is utilized in conjunction with the Cisco ASR9000 or the Cisco NCS5000, which are deployed as the selective access gateway for accessing the Service Cloud. This selection of the concerned Client Service VLAN’s is centralized and is performed at a single interface (inbound interface of the cloud service provider router) and hence, offering a much easier access for configurations. Thanks to the Ethernet Virtual Connection (EVC) and Ethernet Flow Point (EFP) features. Obviously, the same WAN Edge Router used to access the Provider cloud or for WAN connectivity in general, can be used for this selection purpose. It doesn’t have to be dedicated.

Note: Already quickly described a while back in post 10

10 – Ethernet Virtual Connection (EVC)

An Ethernet Virtual Connection (EVC) is a Cisco carrier Ethernet equipment function dedicated to Service Providers and large enterprises. It provides a very fine granularity to select and treat the inbound workflows known as service instances, under the same or different ports, based on flexible frame matching.

 The EFP is a Layer 2 logical sub-interface used to classify traffic under a physical interface or a bundle of interfaces. It represents a logical demarcation point of an Ethernet Virtual Connection (EVC) on a particular interface. It exists as a Flow Point on each interface, through which the EVC passes.

With the EFP, it is therefore possible to perform a variety of operations on ingress traffic flows, such as routing, bridging or tunneling the traffic through many ways by using a mixture of VLAN IDs, single or double (Q-in-Q) encapsulation, and Ether-types.

Figure 11: Ethernet-F0low-Point (EFP) for Selective B-VLAN, C-VLAN toward L3 Service (VRF)

In our use-case, the EFP serves to identify the double tag, and to associate the Backbone VLAN (outer VLAN tag) identifying the Client-VLAN (outer VLAN tag) of choice for a particular Tenant, in order to route the later identified frames into a new segmented Layer 3 network.

Figure 11 represents the physical ingress Layer 2 interface of the ASR9000 or NCS5000 receiving the double tag frames from one of the VXLAN Border Leaf Nodes. Each double tag is identified by the Backbone VLAN and one Client VLAN. The C-VLAN is selected and added to a dedicated Bridge Domain for enhanced forwarding actions. In addition to other tasks such as bridging back the original Layer 2 to a new Layer 2 segment, to address our particular need, a routed interface can be added to the relevant Bridge Domain, offering the L3 VPN network to that particular Tenant for accessing their Service Cloud (Tenant 1 & 3). However, if the requirement is a simple Layer 3 service, the routed interface can be directly tied to the sub-interface (Tenant 2).

Figure 12: Client Infrastructure access simplified

From the point of attachment of the Client infrastructure to the Provider network, the relevant Access Interfaces from the Top-of-Rack switch are simplified to a single logical ingress approach as shown in Figure 12. The selection of the Client-VLAN’s to be routed is completed outside the VXLAN domain at the Service Cloud gateway that connects to the Border Leaf Nodes, which makes it a centralized process (Figure 14). This significantly simplifies the physical configuration from each Client network’s end.

The function of the Ethernet Flow Point offers flexible VLAN matching. When the traffic coming from the Fabric hits the Layer 2 interface transport of the router, the TCAM for that port is used to find out which particular sub-interface the filters match with [outer-VLAN, inner-VLAN], in order to bind, for example, the selected ingress traffic to a dedicated L3 VPN network.

Two approaches are possible. The first one is quite simple, where the selected C-VLAN is directly routed to the VRF of interest.

Figure 13: Connecting a private C-Client to a private Layer 3 Service VRF or L3 VPN ASR9K and NCS5K

Figure 14: EFT used for association of Double TAG with selection of C-VLAN bound to L3 Services (NCS5k)

Figure 15: EFT Sample Configuration for Routing Service

Figure 14 depicts the logical view of the ingress interface of the gateway (ASR9000 or NCS5000) connecting each selected C-VLAN directly to a Layer 3 network. As a result, each Client can connect their private network infrastructure to one of the Layer 3 Service Clouds offered by the Provider. VRF and L3 VPN are supported by both the platforms, ASR9000 and NCS5000.

The second option uses the Bridge Domain to bind the selected network, allowing additional actions for the same data packet.

Figure 16: Connecting a Private C-VLAN to a private L3 Service using a Bridge Domain (BD)

Figure 17: EFP for association of double TAG with selection of C-VLAN bound to a Bridge Domain and/or L3 service (ASR9k)

Figure 17 depicts the logical view of the ingress interface of the gateway (ASR9000) connecting each selected C-VLAN with a Bridge Domain for access to a Layer 3 network. As a result, each Client can connect their private network infrastructure to one of the Layer 3 Service Clouds offered by the Provider.

The key added value with the Bridge Domain attachment is that additional actions can be applied to any Client’s VLAN [B-VLAN, C-VLAN], such as Layer 2 VPN services (VPLS, Dot1Q, Q-in-Q, PBB-EVPN).

Figure18: Layer 2 to BVI Sample Configuration (ASR9000)

Beyond the Layer 3 Services

As mentioned previously, the added value of using the Bridge Domain is the ability to extend the actions beyond the L3 and the L3 VPN with a L2 or L2 VPN transport.

For example, the same double tag frame can be re-encapsulated using another overlay transport such as PBB-EVPN (Figure 20) or Hierarchical VPLS deployment. The filter can be applied for “any” inner-VLAN tag as a global match for further action. As a result, the same Client infrastructure can be maintained in isolation across multiple sites using a hierarchical DCI solution, without impacting the Client VLAN’s. Each set of Client VLAN’s can be directly bound to another double tag encapsulation method such as PBB-EVPN or Hierarchical VPLS while still supporting a very high scalability. This can become a nice alternative to a VXLAN Multipod by keeping each network infrastructure fully independent from a VXLAN data plane and control plane point of view.

Figure19: Logical View Extending the C-VLAN to L2-VPN and L3-VPN via the Bridge Domain

Figure 20: Logical View of Tenant infrastructure extended across PBB-EVPN per-Tenant Service-Cloud

Network and Security as a Service Cloud

In addition to private access to a particular Service Cloud, it is also possible to offer additional “Network and Security as a Service (NSaaS)” for each Client’s host of multi-tier applications.

This feature offers a very granular and flexible solution. For example, as shown in Figure 21, multiple C-VLAN’s supporting a multi-tier application (e.g. VLAN 10 = WEB, VLAN 20 = APP, VLAN 30 = DB) are each bound to a specific Bridge Domain. Each concerned VLAN is sent to a particular Network Service node running in routed mode or transparent mode, to be treated afterward based on the application requirements (Firewalling, Load balancing, IPS, SSL off loader, etc.).

Figure 21: Providing Network and Security Services From the Provider Resources

Conclusion

As of today, Q-in-VNI support with VXLAN EVPN Fabrics is one of the best and easiest method to transport multiple isolated Client infrastructures within a data center across the same physical infrastructure. In association with the Ethernet Virtual Circuit and the Ethernet Flow Point features of the Cisco Service Provider platforms (ASR9000 & NCS5000), each and every Client infrastructure can be selectively extended to the Service Clouds offered by the Hosting Providers.

With the binding of those private VLAN’s to a Bridge Domain, each Client infrastructure can be extended using Layer 2 VPN and/or Layer 3 VPN, maintaining the same level of segmentation across multiple sites. Last but not least, the EFP feature can furthermore be leveraged to offer additional Network and Security as a Service by the Hosting Provider for any of the Client’s applications.

Since 2.0, Multipod for ACI enables provisioning a more fault tolerant fabric comprised of multiple pods with isolated control plane protocols. Also, multipod provides more flexibility with regard to the full mesh cabling between leaf and spine switches.  When leaf switches are spread across different floors or different buildings, multipod enables provisioning multiple pods per floor or building and providing connectivity between pods through spine switches.

A new White Paper on ACI Multipod is now available

http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-737855.html?cachemode=refresh

VXLAN Multipod geographically dispersed

VXLAN Multipod Overview

Note: Since I wrote this article many years ago, VXLAN EVPN and the ASIC have evolved. You should consider this article obsolete and read the post 37 (http://yves-louis.com/DCI/?p=1588) which explains why this solution is not recommended anymore. The question now is why this post is still here? The reason is that I explained some packet walks that the reader could find helpful to understand the differences between VXLAN EVPN Multi-pod and VXLAN EVPN Multi-site.

This article focuses on the single VXLAN Multipod Fabric stretched across multiple locations as mentioned in the previous post 31 through the 1st option.

We have been recently working, with my friends Patrice and Max, during a couple of months, building an efficient and resilient solution based on VXLAN Multipod fabric stretched across two sites. The whole technical white paper is now available for deeper technical details including insertion of Firewalls, Multi-tenancy VXLAN routing and some additional tuning, and it can be accessible from here

One of the key use-case for that scenario is for an enterprise to select VXLAN EVPN as the technology of choice for building multiple greenfield Fabric PoDs. It becomes therefore logical to extend the VXLAN overlay between distant PoDs that are managed and operated as a single administrative domain. Its deployment is just a continuation of the work performed to roll out the fabric PoDs, simplifying the provisioning of end-to-end Layer 2 and Layer 3 connectivity.

Technically speaking, and thanks to the flexibility of VXLAN, we could deploy the overlay network on top of almost any forms of Layer 3 architecture within a datacenter (CLOS, Multi-layered, Flat, Hub&Spoke, Ring, etc.), as long as VTEP to VTEP communication is  always available, and obviously a CLOS (Spine & Leaf model) architecture is being the most efficient.

• [Q] However, can we afford to stretch the VXLAN fabric across distant locations as a single fabric, without taking into consideration the risks of loosing the whole resources in case of a failure?
• [A] The answer mainly relies on the stability and quality of the fiber links established between the distant sites. In that context it is interesting to remind that most of DWDM managed services offer an SLA equal to or lower than 99.7%. Another important element to take into reflection is how much the control plane can be independent from site to site, reducing the domino effect caused by a breakdown in one PoD.

Anyhow, we need to understand how to protect the whole end-to-end Multipod fabric to avoid propagating any disastrous situation in case of a major failure.

Different models for extending multiple PoD across a single logical VXLAN fabric are available. Each option relies on how the Data plane and Routing Control Plane are stretched or distributed across the different sites to separate the routing functions as well as the placement of the transit layer 3 nodes used to interconnect the different PoDs.

From a physical approach, the transit node connecting the WAN can be initiated using dedicated Layer 3 devices, or leveraged from the existing leaf devices or from the spine layer. Diverse approaches exist to disperse the VXLAN EVPN PoDs across long distances.

We can’t say there is one single approach to geographically dispersed the fabric PoDs among different location. The preferred choice should be taken according to the physical infrastructure including distances and physical transport technology, the enterprise business and service level agreement as well as the application criticality levels with the Pros & Cons for each design.

VXLAN EVPN Multipod Stretched Fabric

What we should keep into consideration is the following (non exhaustive list, though):

• For VXLAN Multipod purposes, it is important that the fabric learns the endpoints from both sites using a control plane instead of the basic flood & learn.
• The underlay network intra-fabric and inter-fabrics is a pure Layer 3  network used to handle the overlay tunnel transparently.
• The connectivity between sites is initiated from a pair of transite layer 3 nodes. The transit device used to interconnect the distant fabrics is a pure layer 3 routing device.
• This transit layer 3 edge function can be also initiated from a pair of leaf nodes used to attach endpoints or a pair of border spine nodes used for layer 3 purposes only.
• Actually, the number of transit devices used for the layer 3 interconnection is not limited to a pair of devices.
• The VXLAN Multipod solution recommends to interconnect the distant fabrics using Direct Fiber or DWDM (Metropolitan-area distances). However, technically nothing prevents to use inter-continental distances (layer 3) as VXLAN DP and EVPN CP are not sensitive to any latency concerns (although this is not recommended), but the application is.
• For geographically dispersed PoDs, it is recommended that the control planes (underlay and overlay) are deployed in a more structured fashion in order to keep them as independent as possible from each location.
• Redundant and resilient MAN/WAN access with fast convergence is a critical requirement, thus network services such as DWDM in protected mode and with remote port shutdown, and BFD should be enabled.
• Storm Control on the geographical links is strongly recommended to reduce the fate sharing between PoDs.
• BPDU Guard must be enabled by default on the edge interfaces to protect against Layer 2 backdoor scenarios.

Slight Reminder defining the VXLAN Tunnel Endpoint: 

VXLAN uses VTEP (VXLAN tunnel endpoint) services to map a particular dot1Q Layer 2 frames (VLAN ID) to a VXLAN segment tunnel (VNI). Each VTEP is connected on one side to the classic Ethernet segment where endpoints are deployed, and on the other side to the underlay Layer 3 network (Fabric) to establish the VXLAN tunnels with other remote VTEPs.

The VTEP performs the following two tasks:

• Receives traffic from locally connected endpoints and encapsulates it into VXLAN packets destined for remote VTEP nodes
• Receives VXLAN traffic originating from remote VTEP nodes, decapsulates it, and forwards it to locally connected endpoints

The VTEP dynamically learns the destination information for VXLAN encapsulated traffic for remote endpoints connected to the fabric by using the MP-BGP EVPN control protocol.

With information received from the MP-BGP EVPN control plane used to build information in the local forwarding tables, the ingress VTEP, where the source endpoint is attached to, encapsulates the original Ethernet traffic with a VXLAN header and sends it over a Layer 3 network toward the egress VTEP, where the destined endpoint sits. The latter then de-encapsulates the layer 3 packet to present the original Layer 2 frame to its final destination endpoint.

From a data-plane perspective, the VXLAN Multipod system behaves as a single VXLAN fabric network. VXLAN encapsulation is performed end-to-end across PoDs and from leaf nodes. The spines and the transit leaf switches perform only Layer 3 routing of VXLAN encapsulated frames to help ensure proper delivery to the destination VTEP.

VXLAN VTEP-to-VTEP VXLAN Encapsulation

Multiple Designs for Multipod deployments

PoDs are usually connected with point-to-point fiber links when they are deployed within the same physical location. The interconnection of PoDs dispersed across different locations uses direct dark-fiber connections (short distances) or DWDM to extend Layer 2 and Layer 3 connectivity end-to-end across locations (Metro to long distances).

Somehow, there is no real constraint with the physical deployment of VXLAN per se as it’s an overlay virtual network running on top of a Layer 3 underlay, as long as the egress VTEPs are all reachable.

As a result, different topologies can be deployed (or even coexist) for the VXLAN Multipod design.

Without providing an exhaustive list of all possible designs, the 1^st main option is to attach dedicated transit layer 3 nodes toward the Spine devices. They are usually called Transit leaf nodes as they often sit at the same Leaf layer and are also often used as computing Leaf nodes (where endpoints are locally attached), or as service leaf nodes (where firewall, security and network service are locally attache), beside the extension of Layer 3 underlay connectivity toward the remote sites.

VXLAN Multipod Design Interconnecting Leaf Nodes

Hence, nothing prevents to interconnect distant PoDs through the spine layer devices, as no additional functions nor VTEP are mandated from the transit spine other than the capability to route VXLAN traffic between leaf switches deployed in separate PoDs. A layer 3 core layer can be leveraged for interconnecting remote PoDs.

VXLAN Multipod Design Interconnecting Spine Nodes

The VXLAN Multipod design is usually positioned to interconnect data center fabrics that are located at metropolitan-area distances, even though technically, there is no distance limit. Nevertheless, it is crucial that the quality of the DWDM links is optimal. Consequently, it is necessary to lease the optical services from the provider in protected mode and with the remote port shutdown feature, allowing immediate detection of optical link down from end-to-end. Since the network underlay is layer 3 and can be extended across long distances, it may be worth checking the continuity of the VXLAN tunnel established between PoDs. BFD can be leveraged for that specific purpose.

Independent Control Planes

When deploying VXLAN EVPN we need to consider 3 independent layers:

• The layer 2 overlay data plane
• The layer 3 underlay control plane
• The layer 3 overlay control plane

Data Plane: The overlay data plane is the extension of the layer 2 segments established on the top of the layer 3 underlay. It is initiated from VTEP to VTEP, usually enabled at the leaf node layer. In the context of geographically dispersed Multipod, VXLAN tunnels are established between leaf devices belonging to separate sites and within each site as if it was one single large fabric (hence the naming convention of Multipod).

Underlay Control Plane:  The underlay control plane is used to exchange reachability information for the VTEP IP addresses. In the most common VXLAN deployments, IGP (OSPF, IS-IS, or EIGRP) is used for that purpose, but not limited to. BGP can also be an option. The underlay protocol mainly relies on the Enterprise experiences and requirements.

Overlay Control Plane: MP-BGP EVPN is the control plane used in VXLAN deployments for the host reachability and distribution information across the whole Multipod fabric. It is used by the VTEP devices to exchange tenant-specific routing information for endpoints connected to the VXLAN EVPN fabric and to distribute inside the fabric IP prefixes representing external networks. In the context of Multipod dispersed across different rooms (same building or same campus), host reachability information is exchanged between VTEP devices belonging to separate locations as if it was one single fabric.

Classical VXLAN Multipod deployment across different rooms within the same building

This above design represents the extension of the 3 main components required to build a VXLAN EVPN Multipod fabric between different rooms. This is a basic architecture using direct fibers between two pairs of transit leaf nodes deployed in each physical Fabric (PoD-1 & PoD-2). The whole architecture offers a single stretched VXLAN EVPN fabric with the same data plane and control plane deployed end-to-end. The remote host population is discovered and updated dynamically on each PoD to form a single host reachability information database.

If that design deployed within the same building or even within the same campus can be considered as a safe network design architecture, for long distances (Metropolitan-area distances) the crucial priority must be to improve the sturdiness of the whole solution by creating independent control planes due to the long distances between PoDs.

The approach below shows that the data plane is still stretched across remote VTEP’s. The VXLAN data plane encapsulation extends the virtual overlay network end-to-end. This implies the creation of VXLAN ‘tunnels’ between VTEP devices that sit in separate sites.

However the underlay and overlay control planes are independent.

Resilient VXLAN Multipod deployment for Geographically Dispersed Locations

Underlay Control Plane:  Traditionally in Enterprise datacenter, OSPF is the IGP protocol of choice for the underlay network. In the context of geographically distant PoDs, each PoD is organised in a separate IGP Areas, with Area 0 typically deployed for the interpod links. That helps reducing the effects of interpod link bouncing, usually due to a “lower” quality of inter-sites connectivity compared to direct fiber links.

However, in order to improve the underlay separation between the different PoDs, BGP can also be considered as an alternative to IGP for the transit routing Inter-sites links.

Overlay Control Plane: In regard to the MP-BGP EVPN overlay control plane, the recommendation is to deploy each PoD in a separate MP-iBGP autonomous system (AS) interconnected through MP-eBGP sessions. When compared to the single Autonomous System model, using MP-eBGP EVPN sessions between distant PoDs simplifies interpod protocol peering.

Improving simplicity and resiliency

In a typical VXLAN fabric, leaf nodes connect to all the spine nodes, but no leaf-to-leaf or spine-to-spine connections are usually required. The only exception is the vPC peer link connection required between two leaf nodes configured as part of a common vPC domain. When Leaf devices are paired using vPC, this allows the locally attached endpoints to be dual-homed and take advantage of the Anycast VTEP functionality (same VTEP address used by both leaf nodes part of the same vPC domain).

vPC Anycast VTEP

It is not required to dedicated routing devices for the layer 3 transit function between sites. One of a pair of leaf nodes (usually named  “Border leaf nodes” for its roles connecting the fabric to outside) can be leveraged for the function of “transit leaf nodes” simplifying the deployment. This coexistence can also be enabled on any pair of computing or service leaf nodes. That doesn’t require any specific function dedicated for the transit role, excepted that the hardware must support the Bud Node design (well supported on Nexus series).

vPC Border Leaf Nodes as Transit Leaf Nodes

The vPC configuration should be performed following the common vPC best practices. However it is also important enabling the “peer-gateway” functionality . In addition and depending on the deployment scenario, the topology, the amount of devices and subnets, additional IGP tuning might become necessary to improve convergence. This relies on standard IGP configuration. Please feel free to read the white paper for further details.

IP Multicast for VXLAN Multipod geographically dispersed

One task of the underlay network is to transport Layer 2 multidestination traffic between endpoints connected to the same logical Layer 2 broadcast domain in the overlay network. This type of traffic concerns Layer 2 broadcast, unknown unicast, and multicast traffic (aka BUM).

Two approaches can be used to allow transmission of BUM traffic across the VXLAN fabric:

• Use IP multicast deployment in the underlay network to leverage the replication capabilities of the fabric spines delivering traffic to all the edge VTEP devices.
• If multicast is not an option, it is possible to use the ingress replication capabilities of the VTEP nodes to create multiple unicast copies of the BUM workflow to be sent to each remote VTEP device.

It is important to note that the choice made for the replication process (IP Multicast or Ingress replication) concerns the whole Multipod deployment as it acts as a single fabric. It is more likely that IP Multicast might be enabled for Metro distances, as the Layer 3 over the DWDM links is usually managed by the enterprise itself. However it is probably not given to get IP Multicast option or a limited number of multicast groups from the provider for layer 3 managed services (WAN). The latter will impose the Ingress Replication mechanism deployed end-to-end the VXLAN Multipod. Notice that the choice for Ingress Replication may have an impact in regard to the scalability, depending on the total number of leaf nodes (VTEP). Worth to check first.

If IP Multicast is the preferred choice, then two approaches can be made:

Anycast Rendezvous-Point PIM and MSDP in a Multipod Design

1. Either deploying the same PIM with anycast RP across the two distant PoDs if you want to simplify the configuration. That is indeed a viable solution.
2. Or better and recommended is to keep the PIM-SM domain independently enabled on each site and leverage MSDP for the peering establishment between rendez-vous points across the underlying routed network.

Improving the Sturdiness of the VXLAN Multipod Design

Until all functions are embedded in the future in the VXLAN standardisation, preventing the creation of Layer 2 loops, alternative short-term solutions become necessary. The  suggested approach consists of the following two processes:

• Prevention of End-to-End Layer 2 Loops: Use edge port protection features in conjunction with VXLAN EVPN to prevent the creation of a Layer 2 loops. The best-known feature is BPDU Guard. BPDU Guard is a Layer 2 security tool that should be enabled on all the edge interfaces that connect to endpoints (hosts, firewalls, etc.). It prevents the creation of a Layer 2 loop by disabling the switch port after it receives a spanning-tree BPDU. BPDU Guard can be configured at the global level or the interface level.

VXLAN with BPDU Guard

• Mitigation of End-to-End Layer 2 Loops: A different mechanism is required to mitigate the effects of any form of broadcast storms. When multicast replication is used to handle the distribution of BUM, it is recommended to enable the storm-control function to rate-limit the received amount of BUM traffic carried via the IP Multicast group of interest. The rate limiter of multicast traffic must be initiated on the ingress interfaces of the transit leaf nodes, hence protecting all networks behind it.

Storm-Control Multicast on Interpod Links

If it is strongly recommended to enable Storm Control for Multicast at the Ingress transit interfaces, it is also a priority to consider the current utilisation of multicast traffic produced by the Enterprise applications across remote PoDs, prior to give a threshold value. Otherwise the risk is to make more damage for the user applications than protecting against broadcast storm disaster. As a result, the rate-limit value for BUM traffic should be greater than the application data multicast traffic (use the highest value as a reference) plus the percentage of permitted broadcast traffic.

Optionally, another way to mitigate the effects of a Layer 2 loop inside a given PoD is to also enable storm-control multicast on the spine interfaces to the leaf nodes. More importantly that the previous statement, it is crucial to understand the normal application multicast utilisation rate within each PoD before using a rate limiting threshold value.

Conclusion

The deployment of a VXLAN Multipod fabric allows extending VXLAN overlay data plane in conjunction with its MP-BGP EVPN overlay control plane for Layer 2 and Layer 3 connectivity across multiple PoDs. Depending on the specific use case and requirements, those PoDs may represent different rooms in the same physical data center, or separate data center sites (usually deployed at Metropolitan-area distances from each other).

The deployment of the Multipod design is a logical choice to extend connectivity between fabrics that are managed and operated as a single administrative domain. However, it is important to consider that it behaves like a single VXLAN fabric. This characteristic has important implications for the overall scalability and resiliency of the design.

A true multisite solution with a valid DCI solution is still the best option for providing separate availability zones across data center locations.

As discussed in previous articles, VXLAN data plane encapsulation in conjunction with its control plane MP-BGP AF EVPN is becoming the foremost technology to support the modern network Fabric.

DCI is an solution architecture that you deploy to interconnected multiple data centers to extend Layer 2 and/or Layer 3 with or without multi-tenancy. A DCI architecture relies on a Data Plane for the Transport and a Control plane for an efficient and solid mechanism for endpoint discovery and distribution between sites (and much more).  VXLAN is encapsulated method, it’s not a architecture. If we want to divert VXLAN for DCI purpose, it is crucial that we understand what we need and how to address the requirement with a VXLAN transport.

Some articles from different vendors claiming that VXLAN can be simply leveraged for DCI purposes do frighten me a bit! Either they are a bit light or they don’t understand what interconnecting multiple data centers implies. You could indeed easily built a crap solution to interconnect multiple sites using VXLAN tunnels, if you just wish to extend layer 2 segments outside the data center. Or, you can build a solid DCI architecture based on business requirements using OTV or PBB EVPN or even VXLAN EVPN, however the latter implies some additional features and sophisticated configurations.

It is therefore interesting to clarify how to interconnect multiple VXLAN/EVPN fabrics geographically dispersed across different locations.

If we look at how to interconnect VXLAN-based fabrics at layer 2 and layer 3, three approaches can be considered:

• The 1st option is the extension of multiple sites as one large single stretched VXLAN Fabric. There is no network overlay boundary, nor VLAN hand-off at the interconnection, which simplifies operations. This option is also known as geographically dispersed VXLAN Multipod. However we should not consider this solution as a DCI solution per se as there is no demarcation, nor separation between locations. Nonetheless this one is very interesting for its simplicity and flexibility. Consequently we have deeply tested and validated this design (see next article).
• The second option to consider is multiple VXLAN/EVPN-based Fabrics interconnected using a DCI Layer 2 and layer 3 extension. Each greenfield Data Center located on different site is deployed as an independent fabric, increasing autonomy of each site and enforcing global resiliency. This is also called Multisite in opposition to the previous one Multipod. As a consequence, a efficient Data Center Interconnect technology (OTV, VPLS, PBB-EVPN, or even VXLAN/EVPN) is used to extend Layer 2 and Layer 3 connectivity across separate sites.For the VXLAN multisite scenario, there are 2 possible models that need to be taken into reflection:

Model 1: Each VXLAN/EVPN is a Layer 2 fabric only, with external devices (Routers, FW, SLB) offering routing and default gateways functions (with or without FHRP localization). This one is similar to the traditional multiple DC to interconnect and doesn’t constrain any specific requirements, except the DCI solution itself to interconnect the fabrics in a resilient and secure fashion, as usual.

Model 2: Each VXLAN/EVPN is Layer 2 and Layer 3 fabric.  VXLAN EVPN indeed brings a great added-value with L3 Anycast Gateway as discussed in this post. Consequently most of enterprises could be interested to leverage this function distributed across all sites with the same virtual IP and virtual MAC addresses . This crucial feature, when geographically dispersed among multiple VXLAN fabrics, improves performances and efficiency with the default gateway being active transparently for the endpoints located on each site from their first hop router (leaf node where endpoints are directly attached to). It offers hot live mobility with E-W traffic optimization beside the traditional zero business interruption. However because of the same virtual MAC and same virtual IP address exist on both side, some tricky configurations must be achieved.

.

• A third scenario, which can be considered somehow as a subset of the stretch fabric, is to dedicate the VXLAN/EVPN overlay to extend Layer 2 segments from one DC to another. Consequently, VXLAN/EVPN is agnostic of the Layer 2 transport deployed within each location (it could be vPC, FabricPath, VXLAN, ACI, just to list few of them). VLANs requiring to be extended to outside the DC are prolonged up to the local DCI devices from where layer 2 frames  are encapsulated with a Layer 3 VXLAN header, and sent afterward toward the remote site over a layer 3 network, to be finally de-encapsulated on the remote DCI device and distributed within the DC accordingly. In this scenario vPC is also leveraged to offer DCI dual-homing functions for resiliency and traffic load distribution.

This option can be analogous to OTV or PBB-EVPN from an overlay point of view. However as already mentioned, it must be understood that VXLAN has not been built natively to address all the DCI requirements. Nevertheless the implementation of EVPN control-plane could be expanded in the future to include the multi-homing functionality, delivering failure containment, loop protection, site-awareness, or optimized multicast replication like OTV can already offer since its beginning.

BTW it’s always interesting to highlight that since NX-OS 7.2(0)D1(1), OTV relies on VXLAN encapsulation for the data-plane transport as well as the well known ISIS as the control plane natively built for DCI purposes. Hum, interesting to notice that OTV can also be a VXLAN-based DCI solution, however it implements the right control plane for DCI needs. That’s why it is important to understand that not only a control plane is required for Intra-fabric and Inter-DC network transport, but this control plane must natively offer the features to address the DCI requirements.

VxLAN/EVPN and Integrated Routing Bridging

Summary

As I mentioned in the post  28 – Is VxLAN Control Plane a DCI solution for LAN extension, VxLAN/EVPN is taking a big step forward with its Control Plane and could be used potentially for extending Layer 2 segments across multiple sites. However it is still crucial that we keep in mind some weaknesses and lacks related to DCI purposes. Neither VxLAN nor VXLAN/EVPN have been designed to offer natively a DCI solution (post 26 & 28).

DCI is not just a layer 2 extension between two or multiple sites. DCI/LAN extension is aiming to offer business continuity and elasticity for the cloud (hybrid cloud). It offers disaster recovery and disaster avoidances services for Enterprise business applications, consequently it must be very robust and efficient. As it concerns on Layer 2 broadcast domain, it is really important to understand the requirement for a solid DCI/LAN extension and how we can leverage the right tools and network services to address some of the shortcomings that rely on the current implementation of VxLAN/EVPN offering a solid DCI solution.

In this article we will examine the integrated anycast L3 gateway available with VxLAN/EVPN MP-BGP control plane, which is one of the key DCI requirements for long distances when the first hop default gateway can be duplicated on multiple sites.

Integrated Routing and Bridging

One of the needs for an efficient DCI deployment is the duplicate Layer 3 default gateway solution such as FHRP isolation (24 – Enabling FHRP filter),  reducing the hair-pining workflows while machines are migrating from one site to another. In short, the same default gateway (virtual IP address + virtual MAC) exists and is active on both sites. When a virtual machine or the whole multi-tier application executes a hot live migration to a remote location, it is very important that the host continues to use its default gateway locally and processes the communication without any interruption.Without FHRP isolation, multi-tier applications (E-W traffic) will suffer in term of performances reaching their default gateway in the primary data center.

This operation should happen transparently with the new local L3 gateway, whereas the round-trip workflow via the original location is eliminated. No configuration is required within the Host. All active sessions can be maintained stateful.

Modern techniques such as Anycast gateway can offer at least the same efficiency as FHRP isolation, and not only for Inter-Fabric (DCI) workflows but also for Intra-Fabric traffic optimization. This task is achieved in an easier way because it doesn’t require specific filtering to be configured on each site. The function of the Anycast Layer 3 gateway is natively embedded with the BGP EVPN control plane. And last but not least, in conjunction with VxLAN/EVPN there are as many Anycast L3 gateways as Top-of-Rack switches (Leafs with VxLAN/EVPN enabled).

The EVPN IETF draft elaborated the concept of Integrated Routing and Bridging based on EVPN to address inter-subnet communication between Hosts or Virtual Machines that belong to different VxLAN segments. This is also known as inter-VxLAN routing. VxLAN/EVPN offers natively the Anycast L3 gateway function with the Integrated Routing Bridging feature (IRB).

VxLAN/EVPN MP-BGP Host learning process

The following section is a slight reminder on the Host reachability discovery and distribution process to better understand later the different routing mode.

The VTEP function happens on a First Hop Router usually enabled on the ToR where the Hosts are directly attached. EVPN provides a learning process to dynamically discover the local end-points attached to their local VTEP, distributing afterward the information (Host’s MAC and IP reachability) toward all other remote VTEPs through the MP-BGP control plane. Subsequently, all VTEPs know all end-point information that belongs to their respective VNI.

Let ‘s elaborate the description with the following example.

vPC Anycast VTEP

VLAN 100 (blue) maps to VNI 10000

• Host A: 192.168.1.188
• Host C: 192.168.1.187
• Host E: 192.168.1.193
• Host F: 192.168.1.189

VLAN 30 (orange) maps to VNI 5030

• Host B: 192.168.3.187
• Host D: 192.168.3.193

VLAN 3000 (purple) maps to VNI 300010.

• Host G: 192.168.32.189

In order to keep the following scenarios simple, each pair of physical Leafs (vPC) will be represented by a single logical switch with the Anycast VTEP stamped to it.

vPC Anycast VTEP – Physical View vs. Logical View

VTEP 1: 10.0.2.11:

VLAN 30 and VLAN 3000 are not present in Leaf 1, consequently it is not required to create and map the VLAN 30 to VNI 5030, nor VLAN 3000 to VNI 300010 within VTEP 1. VTEP 1 only needs to get the reachability information for Hosts A, C, E and F (IP & MAC addresses) belonging to VLAN 100. Host A is local to VTEP 1 and Hosts C, E and F are remotely extended using the network overlay VNI 10000.

VTEP 2: 10.0.2.13

VTEP 2 learned Hosts A, C, E and F, attached to VLAN 100, and Hosts B and D, belonging to VLAN 30. Hosts B and C are local.

VTEP 3: 10.0.2.25

VTEP 3 learned Hosts A, C, E and F, attached to VLAN 100, and Hosts B and D, belonging to VLAN 30. Hosts D and E are local.

VTEP 4: 10.0.2.27

VTEP 4 learned Hosts A, C, E, and F on VLAN 100 and Host G on VLAN 3000. Hosts F and G are local.

Notice that only VTEP 4 learns Host G, which belongs to VLAN 3000. Consider Host G to be isolated from the other segments, like for example a database accessible only via routing.

VLAN to Layer 2 VNI mapping

With the process for Host reachability information, the learning and distribution end-points are achieved by the MP-BGP EVPN control plane with an integrated routing and bridging (IRB) function. Each VTEP learns through the data plane (e.g. Source MAC from a Unicast packet or a ARP/GARP/RARP) and registers its local end-points with their respective MAC and IP address information using the Host Mobility Manager (HMM). Subsequently, it distributes this information through the MP-BGP EVPN control plane. In addition to registering its local attached end-points (HMM), every VTEP will populate its Host table with the devices learned through the control plane (MP-BGP EVPN AF).

From Leaf 1 (VTEP1 : 10.0.2.11)

L2route from Leaf 1

The “show L2route” command[1] above details the local Host A (192.168.1.188) learned by the Host Mobility Manager (HMM), as well as Host C (192.168.1.187), E (192.168.1.193) and F (192.168.1.189) learned through the BGP on the next hop (remote VTEP) to reach them.

[1] Please notice that the output has been modified to show only what’s relevant to this section.

From Leaf 3 (VTEP2 : 10.0.2.13)

L2route from Leaf 3

Host B (192.168.3.188) and Host C (192.168.1.187) are local, whereas Host D (192.168.3.193) and Host E (192.168.1.193) are remotely reachable through VTEP 3 (10.0.2.25) and Host F (192.168.1.189) appears behind VTEP 4 (10.0.2.27).

From Leaf 5 (VTEP3 : 10.0.2.25)

L2route from Leaf 5

Host D (192.168.3.193) and Host E (192.168.1.193) are local. All other Hosts except Host G are reachable via remote VTEPs listed in the Next Hop column.

From Leaf 7 (VTEP4 : 10.0.2.27)

L2route from Leaf 7

Host F (192.168.1.189) and Host G (192.168.32.189) are local. VTEP 4 registered only to the Hosts that belong to VLAN 100 and VLAN 3000 throughout the whole VxLAN domain.

With the Host table information, each VTEP knows how to reach all Hosts within its configured VLAN to L2 VNI mapping.

Asymmetric or symmetric Layer 3 workflow

The EVPN draft defines two different operational models to route traffic between VxLAN overlays. The first method describes an asymmetric workflow across different subnets and the second method leverages the symmetric approach.

Nonetheless, vendors offering VxLAN with EVPN control plane support may implement one of these operational models (assuming the Hardware/ASIC supports VxLAN routing). Some may choose the Asymmetrical approach, maybe because it is easier to implement from a software point of view, but it is not as efficient as the symmetrical mode and has some risks that impact scalability. Others will choose the symmetric model for more efficient population of Host information with better scalability.

Assuming the following scenario:

• Host A (VLAN 100) wants to communicate with Host G in a different subnet (VLAN 3000)
• VLAN 100 is mapped to VNI 10000
• VLAN 30 is mapped to VNI 5030
• VLAN 3000 is mapped to VNI 300010
• We assume all Hosts and VLANs belong to the same Tenant-1 (VRF)
• * L3 VNI for this Tenant of interest is VNI 300001 (for symmetric IRB)
• We assume within the same Tenant-1 that all Hosts from different subnets are allowed to communicate with each other (using inter-VxLAN routing).
• Host A shares the same L2 segment with Hosts C, E, and F spread over remote Leafs
• Host G is Layer 2 isolated from all other Hosts. Therefore a Layer 3 routing transport is required to communicate with Host G.

* explained with Symmetric IRB mode

Asymmetric IRB mode:

When an end-point wants to communicate with another device on a different IP network, it sends the packet with the destination MAC address as its default gateway. Its first-hop router (ingress VTEP) performs the routing lookup and routes the packet to the destined L2 segment. When the egress VTEP receives the packet, it strips off the VxLAN header and bridges the original frame to the VLAN of interest. With asymmetrical routing mode, the ingress VTEP (the local VTEP where the source is attached) performs both bridging and routing, whereas the egress VTEP (remote VTEP where the destination sits) performs only bridging. Consequently, the return traffic will take a different VNI, hence a different overlay tunnel.

In the following example, Host A wants to communicate with Host G and sends the packet toward its default gateway. VTEP 1 sees that the destination MAC is its own address and does a routing lookup for Host G. It finds in its Host table the destined IP Host and the Next Hop VTEP 4 to reach it. VTEP 1 encapsulates and routes the packet into the L2 VNI 300010 with VTEP 4 as destination IP address. VTEP 4 receives the packet, strips off the VxLAN header and bridges the frame to its VLAN 3000 toward Host G (with the source MAC as the default gateway address).

When Host G responds, VTEP 4 will encapsulate the frame to its associated VNI 300010 and will route the packet directly through the VNI 10000 where Host A is registered. The egress VTEP 1 will therefore bridge the received packet from VNI 10000 toward Host A in VLAN 100.

Asymmetric Routing

The drawback of this asymmetric routing implementation is the consistent configuration across all VTEPs built with all VLANs and VNIs concerned with routing and bridging communication across the fabric. In addition, it needs to learn the reachability information (MAC and IP addresses) for all Hosts that belong to all VNI of interest.

In the above illustration, VTEP 1 needs to be configured with VLAN 100 mapping VNI 10000, as well as the VLAN IDs which are mapped to the VNIs 5030 and 300010, even though there is no any Host attached to those VLANs. The VLAN ID being local significant, what is crucial is that the mapping to the VNI of interest exists. Therefore, in this example the Host tables on all leafs are populated with the reachability information for Hosts A, B, C, D, E, F and G. In a large VxLAN/EVPN deployment, this implementation of asymmetric IRB adds some complexity for the configuration but above all it may have important impact in term of scalability.

Symmetric IRB

Symmetric routing behaves differently in the sense that both ingress and egress VTEP provide bridging and routing functions. This allows introducing a new concept known as transit L3 VNI. This L3 VNI will be dedicated for routing purposes within a tenant VRF.  Indeed, the L3 VNI offers L3 segmentation per tenant VRF. Each VRF instance is mapped to a unique L3 VNI in the network. Each series of tenant’s VLAN determines the VRF context to which the receiving packet belongs. As a result, the inter-VxLAN routing is performed throughout the L3 VNI within a particular VRF instance.

Notice that each VTEP can support several hundred VRFs (depending on the hardware, though).

In the following example, all Hosts, A, B, C, D, E, F and G, belong to the same tenant VRF. When Host A wants to talk to Host G in a different Subnet, the local VTEP (ingress) sees the destination MAC as the MAC of the Default Gateway (AGM – Anycast Gateway MAC), as Host G is not know via Layer-2 and consequently routes the packet through the L3 VNI 300001. It rewrites the inner destination MAC address with the egress VTEP 4 router MAC address that, this one is unique for each VTEP.  Once the remote VTEP 4 (egress) receives the encapsulated VxLAN packet, it strips of the VxLAN header and does a MAC lookup identifying the destined MAC as being its own. Accordingly, it performs an L3 lookup and routes the receiving packet from the L3 VNI to the destined L2 VNI 300010 that maps VLAN 3000 where Host G resides. VTEP 4 finally maps VNI 300010 to VLAN 3000 and forwards the frame with an L2 destination as Host G.

Symmetric Routing

For return traffic, VTEP 4 will achieve the same packet walk using the same transit L3 VNI 300001, but in the opposite direction.

The following L2route command displays the Host reachability information that exists in VTEP 1’s Host table. It doesn’t show Host G (192.168.32.189), as no VLAN maps locally in Leaf 1 (VTEP 1) the VNI 300010 nor information related to VNI 5030, reducing therefore the population of its Host reachability table to the minimum required information.

L2route from Leaf 1

Nonetheless, Leaf1 knows how to route the packet to that destination. The Next hop is Leaf_4 via the L3 VxLAN segment ID 300001, as displayed in the following routing information.

Show ip route from Leaf 1

The “show IP route” command above shows that Host G is reachable via the VxLAN segment ID 300001 with the next hop as 10.0.2.27 (VTEP 4).

Show bgp l2vpn evpn from Leaf 1

The “show bgp L2vpn evpn” command[1] from Leaf 1 displays the information related to Host G (192.168.32.189), reachable via the next hop, 10.0.2.27 (VTEP 4) using the L3 VNI 300001.

Note also for information only that the label shows the mapping of the L2VNI 300010 where Host G belongs to and the L3VNI 300001 to reach Host G.

[1] Please notice that the output has been modified to display only information relevant to Host G.

Anycast L3 gateway

One of the useful added-values of this, is the capacity to offer the role of default gateway from the direct attached-Leaf (First Hop Router initiated on ToR). All leafs are configured with the same Default Gateway IP address for a specific subnet as well as the same vMAC address. All routers will use the same virtual Anycast Gateway MAC (AGM) for all Default Gateways. This is explicitly configured during the VxLAN/EVPN setup.

Leaf(config)# fabric forwarding anycast-gateway-mac 0001.0001.0001

When a virtual machine processes a “hot live migration” to a Host attached to a different Leaf, it will continue to use the same default gateway parameter without interruption.

Anycast Layer 3 Gateway

In the above scenario, the server “WEB” in DC-1 communicates (1) with its database attached to a different Layer 2 network and therefore a different subnet in DC-2. The local Leaf 1 achieves the default gateway role for server WEB as the First Hop Router, which is simply its own ToR with the required VxLAN/EVPN and IRB services up and running.

While server WEB  is communicating with the end-user (not represented here) and its database, it does a live migration (2) toward the remote VxLAN fabric where its database server resides.

The MP-BGP EVPN AF control plane notices the movement and the new location of the server WEB. Subsequently, it increases the MAC mobility Sequence number value for that particular end-point and notifies all VTEP of the new location. As the Sequence number is now higher than the original value, subsequently all VTEP update their Host table accordingly with the new “next hop” (egress VTEP 4).

Without interruption of the current active sessions, server WEB continues to transparently use its default gateway, locally available from its new physical First Hop Router, now being Leaf 7 and 8 (VTEP 4). The East-West communication between WEB and DB happens locally within DC-2.

This post discusses about design considerations when interconnecting two tightly coupled fabrics using dark fibers or DWDM, but not limited to Metro distances. If we think very long distances, the point-to-point links can be also established using a virtual overlay such as EoMPLS port x-connect; nonetheless the debate will be the same.

Notice that this discussion is not limited to one type of network fabric transport, but any solutions that use multi-pathing is concerned, such as FabricPath, VxLAN or ACI.

Assuming the distance between DC-1 and DC-2 is about 100 km; if the following two design options sound quite simple to guess which one might be the most efficient, actually it’s not as obvious as we could think of, and a bad choice may have a huge impact for some applications. I met several networkers discussing about the best choice between full-mesh and partial-mesh for interconnecting 2 fabrics. Some folks think that full-mesh is the best solution. Actually, albeit it depends on the distances between network fabrics, this is certainly not the most efficient design option for interconnecting them together.

Partial-Mesh with transit leafs design (left)                                                                 Full-Mesh (right)

For this Thread, we assume the same distance between the two locations exists with both option designs (let’s take 100 km for a comprehensive math). On the left side, we have a partial-mesh design with transit leafs to interconnect the two fabrics. On the right side, we have a full-mesh established between the two fabrics.

With a full-mesh approach the inter-fabric workflow will take one hop to communicate with a remote host. As the result the total latency induced here with full-mesh will be theoretically 1 ms (100 km) between two remote end-points + the few micro-secs for traversing the spine in DC-2 (one hop).

Full-Mesh

However, if the 2 locations are interconnected using a transit leaf, the traffic between the two tiers will take 3 hops. In this case, it will add theoretically few micro-secs for traffic crossing the spine switch in DC-1 + 1ms (100km) + few micro-secs (remote transit leaf + spine switch in DC-2 ).

Partial-Mesh using Transit Leaf

With the partial-mesh, each transit leaf interconnects the remote Spine. Thanks to the Equal Cost Multi-Pathing (ECMP) algorithm used to distribute the traffic among all possible equal paths. The server to server workflows (E-W) are therefore established between hosts and confined within the same location (e.g Web/App tier and DB tiers) with a maximum of one hop. A flow sent to the remote location via the DCI connection would consume two additional hops. As the result, ECMP will never elect the link toward the remote spine and the traffic between the application-tiers will be contained inside the same location (DC-1).

In a Partial-mesh design, ECMP contains the local workflow within the same location

However if the two locations were interconnected using a full-mesh design,  nothing would prevent the ECMP hashing to hit a remote spine, as the number of hops is equal across the two locations.

As the result a full-mesh will add theoretically 2 ms (2 x 100km) for a portion of the traffic.

In a Full-mesh design, ECMP will distribute a portion of the local traffic via the remote fabric

If we compare the Full-mesh and the Partial-mesh design, we have the same latency induces by the distances between sites (we can ignore the few micro-secs due to multi-hops ). In Partial-mesh the local communication between two local end-points is contained within the same fabric, while with full mesh, although it saves some few micro-seconds with a maximum of one hop between two end-points, it is not guaranteed that the traffic between two local end-points will be always contained within the same location. Risks are that for the same application, some flows will take a local path (with a negligible latency) and some will take the remote path (with n msec due to the hair-pining via the remote spine switch).

Key considerations

• With long distances between two fabrics, E-W communication between two local end-points may have an impact on the Application performances with a full-mesh design.
• Any network transport that uses multi-pathing is concerned, such as FabricPath, VxLAN or ACI.
• A full-mesh will provide a single hop between the 2 remote end-points. However it will allow distributing the E-W flows equally between the 2 sites. As the result, half of the traffic may be impacted by the latency induced by the distance.
• A Transit leaf design will impose a 3 hops between the two remote end-points, however the latency induced by the local hop in each site is insignificant compared to the DCI distance. A Transit Leafs design (partial mesh) will contain the E-W flows within a fabric locally to each site.

As a global conclusion, for a single logical fabric geographically stretched across two physical distant location, when distances go beyond the campus distances, it may be important to deploy a transit leaf model.

All that said, it is certainly unlikely that several enterprises deploy a full-mesh interconnecting fabric for long distances, but nothing is impossible, it depends who owns the fiber 🙂

A fantastic overview of the Elastic Cloud project from Luca Relandini

http://lucarelandini.blogspot.com/2015/03/the-elastic-cloud-project-porting-to.html?

And don’t miss this excellent recent post which explains how to invoke UCS Director workflows via the northbound REST API.

http://lucarelandini.blogspot.com/2015/03/invoking-ucs-director-workflows-via.html 

Back to the recent comments on  what is “officially” supported or not ?

First of all, let’s review the different Firewall forwarding mode officially supported

ASA cluster  deployed inside a single data center:

Fig.1 Firewall forwarding mode within a single DC. Please note the firewall routed mode supported with the Layer 2 load balancing (LACP) Spanned Interface mode.

When configured in Routed mode (e.g. default gateway for the machines), the same ASA identifiers IP/MAC are distributed among all ASA members of the cluster. When the ASA cluster is stretched across different locations, the Layer 2 distribution mechanism facing the ASA devices is achieved locally using pair of switches (usually leveraged the a Multi-chassis EthernetChannel technique such as VSS or vPC).

Subsequently the same virtual MAC address (ASA vMAC) of the ASA cluster is duplicated on both sites and as the result it hits the upward switch from different interfaces.

Fig.2 ASA and duplicate vMAC address

Therefore when the ASA cluster runs the firewall routed mode with Spanned interface method, in a geographically stretched fashion, it breaks the Ethernet rules due to the duplicate MAC address, with risks of affecting the whole network operation. Consequently by default it is not supported as expressed in the release notes for the Inter-site section.

Fig.3 The routed mode is not supported with Spanned Interface mode for ASA cluster stretched across multiple locations.

The above requires a clarification. In this context, “NOT supported” means that no other design alternative allowing the firewall routed mode to be enabled in conjunction with Spanned interface mode has been tested by the Quality Assurance (QA) process.

However it doesn’t mean that there is no design workaround to address this requirement, and some additional network services may help to keep the ASA in Routed mode.

• The 1st option could be to filter the Virtual MAC address in the interface facing the DCI link.

This option may be desired if you need to use the ASA as the default gateway for the servers for example. However it must be enabled with caution, thus this option is definitely not recommended, except if you know exactly what you are doing.

The first drawback to keep in mind is that if the vMAC address is blocked, it results that the original ASA member cannot maintain the session stateful with a virtual machine  moving to the remote site. Consequently this option will be limited to “cold migration” using subnet extension (e.g. for cost containment or migration purposes).

The second concern is that to address any black hole scenarios, this option imposes as well the ingress traffic to be dynamically redirected to the new location as soon as the application of interest has migrated to (e.g. LISP mobility).

Fig.4 Statefull session is not supported with vMAC filtering

• A 2nd alternative more robust IMO is to isolate the layer 2 domain with a Router

With this option, the L2 domain used to distribute the traffic among the 2 local ASA devices is isolated from the data VLAN, hence the pair of switches facing the ASA members sees the vMAC only from the direct attached Firewalls. Only the data VLAN is stretched across the 2 locations (plus the CCL which is not represented in this diagram)

Fig.5 Router insertion to isolated the vMAC. ASA are configured in firewall routed mode with static routing.

However there are few disadvantages of using this design with the default gateway that resides between the application servers and the ASA clusters:

1. The ASA cannot be the default gateway of the servers due to the inserted first hop router.
2. The routing service on the ASA must be configured with static entries. Indeed, the dynamic routing is centralized to the master, hence routing adjacency on non-master site without a full stretched subnet will not succeed. On the other hand, if the subnet is stretched, we lose the ability to localize transit traffic to the local site.
3. It becomes challenging to secure the E-W traffic (e.g. Web-tier <=> App-tier <=> DB) with the same ASA cluster.

Note for the last bullet that IPS are usually more efficient (and mandatory) between application tiers than a classical firewall function However the ASA offers embedded IPS too, thus this comment I think may be important.

If theoretically the second alternative works as I described it, it is not yet officially supported, and it will require several tests with different scenarios that you may want to run to this validate this option for your environment.

Having said that, a new East-West insertion mode has been qualified and validated with the recent release 9.3(2), technically nothing new, it ‘s just that this one has been deeply tested. In this scenario the E-W traffic can nowbe secured via the same ASA cluster.

Fig.7 ASA in transparent mode. East-West traffic is secured using the ASA cluster.

When the ASA is configured in Firewall Transparent mode, the virtual MAC is not an issue anymore as the ASA members use a BVI address. The BVI address doesn’t appear throughout the data plane. Consequently there is no risk of duplicate MAC address. As the result, the outside router can act as a default gateway for all application servers of interest, allowing the E-W traffic (server to server communication) to be secured via the same ASA cluster. Keep in mind that it requires FHRP isolation to support hot live migration.

Additional details on E-W insertion design can be found in this release-notes 9.3(2)

I hope that post clarifies some statements within the official ASA document and alternative design.

How can we talk about security service extension across multiple locations without elaborating on path optimisation ?  🙂

Path Optimization with ASA Cluster stretched across long Distances

In the previous post, 27 – Active/Active Firewall spanned across multiple sites – Part 1, we demonstrated the integration of ASA clustering in a DCI environment.

We discussed the need to maintain the active sessions stateful while the machines migrate to a new location. However, we see that, after the move, the original DC still receives new requests from outside, prior to sending them throughout the broadcast domain (via the extended layer 2), reaching the final destination endpoint in a distant location. This is the expected behavior and is due to the fact that the same IP broadcast domain is extended across all sites of concern. Hence the IP network (WAN) is natively not aware of the physical location of the end-node. The routing is the best path at the lowest cost via the most specific route. However, that behavior requires the requested workflow to “ping-pong” from site to site, adding pointless latency that may have some performance impact on applications distributed across long distances.

With the increasing demand for dynamic workload mobility across multiple data centers, it becomes important to localize the DC where the application resides in order to dynamically redirect the data workload to the right location, all in a transparent manner for the end-users as well as the applications and without any interruption to workflow.

The goal is to dynamically inform the upward layer 3 network of the physical location of the target (virtual machines, applications, HA clusters, etc.), while maintaining the active sessions stateful and allowing new sessions to be established directly to the site where the application resides. Maintaining the sessions stateful imposes one-way symmetry, meaning that the return traffic must hit the original firewall that owns the session. However, all new sessions must take advantage of the path optimization using dynamic redirection through only local firewalls.

There are two ways to efficiently achieve these requirements:

1. The first is to use LISP mobility, as discussed in post 23 – LISP Mobility in a virtualized environment (update). LISP establishes a network overlay between Tunnel Routers (xTR) in the path between distant sites. When a machine is moved across two data centers, a notification is sent to a LISP database mapping the end-nodes with their respective last known location, which in turn notifies the LISP Egress Tunnel Routers (eTR). The session is therefore LISP-encapsulated to the data center where the application of interest currently resides. The underlying layer 3 network is not concerned or affected by any dynamic change configuration, which is a huge added value.
2. The second is a new network service called LISP IGP Assist. Actually, it’s a sub-function of a LISP First Hop Router (FHR), which is leveraged to detect the machine’s movement and redistribute the LISP route of the specific end-node (/32) into the IGP routing tables on each site. With a more specific route, the routed traffic is natively directed to the DC where the application of interest exists.

1. ASA Cluster and LISP Mobility

In the following scenario an ASA cluster built with four ASA units, two on each site, is stretched across DC-1 and DC-2. The ASA cluster is configured in Spanned Mode (cLACP). Note that it could be configured in Individual Mode, nonetheless it’s going to be exactly the same configuration and behavior for OTV and LISP services. OTV is used here to extend the ASA Cluster Control Link (CCL) as well as the data VLAN’s of the compute layer affected by the movement of the VM’s. We assume that the ASA cluster is configured in Transparent Mode. However, when writing this note in regard to ASA cluster 9.2(4), the routed mode is not yet qualified in Spanned Mode in a DCI scenario.

Let’s go step by step to better dig through the end-to-end workflow.

An end-user from a branch office requests access to an application that resides in DC-1 (1).

The request is intercepted by the Ingress Tunnel Router (iTR).

The iTR solicits a request (2) from the mapping database (aka “Map Requestor Service”) for the location of the application.

The LISP Map Server (MS) relays the request to the eTR in DC-1 where the application resides. This redirection via the eTR aims to validate the path between the relevant eTR and the iTR.

Consequently, the ETR in DC-1 informs the iTR about the location for Subnet A residing in the eTR of DC-1 (3).

Now the iTR records the location of the specific end-node (application). It can now encapsulate the user request with a LISP header and route it toward the eTR in DC-1 (4).

The eTR in DC-1 strips off the LISP header and routes the packet to the next hop to reach the subnet where the application exists.

The first level of the switch computes its LACP hashing algorithm, selecting the link to ASA-2 and forwarding the packet accordingly.

ASA-2 encodes cookies with its identifier and forwards the request to the server (5).

The application “hot” migrates to DC-2 (6) and replies to the user request (8) while maintaining a stateful session.

LISP Control Plane: In the meantime, the LISP FHR detects the presence of the new virtual machine (the application) and notifies its local eTR (7) in DC-2 about this new End-Node Identifier (EID).

Subsequently, the eTR in DC-2 informs (10) the Map Server (M-DB) about the new location of the EID. The Map Server then signals the original eTR in DC-1 about the change of location. The eTR in DC-1 updates its LISP table accordingly with a “Null0” entry for the end-node of interest.

Data Plane: Consequently, the ASA-3 receives the response from the server (8), decodes the TCP cookies, learns the owner of the TCP session and redirects the packet toward ASA-2 over the CCL (9).

The eTR on DC-1 encapsulates the IP packet for the iTR (11), which de-encapsulates the LISP header and routes it to the end-user.

One-way symmetry has been established via the CCL. The session has not been interrupted; the movement of the application has been transparent for the end-user and the application.

The end-user continues using the application and sends the next request to it.

The iTR has cached the original location of the application into its local database. Hence, it still encapsulates the IP packet for the eTR in DC-1 (12).

The eTR in DC-1 has been previously notified that it is no longer the owner of this end-node identifier and replies with a “Solicit Map Request (SMR)” (13) to the iTR in order to confirm with the Map Resolver (14) the new location of the application (EID). It then receives the new location (15) for the application with the destination the egress tunnel router eTR located in DC-2.

As the result, the iTR encapsulates the end-user’s request with a LISP header and sends it toward the eTR in DC-2 (15).

ASA-3 receives the IP packets and determines that the owner is ASA-2, to which it redistributes the workflow via the extended CCL. ASA-2 treats the packet accordingly and routes it toward the next L3 hop, which forwards it to the machine supporting the application (16).

The application responds to its local default gateway (17), thanks to FHRP filtering, which distributes the return packet to ASA-3. The LACP hashing algorithm could have also elected the ASA-4; however, the behavior and final redirection via the CCL would have been exactly the same.

ASA-3 notices again that the owner is ASA-2, to which it redistributes the packet via the CCL.

The eTR encapsulates the IP packet and forwards it to the iTR (18).

The session is maintained stateful with zero interruption.

The drawback of maintaining active sessions stateful is hairpinning workflow for the duration of the session. Thus, a question that comes to mind is:

• is the application response time still performing as before?

If the distance is very long, the active sessions may be impacted the moment the data leaves. Nonetheless, that’s only limited to the current active and stateful sessions.

For any new sessions, the traffic will be directed to the DC where the application resides. In addition, the traffic will be validated and inspected using one of the local ASA members (20).

Hence, while the current sessions are maintained stateful through the firewall that owns the active data flow, any new sessions will take the shortest path to reach the application with their respective local firewalls treating the flow accordingly. The application response time will be improved due to the distance between the end-user and the application being shortened and hairpinning reduced, if not eliminated.

2. ASA Cluster and LISP IGP Assist

Some enterprise network managers may think that LISP IP mobility might be a bit complex or challenging to deploy. The reality is that there is nothing very complex per se, as detailed here, but there are indeed several different LISP functions running on different platforms that require different configurations, including LISP iTR configuration at the branch offices. Another component to take into consideration is that the complexity depends on whether the enterprise owns the WAN edge layer or if it is layer 3 managed services.

Whatever the original reason is to not deploy LISP VM mobility, IGP Assist may be a nifty trick to think about as a slight alternative to LISP Mobility. Indeed, we just need to leverage the basic function of a LISP FHR, detecting any movements from the EID side and relaying this change to the IGP routing tables. Currently, IGP Assist is only supported on the Nexus 7k.

If you remember the great function of Route Health Injection (rest in peace) available on some SLB devices (e.g. CSM, ACE), it will be easy for you to deploy IGP Assist.

The main concept is to inject /32 upward to the Internet access (Intra- or Extranet usually) to direct traffic to the right site where the mobile application currently resides using a more specific route.

Let’s have a look, step by step, how this is achieved, and let’s apply the same scenario that we used with LISP mobility, still with OTV as the DCI solution deployed for the LAN extension.

From a routing point of view, this is traditional IGP access (Enterprise Intra-DC L3 Routing) connected to one or multiple ISPs built with an E-BGP network (L3 Internet Core).

By default, the traffic to the application (subnet A) is directed to DC-1 with the most specific route /25.

Usually for an HA cluster failover, the network manager performs a manual redirection process by withdrawing the /25 of the subnet of interest from DC-1, consequently all traffic to that subnet is re-routed to DC-2 in a few seconds (directed to the next most specific route /24).

And that’s what IGP Assist is aiming to achieve automatically with host-based granular redirection (/32).

The virtual machine (end-node) supporting the application migrates (“hot” live migration) from DC-1 to DC-2 across the LAN extension (1). The application server continues replying to the original request maintaining the whole session stateful without any interruption.

The new EID is detected on DC-2. IGP Assist notifies all LISP FHRs (2), or more precisely, the redundant LISP FHR in DC-2 as well as the two LISP FHRs in DC-1.

Meanwhile, the LISP FHR redistributes the LISP routes into the IGP routing table (3).

As the result, the IGP control plane handles this notification, the data traffic hits the local ASA-3 in DC-2, which notices that the owner of the stateful session is ASA-2. Immediately, ASA-3 redistributes the packet flow to ASA-2 via the CCL, which in turn forwards it toward the requestor throughout the WAN edge layer on DC-1.

Following the LISP route redistribution into the IGP routing protocol, on DC-2 where the detection of the EID has been made, IGP Assist installs a /32 LISP Interface Route of the host (EID or virtual machine supporting the application of concern). On DC-1, the original location of the application, it removes the /32 LISP Interface Route of the same host that has previously moved out there.

Consequently, the host route (/32) is propagated to the core via the DC-2 WAN edge with a more specific route, directing the traffic to its destination directly through DC-2.

Thanks to the ASA clustering this automatically maintains the current session stateful (7). Similar to the LISP IP Mobility previously described, when deploying the ASA cluster in conjunction with LISP IGP Assist, the path to the active DC where the application resides is optimized for all new sessions (6). The current sessions are maintained stateful through the original owner.

The drawback of this solution is that it injects a /32 host route to the L3 core. If the enterprise owns a block of IP address space or the usage of the L3 core is an intranet using a private IP network, this is straightforward. However, it might be challenging, if not impossible, with multiple ISPs to deal with a /32. Thus, in addition to the negotiation process, it may require a mechanism such as an AS path prepending or BGP conditional advertisement to identify the site where the host route is active and to redirect the traffic accordingly.

Conclusion

With the rapid growth of virtual machines and the easy mobility service that brings the abstraction of the OS from the physical host, it has now become usual that resources supporting the application happen on multiple locations that may be separated by long distances.

Therefore it turns out the IT manager is able to dynamically redirect in real-time the data user workflow in a very granular fashion to the site where the application resides.

In the meantime, the security stateful firewalls impose one-way symmetrical establishment with the return traffic, meaning that a “ping-pong” effect extending the workflow grows as the multi-tier application moves from one site to another.

In order to reduce the pointless latency from the hairpinning workflow as much as possible while achieving elasticity with the virtual DC through stateful live migration, we can leverage multiple network and security services to work in conjunction. Nevertheless, these solutions are not directly related. Hence, any one of them can be enabled for a specific use, for exampe if you think that reducing the latency has no impact on your application’s performance (e.g. metro distances).

What follows is a high-level summary of the different options with their respective added values:

Option A:

1 – ASA clustering spanned across multiple locations

• All ASA units are active.
• Sessions are maintained stateful with redirection via the CCL and without any artificial cheats such as source NAT.
• New sessions are treated locally.
• The maximum distance time-wise for the Cluster Control Link (CCL) is 20ms roundtrip (theoretically 2000km maximum between two sites[1]).
• All configurations are synchronized between all ASA units.
• Not limited to two sites.

2 – OTV for DCI LAN extension
(A long list. The following items constitute the most important ones only.)

• Failure domain containment
• Eliminate flooding of unknown unicast
• STP domain confined inside the physical data center
• Control plane independence
• Site independence
• Dual-homing with independent paths
• Reduced hairpinning
• Internal site behaves like any traditional switch with local lookup and forwarding
• FHRP isolation support
• Fast convergence
• Transport-agnostic
• IP, MPLS, Dark Fiber, etc.
• IP multicast transport or unicast-only
• ARP caching
• VLAN translation
• Control plane learning
• Native multi-homing (no STP)
• Load balancing
• Today: VLAN-based
• Future: flow-based

3 – LISP IP mobility for ingress path optimization

• For our purpose of a “hot” live migration across two sites, we leveraged LISP mobility in Extended Subnet Mode or ESM (LAN extension).
• LISP IP mobility can be enabled without LAN extension for “cold” migration – Across Subnet Mode or ASM.
• Provides dynamic redirections for ingress traffic toward the site where the active application resides.
• Requires an additional egress redirection such as FHRP localization.
• Offers host-based granular and dynamic signal notification.
• Doesn’t impose any configuration change on either the IP core or in the DNS service.
• Allows end-point migration to a new location without changing the assigned IP address.
• Detects dynamically any IP movements based on data plane events.
• Selective prefix allowed to roam.
• Works in conjunction with any type of LAN extension.
• Endpoint-agnostic
• No special agent running on the host required.

4 – FHRP isolation for egress path optimization

• Available for any network redundancy protocols (VRRP, HSRP).
• Offers same active default gateway in different locations.
• Optimizes the outbound traffic (server to client).
• Optimizes the intra-tier application traffic (server to server).

 Option B:

idem as above

idem as above

• Simple and easy configuration
• Host route-based notification
• Endpoint-agnostic
• No special agent running on the host required.
• Works in conjunction with any type of LAN extension.
• Can be deployed without LAN extension for “cold” migration.
• Requires a LISP Mapping database to signal the remote LISP FHR engines.
• Detects dynamically any IP movements based on data plane events.

4 – FHRP isolation
idem as above

Additional official guides (to list just a few):

• LISP Configuration Guide
• OTV Implementation Guide
• FHRP Isolation
• ASA Clustering and Inter-Site ASA Clustering
• Disaster Recovery as a Service (DRaaS)

Note: Since I wrote the following articles on ASA clustering stretched across multiple locations, additional improvements have been made to address some of the concerns listed in post 27.x. Please have a look at the ASA release-notes (especially 9.5(1) and 9.5(2)).

• 9.1(4) Geographically dispersed ASA cluster up to 10ms of Latency
• 9.2(1) Validated Spanned Interface mode (L2) North-South Insertion
• 9.3(2) Spanned Interface mode (L2) – East-West Insertion
• 9.5(1) Site Specific Identifier and MAC address
• 9.5(2) LISP Inspection for Inter-site Flow Mobility

Use the last configuration guide for the updated features, not discussed in this post

https://www.cisco.com/c/en/us/td/docs/security/asa/asa97/configuration/general/asa-97-general-config/ha-cluster.html#ID-2170-000001d3

Stateful Firewall devices and DCI challenges

Having dual sites or multiple sites in Active/Active mode aims to offer elasticity of resources available everywhere in different locations, just as with a single logical data center. This solution brings as well the business continuity with disaster avoidance. This is achieved by manually or dynamically moving the applications and software framework where resources are available. When “hot”-moving virtual machines from one DC to another, there are some important requirements to take into consideration:

• Maintain the active sessions stateful without any interruption for hot live migration purposes.
• Maintain the same level of security regardless the placement of the application
• Migrate the whole application tier (not just one single VM) and enable FHRP isolation on each side to provide local default gateway (which works in conjunction with the next bullet point)
• While maintaining the live migration, it can be crucial to optimise the workflow and reduce the hair-pining effect as much as we can since it adds latency.  As such, the distances between the sites as well as the network services used to optimize and secure the multi-tier application workflows amplify the impact of performances.

As with several other network and security services, the firewall is a stateful device that imposes a one-way symmetrical establishment. That means return traffic must hit the owner of the session, otherwise, the packet is dropped. Traditionally firewalls are deployed by a pair of devices in an Active/Standby manner, with dedicated layer 2 adjacency links to synchronize the states of all sessions and to probe the health of its peer. When the active firewall is stopped, the standby takes over, maintaining all active sessions stateful in a transparent manner for the application and for the end-user. As of today, most enterprises are deploying their perimeter firewalling in Active/Standby mode mainly for tightly coupled data center designs (metro distances using fiber links).

Figure 1:  Typical tighly-coupled DC deployment with firewalling. The primary DC-1 attracts all the traffic for the application of interest (best metrics). By default it is expected to maintain the session workflow within the same DC. 

As discussed in this previous high level post 13 – Network Service Localization and Path Optimization,  as a result of state failover of network services as well as application mobility, it is not rare to see 10 to 20 roundtrips between the two sites for the same active session. This is forced by all the stateful devices in the path imposing a one-way symmetrical establishment with the return traffic. This includes the security WAN edge, IPS, SSL offloader, SLB devices as well as the default gateways between application tiers, just to list the most common stateful devices.  Figure 2: The same application has moved to the secondary DC and a failover happened on the first firewall. NB: this is a basic design to keep the logic simple—usually additional stateful devices (SLB, SSL, IPS, WAAS, etc.) exist along the path, hence you can infer a longer final ping-pong effect.

If we consider that the signal propagation delay takes 1ms roundtrip to travel a 100km distance from each data center, 10 roundtrips bring almost 10ms between request and response for the same session, which might have a performance impact on the application. In the context of metro distances, it has been usually well accepted by network managers to work in “degraded” mode during maintenance windows, as these were fully controlled by the network and security organizations.

With the increased demand of virtual machines and dynamic workload mobility, it becomes challenging to control all of the component states and placement impacting the application workflow. Hence, the desire to control dynamically the optimum path to reach the application.

ASA Firewall clustering

Last year at Cisco Live in London, I presented a new concept based on firewall clustering to improve the DCI architecture; however, this enhanced solution was not yet supported due to some limitations with the ASA code (9.0) as well as the lack of testing.

Since v9.1(4) and recently version 9.2(1), the ASA clustering software has been improved to support long distances between members of a cluster (up to 10ms one-way latency) and several designs have been tested and qualified in DCI deployment scenarios. Thus, the excitement to post this article now :).

There are several detailed documents available on ASA clustering itself. Hence, for purposes of this post let’s focus only on the mechanisms that we can leverage in a DCI scenario. For further details on the ASA cluster, I recommend to read the configuration guide, which gives all details and explains the concept and nomenclature of ASA clustering. You will also find many great posts from others on the web.

Originally, ASA clustering aims to provide high-scale firewalling by stacking several physical ASA devices to form a single logical high-end firewall. All ASA devices are active and work in concert to pass connections as a single firewall.

To achieve this function, a “new” component called Cluster Control Link (CCL) was created to collapse all physical members of the ASA cluster together to form a single logical firewall. The CCL is used for the control plane, health-check, state sync, config sync as well to redirect the data plane traffic to the original owner of the session when needed. Remember, classical A/S or clustering A/A mode, it is mandatory that the return traffic hits the owner of the session, otherwise the packet will be dropped. With ASA clustering, this rule still applies, but instead of dropping an asymmetric flow, the firewall that owns the session is known by all other ASA devices, thus the packet is automatically redirected to the original owner via the CCL. The traffic is load-distributed in a clever fashion between the members of the ASA cluster.

There are two possible modes to load balance the data traffic from the upstream device across the ASA units:

• Individual Interface Mode (layer 3) using ECMP or PBR
• Spanned Ether-Channel Mode (layer 2) using LACP

Figure 3: Individual Interface Mode (left) versus Spanned Ether-Channel Mode (right) in a fully redundant deployment using Multi-chassis Ether-Channel (MEC).

In both modes, each ASA device is dual-homed using a virtual port-channel toward a Multi-chassis Ether-Channel engine (e.g. vPC) and the traffic from the upstream device is layer 3 load distributed (ECMP or PBR) for the Individual mode (left) or is layer 2 distributed (LACP) for the Spanned mode. It is not possible to mix different modes for the same cluster.

From a protocol point of view, a unique device can exist on each side of the LACP establishment between two entities.

Figure 4: A single logical device back to back for the LACP establishment.

Thus, in Spanned Mode, in order to form a logical LACP peer device, on the network side, the upstream pair of Nexus switches uses a virtual port-channel (vPC) and on the firewall side, the ASA cluster uses an enhanced LACP mode called cluster LACP (cLACP), which forms a virtual port-channel extended across all ASA units in the cluster, using the same IP address and the same virtual MAC address.

The ASA clustering deployment design is flexible. It can be deployed using a single port-channel to the same insertion point at the aggregation layer. As a result, both external (non-secured) and internal traffic (secured) traverse the same physical port-channel. The separation is achieved at layer 2 using VLAN tagging. The other method is to deploy the ASA cluster in sandwich mode between two Virtual Device Contexts (VDC) in order to physically separate external and internal traffic, each dedicated to an inside and outside port-channel as discussed in this post. This article relies on that method of sandwich mode providing solid hierarchy architecture.

Figure 5: Don’t be confused between the forwarding mode of the firewall and the interface distribution mode among the ASA cluster. The ASA cluster can be running in Routed Mode or Transparent Mode like most firewalls, and with or without multiple contexts. It is important to clarify which mode is supported. For example, if an enterprise wants the firewall to run in Transparent Mode, the only option to distribute the load among the ASA units is Spanned Ether-Channel Interface Mode.

Whichever load balancing protocol is used, there is no symmetrical algorithm that ensures return traffic re-uses the same path in reverse. Hence, it is not guaranteed to hit automatically the original owner of the session. The ASA cluster gets around this one-way symmetrical establishment by redirecting the session to its owner via the CCL.

Figure 6: TCP hand check establishment with redirection to the session owner over the CCL

• In the example above, a new session (TCP SYN) is established, hitting the ASA 2, which encodes SYN cookies with its own information (1) and then forwards the SYN packet to the next destination (2).
• When the TCP SYN-ACK is responded to, the traffic is load balanced based on the layer 2 or layer 3 source and destination identifiers and sent toward a new ASA; in our example the TCP SYN/ACK arrives at ASA 3.
• ASA 3 decodes the owner information from the SYN cookie and notices that ASA 2 is the owner (4) for that session.
• ASA 3 immediately forwards the packet to the owner unit ASA 2 over the CCL, which in turn forwards the SYN ACK via its inbound interface.

Figure 7: To increase resiliency, it is recommended to dual-home each data interface using a port-channel split between the two upstream switches. There is one port-channel per ASA device.

ASA Firewall clustering spanned across multiple sites

Now that we understand how ASA clustering is created, let’s see how to leverage the clustering mode for a DCI solution.

For our purposes and examples, an ASA cluster formed with four physical ASA units is stretched across two DCs, with two ASA members on each site. We will keep this scenario for the whole article, but nothing prevents us from adding more ASA units (up to 16 units per ASA cluster are currently supported) stretched across three or more DCs interconnected using LAN extension (we are using OTV for multi-site LAN extension).

ASA cluster Configuration

Some of the added value of deploying the ASA cluster is that configuration is synchronized between all units, hence there is no need to manually replicate all policies on each unit, thus avoiding risk of misconfigurations.

Extending the Cluster Control Link across 2 DC

The first component to extend between the DCs is the Cluster Control Link. This CCL connection must be fully resilient, as discussed previously, and extended across sites using a solid DCI LAN extension. Each ASA uses a dedicated port-channel split between the two vPC peers. From the upstream logical switch, the port-channels are not related except that they carry the same VLAN from ASA to ASA units across the two sites. From a security point of view, it is preferable to deploy it inside the secure perimeter.

Figure 8: CCL deployment, each ASA uses a dedicated port-channel split between the two vPC peers.

As discussed above, there are 2 mode to load distribute the data traffic among the ASA members: A layer 3 mode called Individual interface mode and a layer 2 mode called Spanned Etherchannel mode. Both modes are valid for DCI with some slight different added values for one mode versus the other that will be discussed has as we move forward.

Extending the Data plane using Individual Interface Mode

The ASA units are deployed in sandwich mode between the inside and outside routers and ECMP is used to load distribute the traffic across the local ASA members. The CCL as well as the data plane VLANs are extended between sites.

Figure 9: For the purposes of this topic, the CCL is represented using a logical straightforward link (orange) extended between the two DCs; however, in the final deployment it should be distributed in a sturdy and redundant fashion as described in Figure 8.

The application data traffic (layer 2) is isolated by a layer 3 hop. The CCL is also isolated from the data workflow. However, nothing prevents network managers from collapsing the CCL VLAN with the application data VLANs within the same overlay network as the segmentation is therefore maintained at layer 2 with dot1Q tagging.

From a logical layer 3 point of view, IGP adjacency is established between the inside and outside routers through the local ASA units, as well as between sites (higher cost across sites used for disaster recovery purposes).

Figure 10: The default gateway is unique and still active in DC-1. After the migration of the application, the traffic hits the original router, returns to DC-2 in regard to the communication with the application upper tier, to finally return to DC-1. The ping-pong effect starts after the move.

• In the scenario on the left, the primary DC-1 attracts the request for the application that exists in DC-1.
• The default gateway for the application is active in DC-1 and standby in DC-2.
• The ASA-1 (far left) becomes the owner of the session and routes the packet to the application.
• In the scenario on the right, the application has moved (hot stateful live migration) and continues to respond with no interruption.
• The return traffic hits the active default gateway in DC-1 which routes the packet toward the frontend server.
• The frontend server responds to end-user via its default gateway (DF) active on DC-1.
• The DG on DC-1 distributes the packet to the ASA-2.
• ASA-2 checks the owner of the session and redirects the packet to ASA-1 over the CCL.
• The workflow exits DC-1 toward the end-user.
• The session is maintained stateful with zero interruption

Beyond the ASA clustering, the concern is that, even if the tiers of the applications are all moved to the distant DC-2,  the routed communication between the tiers is established via the active default gateway still located in DC-1. Hence the traffic from the back-end to the front-end is hair-pinned via DC-1.

Consequently there is a the great interest for most of network managers to enable HSRP isolation to improve the server to server communication as shown below.

Figure 11: HSRP isolation reduces the hair pining, however the return traffic must still hit the owner of the session to maintain the establishment stateful.  

With FHRP isolation techniques, server-to-server communication is routed locally, eliminating the pointless latency (far right bottom). The outbound traffic from the frontend server toward the end-user is routed to the upstream local firewall (shortest path). However, as we want to maintain the session as stateful, the local ASA-3 redirects the traffic workflow to its original owner (ASA-1) via the CCL extension.

While machines migrate from one location to another, the session is maintained stateful with zero interruption. However, although the application has moved through DC-2, the next request will hit DC-1 until we inform manually or dynamically the layer 3 network about the move. This is discussed in Part 2, next.

The added value with Individual Interface Mode is that it maintains the layer 2 failure domain in isolation from the ASA control and data planes. Traffic is routed up and down the ASA cluster. The application data VLAN can be extended in a transparent fashion using any validated DCI solution. However, the other side of the coin is that the ASA unit cannot be the first hop default gateway for the application, as another layer 3 router separates it. Thus, Individual Interface mode might be challenging for enterprises that would like the firewall to be the default gateway of the application servers. Another point to mention about Individual Interface Mode is that it doesn’t support firewalls configured in Transparent Mode.

Extending the Data plane using Spanned Ether-channel Mode

In our cluster, all ASA members share the same IP address and can be the first hop default gateway for the application (not yet qualified though). However, for the last option we need to be very cautious with the layer 2 extension and the vMAC address.

Figure 12: The ASA Cluster LACP (cLACP) is spanned across the 2 data centers. The LACP established from the vPC peers is local. A layer 3 device isolates the ASA Spanned Ether-channel interface from the application data VLAN.

From each vPC peer, a local port-channel is established on each site between the local ASA units. From the ASA cluster, a single LACP port-channel is spanned across the two distant DCs (cLACP). cLACP imposes to span the same port-channel across the same ASA cluster, therefore the same vPC domain identifier must be identical on each vPC peer.

In regard to workflow, the behavior is the same as with the Individual Interface Mode. Note a slight difference in case of failure; convergence with LACP should happen faster than with ECMP or PBR.

Figure 13: The ASA Cluster is running in transparent mode, HSRP isolation is enable improving the workflow while the session is maintained stateful.

When the ASA cluster is running in routed mode, the members share the same IP address and same vMAC address. Consequently to avoid any duplicate MAC address to appear from different switch ports, a router on each site can be added (preferred method) to separate the layer 2 data traffic between the spanned ether-channel and the LAN extension. Indeed, if the VLAN attaching the front-end servers is L2-adjacent with the firewall, then the vPC peer will detect the same vMAC address bouncing from different interfaces (toward the ASA members and from the data LAN extension, which is definitely not an expected situation in Ethernet).

Figure 14: The same vMAC is learnt from both sides of each vPC peers. A challenging situation definitely not supported by the protocol Ethernet.

To prevent this situation with the same MAC address learnt on different sides, a layer 3 gateway is inserted to separate the L2 data traffic between the spanned ether-channel and the extended data VLAN. It also prevents layer 2 loop in case of a human design mistake.

Figure 15: a router inserted between the data VLAN and the spanned port-channel isolates the duplicate MAC address.

In the figure above, the design on the left shows the layer 3 separation between the spanned ether-channel VLAN 10 and the application data VLAN 20. Only VLAN 20 is extended. On the right side, the application servers are L2-adjacent with the inside interfaces of the ASA units via VLAN 10. As a result, the vPC peers on both sites learn the duplicate vMAC address from their respective ASA units and from the extended VLAN 10, which is not acceptable.

If you are willing to offer the default gateway from the firewall (not yet qualified though), it definitely requires filtering the vMAC address between sites as well as the ARP requests.

Figure 16: Careful: Please don’t get me wrong – currently this is neither recommended nor supported. Don’t do the following until you understand the exact ramifications 🙂 .

To filter the vMAC address with OTV, you will need to perform the following on the internal interface of each OTV edge device.

The following access list will block the vMAC 1111.2222.3333 shared between all ASA units:

In addition we need to apply a route-map on the OTV control plane to avoid communicating vMAC information to the remote OTV edge devices.

Indeed OTV uses the control plane to populate each remote OTV edge device with its local layer 2 MAC table. As this MAC table is built from its regular MAC learning, it is important that OTV doesn’t inform any remote OTV Edge device about the existence of this vMAC address as it exists on each site.

However a possible drawback of the Spanned mode is that all ASA are active and share the same IP and MAC addresses, hence an ARP request will hit all units that forms the ASA cluster and all members will reply with the same source (IP and vMAC). Regarding the local ASA, the reply passes along the unique local port-channel, which is fine, however it could be tricky if the reply comes from the remote site via the DCI link. Fortunately the vMAC of the remote ASA will be blocked as described with the previous access list. Therefore the ARP reply will come only from the local ASA, which is the desired behaviour. However to reduce the broadcast traffic, you may want to filter the ARP request destined to the default gateway, across the DCI connection.

Some additional recommendations

Just a slight note to clarify some VxLAN deployment for an hybrid network (Intra-DC).

As discussed in the previous post, with the software-based VxLAN, only one single VTEP L2 Gateway can be active for the same VxLAN instance.

This means that all end-systems connected to the VLAN concerned by a mapping with a particular VNID must be confined into the same leaf switch where the VTEP GW is attached. Other end-systems connected to the same VLAN but on different leaf switches isolated by the layer 3 fabric cannot communicate with the VTEP L2 GW. This may be a concern with hybrid network where servers supporting the same application are spread over multiple racks.

To allow bridging between VNID and VLAN, it implies that the L2 network domain is spanned between the active VTEP L2 Gateway and all servers of interest that share the same VLAN ID. Among other improvements, VxLAN is also aiming to contain the layer 2 failure domain to its smallest diameter, leveraging instead layer 3 for the transport, not necessarily both. Although it is certainly a bit antithetical to VxLAN purposes, nonetheless if all leafs are concerned by the same mapping of VNID to VLAN ID, it is feasible to extend the Layer 2 via the fabric using a layer 2 multi-pathing protocol, such as FabriPath.

In the following example, the server 4 attached to leaf 4 cannot communicate with the VTEP L2 GW located on leaf 1. As a result, VM-1 cannot communicate with server 4.

Fortunately the hardware solves this. The great added value of enabling the VTEP L2 gateway on the hardware switch (ToR) is that it is distributed and active on each leaf. Thus communication between VTEP on each switch is handled using the VxLAN tunnel. Hence, VNID 5000 can be bridged with VLAN 100 on leaf 4 and therefore VM-1 can communicate with server 4.

The other interesting added-value with the hardware-based anycast L2 gateway is the VLAN translation using the VLAN stitching, that can be useful for some migration purposes. Each leaf can map the same VNID with a different VLAN on its own side. In the following example VNID 5000 can be bridged with VLAN 100 on leaf 1 and VLAN 200 on leaf 6. Consequently, VLAN 100 and VLAN 200 share now the same broadcast domain.

If the software-based solution of VxLAN is a flexible solution in a fully virtualised environment, it is not always so well adapted to the hybrid network built with a mix of virtual and physical devices spread over unorganised racks.

Hope that clarifies the choice of VxLAN mode that you wish to deploy.

Achieving the high level of flexibility, resource availability, and transparency necessary for distributed cloud services DCI requires four components:

• Routing Network: The routing network offers the traditional interconnection between remote sites and gives end-users access to the services supported by the cloud. This component is improved using GSLB-based services such as DNS, HTTP redirection, dynamic host routes, and LISP.

• LAN Extension: The technical solution for extending the LAN between two or more sites using dedicated fibers or Layer 2 over Layer 3 transport mechanisms for long distances.

• Storage Services: The storage services used to extend access between SAN resources. SANs are highly sensitive to latency and therefore impose the maximum distances supported for the service cloud. It is preferable to use an Active/Active replication model to reduce the latency to its minimum value.

• Path Optimization Solution: The path optimization solution improves the server-to-server traffic as well as the ingress and the egress workflows.

Unlike the classical data center interconnection solutions required for geo-clusters that can be stretched over unlimited distances, DA and live migration for the service cloud require that active sessions remain stateful. As a result, maintaining full transparency and service continuity with negligible delay requires that the extension of the LAN and the SAN be contained within metro distances.

Enterprises and service providers may still have strong requirements to extend the LAN and SAN over very long distances, such as the need for operation cost containment or DP in stateless mode. These needs can be addressed if interrupting (even for a short period of time) and restarting sessions after workloads are migrated is acceptable to the system managers. Those requirements can be achieved using tools such as Site Recovery Manager from VMware© or an active-active storage solution such as EMC VPLEX Geo© for more generic system migration.

The ability to avoid disasters is improved by distributing physical compute and network resources between data centers that are geographically distributed over long distances. Geographic distribution provides higher elasticity and almost unlimited flexibility of the resources required to dynamically deploy VM loads.

The network side can transparently support distributed applications by extending Layer 2 between multiple sites. Yet by definition, the Layer 3 traffic carried between users and active applications through the cloud does not have native knowledge of the physical IP device locations, other than the network prefix given through the most significant bit-group of the IP address. The IP subnet is a logical visible subdivision of the network that is usually limited to the local network. It therefore delimits its broadcast domain defined by the system mask. The IP subnet is usually established by the enterprise or service provider network team. In general, an IP subnet addresses a set of IP equipment that belongs to the same VLAN.

Traditionally, if an IP subnet or a VLAN is associated with a physical location inside a data center, with the concept of interconnecting cloud resources, the broadcast domain is stretched over the distances that separate the data centers (DCI theoretically can be established up to unlimited distances with L2 over L3 transport). Therefore, the concept of location induced natively by the IP subnet subdivision loses one of its original functions of localization.

Thus, depending on the distance between the remote sites, the native routing mechanism can have an impact on performance for three major types of communication.

1. Traffic from the user to the server
2. Traffic from the server to the user
3. Traffic from server to server (such as in a multi-tier application)

Modern firewalls, load balancers, and most stateful devices support the concept of virtual context, which is the ability to support multiple virtual firewalls or virtual load balancers. Up to 250 virtual contexts, fully autonomous and isolated from each other, can be enabled on a single physical appliance or service module.

To offer the high availability service required for business continuity, firewalls and load balancers work by pairing physical devices. Both devices remain active while the virtual contexts run in an active/standby fashion. In addition to providing redundancy, this mechanism distributes the active context between the two devices, improving the total throughput for active workflows.

When interconnecting multiple data centers and deploying firewalls and other stateful devices such as load balancers, the distance between remote sites is an important consideration.

When data center sites are deployed in close proximity (such as within the few kilometers that is typical for large campus deployments), they can be considered a single, physically-stretched data center location. Under these premises, it would probably be acceptable to deploy the stateful devices in a stretched fashion, with one physical member of the HA pair in each data center site. For deployments where the distance between locations is farther, a pair of HA devices is typically deployed in each physical data center.

There are some important reasons to maintain each active/standby pair within the same data center. Redundancy is controlled by two devices[1], which means that dispersing the active/standby contexts in two different locations would limit the maximum number of data centers to two.  On the other hand, keeping the active/standby pair of network services inside the same physical data center, allows replicating the same security policies in more than two data centers.

In addition, the link between the physical devices used for health check and process synchronization (replication of the active flows for stateful failover) must be extended in a very solid fashion. Due to its function of fault tolerance, it is also very sensitive to latency.

Last but not least, security and optimization functions usually require maintaining a stateful session. Therefore, for the same session, the traffic should be returned to the original virtual context that acknowledged the first flow, otherwise the flow will be dropped.

This behavior of symmetrical paths should be controlled and maintained, especially with the migration of VMs over a LAN extension as explained in the next topics.

Cisco has recently introduced a new feature called OTV that extends Layer 2 traffic between multiple sites over a Layer 3 network. The edge devices that interconnect data centers are known as OTV edge devices.

OTV dynamically encapsulates Layer 2 packets into an IP header for the traffic sent to the remote data centers. Routing Layer 2 traffic on top of a Layer 3 network is known as “MAC routing” transport. MAC Routing leverages the use of a control protocol to propagate MAC address reachability information, this is in contrast with the traditional data plane learning done in technologies like VPLS.

In the next example, MAC 1 sends a Layer 2 frame to destination MAC 2. On the MAC table of the OTV edge device (DC1), MAC 1 is a local address (Eth1), while the destination MAC 2 belongs to a remote location reachable via the IP address B (remote OTV Edge device).

The local OTV-ED encapsulates the Layer 2 frame using an IP header with as IP destination “IP B”. The remote OTV-ED removes the IP header and forwards the frames to its internal interface (Eth 5). Local Layer 2 traffic is treated like any classical Ethernet switch (i.e. MAC 2 <=> MAC 3 on DC2).

A control plane protocol is used to exchange MAC reachability information between network devices, extending the VLANs between the remote sites while the learning process inside the data center is performed as in any traditional Layer 2 switch. This mechanism of advertisement destined to the remote OTV edge device differs fundamentally from classical Layer 2 switches, which traditionally leverage the data plane learning mechanism based on L2 source MAC address discovery: if the destination address is unknown after a MAC lookup on the MAC table, the traffic is flooded everywhere.

With OTV, the process for learning MAC addresses is performed by advertising the local MAC tables to all remote OTV edge devices. Consequently, if a destination MAC address is not known, the packet destined to the remote data center is dropped.

This technical innovation has the advantage of removing the risk of broadcasting unknown Unicast addresses from one site to another. This technique is based on a routing protocol, and provides a very stable and efficient mechanism of MAC address learning and Layer 2 extension while maintaining the failure domain inside each data center.

While OTV natively maintains the STP and the failure domain within each local data center, it provides the ability to deploy multiple OTV edge switches in the same data center in active mode. This function is known as Multi-Homing.

OTV works across any type of transport (Fiber, TCP/IP, MPLS) extended between the remote sites with the reliability and effectiveness of the Layer 3 protocol.

In addition to these primary functions, which are essential for the cloud networking, OTV offers several very important innovations:

OTV connects two or more sites to form a single virtual data center (Distributed Virtual Data Center). No circuit states are required between the remote sites to establish the remote connection. Each site and each link are independent and maintain an active state. This is known as “Point to Cloud” service, which allows a data center to be securely attached or removed at any time without configuring the remote sites and without disturbing cloud services.

OTV offers a native multicast traffic optimization function between all remote sites. OTV is currently available on Cisco Nexus 7000 Series Switches and the Cisco ASR 1000 Series Aggregation Services Routers.

Ethernet Virtual Connection (EVC) is a Cisco carrier Ethernet equipment function dedicated to service providers and large enterprises. It provides a fine granularity to select and treat the inbound workflows known as service instances, under the same or different ports, based on flexible frame matching.

Ethernet Virtual Connection

One of benefits of EVC is the ability to address independent Ethernet encapsulation on the same physical interface with a mix of different services such as dot1q trunk, dot1q tunneling, EoMPLS Xconnect, VPLS attachment circuit, routing, and Integrated Routing and Bridging (IRB). Each service instance can match a unique identifier or a range of identifiers (i.e. a VLAN tag deployed mainly in the context of DCI).

Another important feature is the ability to aggregate multiple service instances into the same transport virtual instance. For example, EVC can multiplex multiple VLANs into the same Bridge Domain (BD) connected to a Virtual Forwarding Instance (VFI) of a VPLS.

Multiplexing Multiple VLANS

Another important element for a multi-tenancy environment is the ability for each service instance to change the VLAN tag using a new identifier, which allows dynamic VLAN ID translation.

In addition, the flexibility of the service mapping of EVC allows improving the scalability in terms of service instances, VLAN per Bridge Domain or per EoMPLS Xconnect or per VFI.

Under the same or different physical interface, multiple service instances can be mapped into the same bridge-domain to provide L2 local bridging between physical interfaces and leverage the usage of VPLS to bridge L2 frames across the MPLS core.

If we leverage the concept of EVC in the public cloud context, a customer service is an EVC. This EVC is identified by the encapsulation of the customer VLANs within an Ethernet island or Bridge Domain, and is identified by a globally unique service ID. A customer service can be point-to-point or multipoint-to-multipoint.

The next drawing shows two customer services: Service Green is point to point; Service Red is multipoint to multipoint.

The diversity of services required in a cloud computing environment and the constraints related to the type of applications moving over the extended network require a set of diversified DCI solutions. Cisco offers three groups of technical solutions that meet these criteria:

Point-to-Point Interconnections: For point-to-point interconnections between two sites using a dedicated fiber or a protected dense wavelength-division multiplexing (DWDM⁸)mode, Cisco offers Multi-Chassis Ether Channel (MEC) solutions that allow multiple physical links of a Port Channel to be distributed over two different chassis. MEC is available through two approaches:

• A single control plane managing the two chassis: This method is available on the Catalyst 6500 series with the function of Virtual Switching System (VSS).
• An independent control plane: This option is available on Cisco Nexus 5000 and Cisco Nexus 7000 Series switches with the function of a virtual Port-Channel (vPC).

These options can provide active physical link and edge device redundancy to ensure the continuity of traffic between the remote sites, for type 1 and type 2 faults. Both approaches eliminate the use of the Spanning Tree protocol to control the loops. In addition, the MEC solution improves bandwidth utilization.

MEC Solution

Multiple Site Interconnections: For multi-site interconnections using optical links or using a DWDM service running in protected mode, FabricPath (TRILL) can quickly and seamlessly connect multiple remote sites in a fabric fashion, remove the extension of the Spanning Tree Protocol between remote data centers, and offer huge scalability compared to classical Ethernet. FabricPath is available on the Cisco Nexus 7000 Series Switches, with upcoming availability on Cisco Nexus 5500 Series⁹ Switches. FabricPath can also be used in a point-to-point model, which supports tying additional data centers into the cloud without impacting the production network or affecting existing connections.

Security: Traffic sent through the DCI Layer 2 extension can also be encrypted between a Cisco Nexus 7000 Series Switch deployed at the network edge using the Cisco feature called TrustSec (CTS). With CTS, encryption is performed by the hardware at line rate without impacting the performance or the latency of the traffic crossing the inter-site network. CTS offers a rich set of security services including the confidentiality of data transmitted over the WAN via a standard encryption mechanism (802.1AE).

Layer 2 switching over the WAN or the metro network, whether it is a native Ethernet frame format or a Layer 2 over TCP/IP over any type of transport, should not add any latency to that imposed by the physical distance between sites (mostly dictated by the speed of light). Switching technologies used to extend the Layer 2 over Layer 3 (L2oL3)and obviously the native Layer 2 protocol must be computed by the hardware (ASIC) to achieve line rate transport. The type of LAN extension and the choice of distance are imposed by the maximum latency supported by HA cluster framework and the virtual mobility.

It is crucial to consider whether the number of sites to be connected is two or more than two. Technologies used to interconnect two data centers in a back-to-back or point-to-point fashion are often simpler to deploy and to maintain. These technologies usually differ from more complex multipoint solutions that provide interconnections for multiple sites.

The drawback to simple point-to-point technology is reduced scalability and flexibility. This is a serious disadvantage for cloud computing applications, in which supporting an increasing number of resources in geographically distributed remote sites is critical.

Therefore, enterprises and service providers should consider possible future expansion of cloud networking and the need to operate without disruption when considering Layer 2 Extension technologies. Solutions for interconnecting multiple sites should offer the same simplicity as interconnecting two sites, with transparent impact for the entire site. Dynamically adding or removing one or several resource sites in an autonomous fashion is often referenced as “Point to Cloud DCI”. The network manager should be able to seamlessly insert or remove a new data center or a segment⁷of a new data center in the cloud to provide additional compute resources without modifying the existing interconnections and regardless of the status of the other remote sites.

Whatever DCI technology solution is chosen to extend the Layer 2 VLANs between remote sites, the network transport over the WAN must also provide secure ingress and egress access into those data centers.

When extending Layer 2, a number of rules must be applied to improve the reliability and effectiveness of distributed cloud networking:

• The spanning tree domain should not be extended beyond a local data center, although all the links and switches that provide Layer 2 extension must be fully redundant.
• The broadcast traffic must be controlled and limited by the edge devices to avoid the risk of polluting remote sites and should not have any performance impact.
• All existing paths between the data centers must be forwarded and intra-cloud networking traffic should be optimized to better control bandwidth and latency.
• Some workflows are more sensitive than others. Therefore, when possible, diverse path should be enabled for some services such as heartbeat, used by clusters, or for specific applications such as management or monitoring.
• The Layer 3 services may not be able to natively locate the final physical position of a VM that has migrated from one host to another. This normal behavior of routed traffic may not be efficient when the Layer 2 network (Broadcast Domain) is extended over a long distance and hosts are spread over different locations. Therefore, the traffic to and from the default gateway must be controlled and restricted onto each local data center when appropriated. Similarly, the incoming traffic should be redirected dynamically on the physical site where virtualized applications have been activated.
• For long-distance inter-site communication, mechanisms to protect the links must be enabled and rapid convergence algorithms must be provided to keep the transport as transparent as possible.
• The VLANs to be extended must have been previously identified by the server and network team. Extending all possible VLANs may consume excessive hardware resources and increase the risk of failures. However, the DCI solution for LAN extension must provide the flexibility to dynamically remove or add any elected VLAN on demand without disrupting production traffic.
• Multicast traffic should be optimized, especially in a cloud architecture made up of geographically dispersed resources.

When a failover occurs in an HA cluster, the software components have to be restarted on the standby node. Assuming the storage has been replicated to the remote location using synchronous or asynchronous mode⁵, the standby node can continue to handle the application safely.

For cloud computing, it is usually necessary to keep the session stateful during and after the migration of a VM. When stateful session is necessary during the movement of the system, the distance between the two physical hosts is driven by the maximum latency supported, the synchronous replications for data mirroring, and the services associated with the storage replication. This means a maximum distance of 100 km between two hosts should not be exceeded when using an Active/Active storage mode requiring synchronous data replication engines. It is important to note that this distance drops to 40-50 km when using the default shared storage mode.

The elasticity of cloud computing is also driven by the requirement of the active sessions to be maintained with no interruption of service, therefore live migration services in real time are limited to metro distances due to the synchronous mirroring (zero RPO). Beyond metro distances⁶and using current storage tools, the whole cloud computing solution becomes a DRservice. Therefore service functions such as Site Recover Manager (SRM®) are very efficient and built for that purpose, but in a stateless session mode.

 Four Components of DCI

Cisco addresses DCI requirements with a complete set of network and storage architecture solutions. The Cisco solution is based on the four components that are critical to providing transport elasticity, flexibility, transparency, and resiliency between two or more sites:

• Layer 2 extension

• SAN extension

• Routing interconnection between sites

• IP Localization for Ingress, Egress and Server to Server workflows