21 – Data Center Interconnect – summary

Achieving the high level of flexibility, resource availability, and transparency necessary for distributed cloud services DCI requires four components:

  • Routing Network: The routing network offers the traditional interconnection between remote sites and gives end-users access to the services supported by the cloud. This component is improved using GSLB-based services such as DNS, HTTP redirection, dynamic host routes, and LISP.
  • LAN Extension: The technical solution for extending the LAN between two or more sites using dedicated fibers or Layer 2 over Layer 3 transport mechanisms for long distances.
  • Storage Services: The storage services used to extend access between SAN resources. SANs are highly sensitive to latency and therefore impose the maximum distances supported for the service cloud. It is preferable to use an Active/Active replication model to reduce the latency to its minimum value.
  • Path Optimization Solution: The path optimization solution improves the server-to-server traffic as well as the ingress and the egress workflows.

Unlike the classical data center interconnection solutions required for geo-clusters that can be stretched over unlimited distances, DA and live migration for the service cloud require that active sessions remain stateful. As a result, maintaining full transparency and service continuity with negligible delay requires that the extension of the LAN and the SAN be contained within metro distances.

Enterprises and service providers may still have strong requirements to extend the LAN and SAN over very long distances, such as the need for operation cost containment or DP in stateless mode. These needs can be addressed if interrupting (even for a short period of time) and restarting sessions after workloads are migrated is acceptable to the system managers. Those requirements can be achieved using tools such as Site Recovery Manager from VMware© or an active-active storage solution such as EMC VPLEX Geo© for more generic system migration.

This entry was posted in DCI. Bookmark the permalink.

5 Responses to 21 – Data Center Interconnect – summary

  1. santsboy says:


    in the following cases which will be the most appropriate solution and why?
    – Dark Fiber Layer 2 DCI: vPC or OTV?
    – Dark Fiber Layer 3 DCI: 1 layer 3 connection using OTV or 1 layer 3 connection and another layer 2 connection using vPC?

    Thank you.


    • Yves says:

      Hi Santsboy,

      I assume you are talking about a twin-DC (2 DC in point-to-point fashion) as you mentioned vPC.
      The preferred answer goes with OTV if you have this option, and keep vPC as a possible option for DCI if you can’t deploy OTV for any reasons.

      If actually both solutions are validated design for DCI, it is important to understand the pros&cons.
      Thus let me try to argument a bit my answer.

      It is important to understand that, if vPC has been diverted to offer a nice DCI solution, OTV has been built from the ground up to address natively all DCI requirements – Idem for E-VPN BTW.

      When you need to deploy a DCI solution, there are several requirements that need to be considered.
      Let me try to build a list of DCI requirements.

      Failure domain: By default when using vPC, the failure domain is spanned across the two DC. The MAC learning process is performed through the Data plane, in case of unknown destination the vPC layer needs to flood the request to all possible destinations, including the remote site(s). This is one major cause of broadcast storm that can disrupt both DC at the same time, which is what you definitely want to avoid. Consequently with vPC to be more secure and efficient for DCI, you need to rate limit the broadcast of UU. However make sure to understand the current average of Broadcast and Multicast usage prior to set the threshold for the rate limiters.
      OTV uses a concept of MAC routing to exchange the MAC addresses reachability via the control plane. In short each OTV edge device populates locally its CAM table like any standard STP switch and distributes this information to all distant OTV ED’s. When the L2 destination is unknown, the L2 frame is not sent through he overlay (like any Layer 3 protocol, drop when you don’t know the destination, which makes layer 3 very robust)

      STP or not STP: Another aspect of vPC which is not related to the previous one, is that you need to filter the STP BPDU to isolate the STP domain on each site. Why ? vPC provides by nature a dual-homing solution fully resilient with all links forwarding w/o control of STP. For intra-DC we strongly recommend to keep STP running in background in case of for example a cabling misconfiguration (it’s NOT difficult to build a loop inside a DC). Back to DCI and contrary to the intra-DC, we tend to “assume” that we cannot easily create a layer 2 back-door extension between the two sites (well, to be discussed later!), thus, with that assumption we can avoid to run STP over the DC interconnection. The reason is that you can have only one root bridge per STP domain which usually sits at the aggregation layer, one vPC peer is the root bridge, the other vPC peer is the secondary root bridge. One fails the other takes over. If you extend your STP domain between the 2 sites, it becomes challenging to elect the primary and secondary root bridge as now you have 4 options. That’s without counting the TC from one side impacting the forwarding state of remote links. So you need to filter the BPDU. But is a layer 2 backdoor link definitely impossible ?

      Multi-homing vPC offers natively a dual-homing solution (maximum 2 vPC peer members) for DCI which is enough for most of DCI cases. But it doesn’t address the L2 back-door with STP isolation. OTV natively supports multi-homing using the site VLAN. A L2 back-door is automatically detected although BPDU are not sent through the OTV overlay network, and new election of authoritative device is performed automatically. You can have 4 or more OTV edge devices and distribute the load between all the devices on a per-VLAN basis.

      Dedicated L3 links or L3 over L2 links ? With vPC currently you still need dedicated parallel links for dynamic Layer 3 adjacency. If you use distinct L3 subnets on each site, you can deploy some tricky static routing over vPC, this works fine. OTV allows you to run a dynamic routing protocol on top of the OTV overlay. What you cannot do with the current release is to route and encapsulate in IP the same VLAN at the same time. Thus the option to dedicate a virtual device context for the OTV device. If the VLAN is routed by another device (FW) or the upward core layer, then you don’t need to dedicate a VDC.

      Is L3 transport required to run OTV over dark fiber? What you need is just an IP address for each OTV device. In a twin-DC design (pt-2-pt) an IP MC transport is not needed.

      Fast convergence Today OTV offers sub-second fast convergences for standard failures and sub-5 secs for extreme failures.

      Hope that helps


      • santsboy says:

        Thank you very much Yves for this clear explanation, now I see the picture. BTW you were right I was talking about Twin-DCs.

        Just a quick question. When you say: “In a twin-DC design (pt-2-pt) an IP MC transport is not needed.”

        you mean in a twin-DC layer 3 link on the DCI is not needed?

        Thanks a lot for the help.


        • Yves says:

          What I meant was that with dark fiber for a twin-DC, in pt-to-pt model, you don’t necessarily need a layer 3 routing backbone to enable OTV. You just need an IP identifier on each OTV edge device. If you have only 2 sites, the transport doesn’t need to be multicast enabled for the OTV edge devices to join an ASM group. You can if you wish, but I’m not sure it brings a lot of added-values.
          PS: I’m talking about the transport of OTV overlay, I’m not considering any usage of IP routing or IP multicast for other apps purposes.

          regards, yves

          • santsboy says:

            Thanks Yves for the clarification.

            Much appreciate it.



Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.