8 – Extended Layer 2 over Layer 3 (L2 over L3) – MPLS


For point-to-point networks across very long distances, Ethernet over Multiprotocol Label Switching (EoMPLS) Pseudowire can be useful. The EoMPLS service is supported natively on Cisco Catalyst 6500 Series Switches with the Sup720 and Sup2T cards. In conjunction with the VSS function (clustered switches), the resiliency of the L2 VPN service can be easily improved for a DCI LAN extension. VSS provides a fully-redundant physical solution that enables a logical L2 over L3 link (Pseudowire) flawlessly and without the need to activate the Spanning Tree protocol between the remote sites. EoMPLS is also supported on Cisco ASR1000 Series Routers. L2 over L3 extends the Layer 2 Pseudowire over unlimited distances.

With an additional SIP or ES+ card on the Catalyst 6500, the EoMPLS function can be encapsulated directly into a GRE tunnel. This gives the option to extend the Layer 2 VPN over a pure IP network. In this case, the technical knowledge and experience required for an MPLS environment is no longer imposed. In addition, the GRE tunnel may be encrypted using the standard point-to-point encapsulation method of IPSec.

For Multiple data center interconnections, Cisco offers two technologies to address the requirements of cloud computing:

  • VPLS
  • OTV

Virtual Private LAN Service (VPLS) is available via two approaches:

  • A-VPLS: Advanced VPLS (A-VPLS) is designed for enterprise environments. A-VPLS is available on Cisco Catalyst 6500 Series Switches using a SIP-400 or ES+ WAN card10. This option takes advantage of the system virtualization capabilities of the Catalyst 6500 VSS so that all physical links and edge switches are redundant and active without extending the Spanning Tree protocol between sites. A-VPLS has been specifically designed to offer simplicity of implementation with all the features and performance of the MPLS transport protocol. This feature can also be implemented on a pure IP core network via a GRE tunnel.
  • H-VPLS: H-VPLS is designed for service provider environments, in which very high capacity interconnections and segmentation are required. It can process information in very large private, public and hybrid cloud environments with large numbers of multi-tenants. H-VPLS is available with the Cisco ASR9000 Series Routers.Chassis Link Aggregation Group (MC-LAG)MC-LAG enables downstream devices to dual-home one or more bundles of links using the Link Aggregation Control Protocol (LACP) 802.3ad in an active/standby redundancy mode, so the standby takes over immediately if the active link(s)fails. The dual-homed access device operates as if it is connected to a single virtual device. MC- LAG is usually enabled on the provider edge (PE) device.Cisco Router 7600 Series Routers and the Cisco ASR9000 Series Aggregation Services Routers support this feature. With MC-LAG, the two routers function as Point of Attachment (POA) nodes and run an Inter-Chassis Communication Protocol (ICCP) to synchronize state and to form a Redundancy Group (RG). Each device controls the state of its MC-LAG peer for a particular link-bundle; one POA is active for a bundle of links while the other POA is a standby. Multiple active link bundles per chassis are supported11.MC-LAG can work in conjunction with L2VPN such as VPLS, but other network transport services such as EoMPLS, L3VPN or QoS can be leveraged as well.

    A use case in the context of DCI LAN extension is that at the edge of a provider’s network, each customer edge (CE) device supporting LACP is dual-homed to two provider edge (PE) devices and distributes the load on a VLAN-based hashing mechanism onto multiple link bundles. Then the MC-LAG device bridges and extends the concerned VLANs over an MPLS core using a VPLS Pseudowire. The MEC function can be enabled on the aggregation layer to improve Layer 2 multipathing intra-data center, so all Layer 2 uplinks from the access layer to the aggregation layers are forwarded.

    MC-LAG offers rapid recovery times in case of a link or node failure, while VPLS addresses the traditional fast convergence, fast reroute and path diversity features supported by MPLS.

    MEC and MC-LAG Function


This entry was posted in DCI. Bookmark the permalink.

2 Responses to 8 – Extended Layer 2 over Layer 3 (L2 over L3) – MPLS

  1. santsboy says:

    Hi Yves,

    for a VPLS network, we are forced to use asynchronous storage replication, right? so this means that with VPLS, if we do a hot move the user will notice a downtime?




    • Yves says:

      Hi Santsboy

      No, you are not forced to use asynchronous data replication with VPLS. The replication mode is related to the latency, dictated itself by the distance, as well as the bandwidth.
      VPLS encapsulation and forwarding, like OTV, is usually performed by hardware thus there is no impact in term of latency from the packet encapsulation point of view.
      PS: If not performed by hardware, then forget about using it for DCI. You may want to use VPLS for metro distances across fiber links because you have more than 2 sites to interconnect and/or you may want to use Layer 3 fast convergence and/or path diversity and/or traffic engineering etc.. but it doesn’t mean that you have to carry your storage traffic over VPLS. You can if you need, but you may have different options.

      Just a reminder: VPLS is used to extend the VLAN’s of interests between multiple sites.
      The storage uses a different links on a different side for different purposes. Usually for synchronous storage replication we use FC or FCoE over dark fibres, but it could be FCIP for a L3 core/MAN (still dictated by the distance as long as you L3 is line rate), thus you don’t necessarily need VPLS to carry your FCIP traffic, if you can use the L3 network transport.

      That said, assuming that you run FCIP over your L2 extension (e.g. VPLS), or assuming that the network distances are the same as the storage distances (e.g. FC over fiber), whichever the L2VPN protocol you want to use, then you can assume that up to 100 km you can replicate your storage data in synchronous mode. Actually it depends also on the application and the maximum latency it supports, some may accept 200km, but usually we say 100km max for sync data replication (on the storage side). The main reason is that you need to wait for the write acknowledgement from the remote volume before you can treat the next I/O, impacting the TPS in regard to the distance it takes to ack the write cmd from the remote volume and thus the application performances.

      For synchronous replication the SCSI protocol (FC/FCoE/FCIP..etc) takes two round trips
      A ==> B = Ready to receive ?
      B ==> A = Yes (A waits for response)
      A ==> B = sends write cmd
      B ==> A = Write Ack (A waits for ack)
      For each write cmd each round trip is about 10 μs per kilometre, which translates this into 20μs/km for 2 round trips for Synch data replication.
      Meaning that in synch data replication it takes 1ms to write the data across 50km and afterward to treat the next I/O.
      Thus it takes 2ms for 100km in synchronous mode versus 1 ms for 100km in asynchronous mode (only 1 round-trip).
      PS: Note that this is the theoretical math, however you can use IO acceleration on some storage fabric to reduce the latency by 2.

      Thus back to your second comment, “if we do a hot move, the user will notice a downtime”, theoretically from a network access point of view (user ==> Apps), the answer is no for metro distances (data synchronous replication is limited to metro distances). For very long distance (asynchronous data replication), the user may notice some delays due to network hair-pining traffic (thus the need to optimise the ingress traffic using LISP mobility or AGP assist).
      Having said that, although the user may not notice any downtime, the application may suffer from performance degradations due to shared storage (the Virtual machine moves to a remote location while the volume keeps local on its original location).
      Check this note for more explanation http://yves-louis.com/DCI/?p=256

      PS: It’s difficult to give you the full picture in few sentences, there are so many options that you can enable to optimise the latency on the storage side.
      It also depends if your storage is SAN (FC, FCoE, FCIP) or NAS (IP). But just think about the distance being the most critical criteria to decide if you can go with Synch data replication or not (Asynchronous replication can be initiate across unlimited distances).

      Hope that helps a bit…


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.