27 – Bis-Bis – Stateful Firewall devices and DCI challenges – Part 1 (cont)

Back to the recent comments on  what is “officially” supported or not ?

First of all, let’s review the different Firewall forwarding mode officially supported

ASA cluster  deployed inside a single data center:

Firewall forwarding mode within  single DCFig.1 Firewall forwarding mode within a single DC. Please note the firewall routed mode supported with the Layer 2 load balancing (LACP) Spanned Interface mode.

When configured in Routed mode (e.g. default gateway for the machines), the same ASA identifiers IP/MAC are distributed among all ASA members of the cluster. When the ASA cluster is stretched across different locations, the Layer 2 distribution mechanism facing the ASA devices is achieved locally using pair of switches (usually leveraged the a Multi-chassis EthernetChannel technique such as VSS or vPC).

Subsequently the same virtual MAC address (ASA vMAC) of the ASA cluster is duplicated on both sites and as the result it hits the upward switch from different interfaces.











Fig.2 ASA and duplicate vMAC address

Therefore when the ASA cluster runs the firewall routed mode with Spanned interface method, in a geographically stretched fashion, it breaks the Ethernet rules due to the duplicate MAC address, with risks of affecting the whole network operation. Consequently by default it is not supported as expressed in the release notes for the Inter-site section.

Firewall forwarding mode with dual DCFig.3 The routed mode is not supported with Spanned Interface mode for ASA cluster stretched across multiple locations.

The above requires a clarification. In this context, “NOT supported” means that no other design alternative allowing the firewall routed mode to be enabled in conjunction with Spanned interface mode has been tested by the Quality Assurance (QA) process.

However it doesn’t mean that there is no design workaround to address this requirement, and some additional network services may help to keep the ASA in Routed mode.

  • The 1st option could be to filter the Virtual MAC address in the interface facing the DCI link.

This option may be desired if you need to use the ASA as the default gateway for the servers for example. However it must be enabled with caution, thus this option is definitely not recommended, except if you know exactly what you are doing.

The first drawback to keep in mind is that if the vMAC address is blocked, it results that the original ASA member cannot maintain the session stateful with a virtual machine  moving to the remote site. Consequently this option will be limited to “cold migration” using subnet extension (e.g. for cost containment or migration purposes).

The second concern is that to address any black hole scenarios, this option imposes as well the ingress traffic to be dynamically redirected to the new location as soon as the application of interest has migrated to (e.g. LISP mobility).

Statefull session is not supported with vMAC filter

Fig.4 Statefull session is not supported with vMAC filtering

  • A 2nd alternative more robust IMO is to isolate the layer 2 domain with a Router

With this option, the L2 domain used to distribute the traffic among the 2 local ASA devices is isolated from the data VLAN, hence the pair of switches facing the ASA members sees the vMAC only from the direct attached Firewalls. Only the data VLAN is stretched across the 2 locations (plus the CCL which is not represented in this diagram)

Router insertion

Fig.5 Router insertion to isolated the vMAC. ASA are configured in firewall routed mode with static routing.

However there are few disadvantages of using this design with the default gateway that resides between the application servers and the ASA clusters:

  1. The ASA cannot be the default gateway of the servers due to the inserted first hop router.
  2. The routing service on the ASA must be configured with static entries. Indeed, the dynamic routing is centralized to the master, hence routing adjacency on non-master site without a full stretched subnet will not succeed. On the other hand, if the subnet is stretched, we lose the ability to localize transit traffic to the local site.
  3. It becomes challenging to secure the E-W traffic (e.g. Web-tier <=> App-tier <=> DB) with the same ASA cluster.

Note for the last bullet that IPS are usually more efficient (and mandatory) between application tiers than a classical firewall function However the ASA offers embedded IPS too, thus this comment I think may be important.

If theoretically the second alternative works as I described it, it is not yet officially supported, and it will require several tests with different scenarios that you may want to run to this validate this option for your environment.

Having said that, a new East-West insertion mode has been qualified and validated with the recent release 9.3(2), technically nothing new, it ‘s just that this one has been deeply tested. In this scenario the E-W traffic can nowbe secured via the same ASA cluster.

ASA in transparent mode.

Fig.7 ASA in transparent mode. East-West traffic is secured using the ASA cluster.

When the ASA is configured in Firewall Transparent mode, the virtual MAC is not an issue anymore as the ASA members use a BVI address. The BVI address doesn’t appear throughout the data plane. Consequently there is no risk of duplicate MAC address. As the result, the outside router can act as a default gateway for all application servers of interest, allowing the E-W traffic (server to server communication) to be secured via the same ASA cluster. Keep in mind that it requires FHRP isolation to support hot live migration.

Additional details on E-W insertion design can be found in this release-notes 9.3(2)

I hope that post clarifies some statements within the official ASA document and alternative design.




This entry was posted in DCI. Bookmark the permalink.

4 Responses to 27 – Bis-Bis – Stateful Firewall devices and DCI challenges – Part 1 (cont)

  1. Thanks for this clear article. I really like the figures.
    I have a few comments:

    A)”The 1st option could be to filter the Virtual MAC address in the interface facing the DCI link.”
    In this scenario, I don’t see why vMtions are not possible. You say:
    “The first drawback to keep in mind is that if the vMAC address is blocked, it results that the original ASA member cannot maintain the session stateful with a virtual machine moving to the remote site.”
    A few lines above, “When configured in Routed mode (e.g. default gateway for the machines), the same ASA identifiers IP/MAC are distributed among all ASA members of the cluster.”
    First off, I suppose you mean that the vMAC is filtered , not .
    If a VM is vmotionned to the second DC, its outbound traffic will hit one of the redundant FWs in its new site; if we assume that this VM has communicated with the outside before being moved, both of them are just forwarders for the traffic coming from that VM; so the packet will be passed to the owner through the CCL in the first DC.
    Where is the issue? The traffic path is not optimal, but nothing more, or am I missing something?
    Also, another option to avoid inbound traffic black holing for that VM is RHI/RISE.

    B) “A 2nd alternative more robust IMO is to isolate the layer 2 domain with a Router”.
    1. and 2. In this scenario, the FW routed mode has no use and should be replaced with a transparent mode.
    3. ASAv could be use here, 1 / tenant or 1 for the entire site, depending on the requirements and the amount of E/W traffic.

    • Yves says:

      Hi Jean-Christophe,

      A) let me rephrase and reorder a bit the sentences 🙂
      First of all: “When configured in Routed mode (e.g. default gateway for the machines), the same ASA identifiers IP/MAC are distributed among all ASA members of the cluster.”
      The above statement is the behaviour of the ASA firewall in routed mode, whatever you decide to filter or not the vMAC.

      Just a reminder, although this is not a issue with a single local Multi-chassis Ether-channel (the same vMAC appears always and only via the same ECLB), the result of the above for the DCI scenario (1 MEC on each site and 1 CLACP from the ASA cluster) is that the same vMAC appears on different interfaces of the vPC peer (MEC); one from the local ASA members (ECLB), and one from the DCI L2 extension. As the vMAC is not needed across the DCI (CCL will use the BIA to communicate across the cluster), therefore the first easier way to address this duplicate vMAC address that we can think of, is to filter this vMAC.
      That said, keep in mind this virtual MAC will be the source MAC address for each L3 packet sent from any ASA routed device toward the servers.

      Secondly you are correct stating that, after the move, the return traffic from the server will hit the local ASA device, which will notify the original owner via the syn cookies and will redirect the return traffic back to the owner ASA. However, next step, when the traffic (same stateful session) continues from outside, there are therefore two options:
      1 – there is no path optimisation deployed, thus the flow is attracted to DC-1 (more specific L3 metric), hits the ASA in DC-1, which routes it toward the VM (moved to DC-2). Traffic is sent to the OTV DCI link, but the ASA vMAC is blocked. It results a black-hole.
      2 – there is path optimisation deployed (e.g. LISP), thus the flow is attracted to DC-2 (where the VM has moved), hits the ASA in DC-2 which redirects the traffic accordingly to the original owner (ASA in DC-1). Same story as above, the vMAC being filtered, the traffic is therefore black-hole.

      This is the reason why I’m saying, this can work only for “COLD” migration, in the sense that all sessions after the cold migration of the VM are always established from outside ,whatever you decide to enable a ingress path optimisation or not.

      Does that make sense ?

      B) ASA in transparent mode : I agree with you, and that’s what we are recommending and somehow mandatory if you need E-W firewall insertion (for inter-tier application communication). However as far as I’m aware, there are still many companies (almost 20% in EMEAR) which are imposed to use the firewall in routed mode (security rules & policies). For those enterprises, the solution is to isolate the FW and server with a L3 router.

      That said, even ASA cluster configured in transparent mode, if you don’t need E-W insertion security, then a router inserted between the ASA devices and the servers is certainly more solid and secure than transparent mode from the core router toward the application tiers, and use an ASAv for the E-W isolation inside each virtual tenant container.

      This is my personal point of view, some may think different, and that’s fine. At least having additional designs (e.g. inserted router between ASA and Server) offers more options, more flexible architecture.

      Thank you, yves

  2. A) Thanks for the clarification. Isn’t there an automated way to implement some ACLs rules allowing the traffic of vMotionned VMs on the DCI link with ACI APIC?

    B) If some companies absolutely need routed mode on the clustered firewalls, how about using individual L3 mode (instead of spanned EtherChannel); this would
    – allow dynamic routes
    – optimize the E/W traffic which could be routed back by one of the clustered firewalls. That means 2 hops less than in Fig. 7 (ASAs in transparent mode).
    – make the duplicate vMAC issue between DCs disappear, since each unit has a separate IP/MAC address pair on its data interfaces. This would allow vMotionned VMs between DCs to continue N/S and E/W communication.
    To avoid traffic trombones, there’s also the possibility to implement a FHRP on the L3 switches in both DCs.

    • Yves says:

      Hi Jean-Christophe,
      A) sorry, can you elaborate your thoughts ?
      B) Yes all points your made are correct. Individual mode is the recommended way to ECMP load distribute the workflow among the local FW in routed mode. What I meant was FW in routed mode in order to be the default gateway for the application tiers.

      does that make sense ?

      thank you, yves

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.