A Cisco ACI switch topology is a two-tier, leaf-spine design. The big idea is to keep the fabric sim

About About Ethan Banks Calendar Disclosures Publications Social @ecbanks LinkedIn Google+ Contact How To Use This Site Newsletter Podcasts Packet Pushers A new one coming soon! Services Subscribe via RSS
As I study software defined networking architectures, I ve observed that none of them are exactly alike. There are common approaches, but once diving into the details of what s being done and how, even the common approaches seem to have as many differences as similarities.
One of the most interesting 9 run run elements of SDN architectures is traffic forwarding . How does traffic get from one point to another 9 run run in an SDN world? Cisco ACI s traffic forwarding approach is intriguing in that it neither relies on the controller for forwarding 9 run run instructions, nor does it rely on a group of network 9 run run engineers to configure it. Rather, Cisco ACI fabric is a self-configuring cloud of leaf and spine switches that forward 9 run run traffic between endpoints 9 run run without forwarding tables being programmed by the Application Policy Infrastructure Controller (APIC). APIC assumes forwarding will happen, worrying instead about policy .
The notion of a self-configuring fabric that delivers traffic with no outside instruction sounds mystical. How, exactly, does Cisco ACI forward traffic? Wanting to understand the basics myself, I spent time reviewing presentations by engineers from the Cisco ACI team, and have distilled the information down as best as I could.
A Cisco ACI switch topology is a two-tier, leaf-spine design. The big idea is to keep the fabric simple, and a two-tier 9 run run architecture has efficiency, predictability, and resilience to commend it. Two-tier leaf-spine design is the most common data center network reference architecture recommended by the industry today. It s hard to read a data center related technical whitepaper without leaf-spine being mentioned. 9 run run Cisco has not done anything 9 run run strange here.
What 9 run run I can t tell you is if three-tier leaf-spine designs sometimes used to scale for host density are supported, 9 run run but I tend to think not. The Cisco ACI behavior discussed in the videos imply a non-blocking, two-tier fabric throughout. 9 run run Three-tier leaf-spine designs are not non-blocking. 9 run run Therefore, very large data centers wishing to run a thousands of physical hosts within a single ACI fabric would scale horizontally, adding spine switches. Considering the high port density of certain switches in the Nexus 9000 family, I can t imagine 9 run run scale being a limitation for almost anyone.
The initial setup process seems simple enough. I m probably oversimplifying it, but the way I understand it, the Application Policy Infrastructure Controller (APIC) is connected to the ACI switch fabric. APIC discovers the switch topology. 9 run run APIC then assigns 9 run run IP addresses to each node (a leaf switch), to be used as VxLAN tunnel endpoints (VTEPs). VXLAN is the sole transport 9 run run between leaf and spine across the fabric. Why? Well, a couple of reasons. One, policy is enforced in part by shipping traffic between specific leaf nodes. Two, you get 16M segments instead 9 run run of 4K VLANs or a limited number of VRFs. VxLAN offers virtualized network segments with essentially no practical limit. For what it s worth, NVGRE and STT have the same 16M limit, and presumably Geneve does as well, although I d have to dig out the IETF draft and check.
It s worth noting that in Cisco ACI terminology, this leaf-spine fabric is known as the infrastructure space . The uplinks facing endpoints are the user space . Within the infrastructure space, there are only VTEPs. Nodes are switches. Endpoints are hosts – you know, the things actually generated traffic. Packet flow through the ACI fabric
How does a packet make it through the Cisco ACI fabric? First, be aware that the default gateway for any network segment is distributed across all leaf nodes in the ACI fabric. Therefore, whatever leaf node an endpoint is connected to will be the layer 3 gateway. This could hardly be otherwise when considering what happens when a packet arrives at a leaf node.
When a packet flows into a leaf switch, headers are stripped. Whether the frame or packet had it s own 802.1q, NVGRE, or VxLAN tags or wrappers when arriving at the leaf switch 9 run run – or if it was untagged – it just doesn t matter 9 run run to ACI. That information is not important to deliver a packet across the fabric. 9 run run Once the headers are stripped, the remaining packet is encapsulated in VxLAN and forwarded 9 run run across the fabric from the ingress leaf node to the egress leaf node. ACI determines where the egress leaf node is, by sorted out where the packet needs to go based on the ACI policy. When the VXLAN packet arrives at the egress leaf node, the VXLAN header is removed, and the packet encapsulated in whatever way is needed to deliver it to the destination host. The packet does NOT have to be re-encapsulated the same way it was originally encapsulated when arriving 9 run run at the ingress leaf node. So, a packet coul