VXLAN Part I. Why do we need VXLAN?

Introduction

This section examines the challenges that virtualization of servers causes for Datacenter networks with traditional three-layer architecture and how VXLAN can respond to these challenges. At the end of this article, you can find a mindmap for memory builder.

Challenges for existing Datacenter networks

Figure 1-1 shows a hypothetical 3-tier Cloud Service Provider DC network consisting of the following components.

• Access layer (L2): Twenty of the 48-port switches. Access - Distribution links 2 x 10Gbps MEC (Multichassis EtherChannel).
• Distribution layer (L2 / L3): Two distribution switches, which together form a virtualized switch. The default gateway for server segments is in distribution switches. Distribution - Core Links are L3.
• Core Layer (L3): Two Core switches

Figure 1-1: The hypothetical Cloud SP Datacenter network.

Assume that there are 48 physical servers connected to each access switch. Each of these servers includes five different tenants (dedicated virtualized environments) with their own virtual routing (VRF). One tenant consists of three broadcast domains; Presentation, Application and Database, each with two virtual machines backing up each other. The customer manages his own tenant and can define the VLANs IDs, the mac addresses of virtual machines and the IP address architecture. The mobility of virtual machines is unlimited. Based on this information, theoretically, there can be:

• Physical servers: 960 => 20 (ToR) x 48 (port per ToR)
• VMs / Mac addresses / ARP entries: 28 800 => 960 (hosts) x 30 (VM per host)
• Broadcast Domains: 14 400 => 5 (tenant per host) x 3 (VLANs per tenant) x 960 (hosts)
• Tenants / VRF: 4800 => 960 hosts x 5 tenants

Although our example is purely theoretical, it can identify the challenges that today's Datacenter service providers have:

VLAN id limitation: with 12-bits there can only be 4096 different VLANs. In small and medium-size data centers this is more than enough, but in massive Public Cloud Service Provider data centers, this may not be enough.

Multi-tenant: In multi-tenant environments where customers can define both VLAN ids and mac addresses of virtual machines, overlapping may occur.

Mac table size: There are 28,800 virtual machines connected to our example network, which means that switches might have 28 000 mac addresses in their mac address table. Our example demonstrates that the number of mac entries on switches can be considerably large through the server virtualization. If there are more mac addresses that can be stored in switch mac table, the switch may not learn new mac addresses before the unused mac addresses are aged out. This could lead to unnecessary flooding due to unknown destination mac-addresses.

Note! Cisco Nexus 9500/9300 Series Switches have tested support for 90,000 mac addresses.                             

ARP table size: In our example network, the gateway function is on distribution layer switches. Server virtualization also increases the number of IP-MAC entries stored on the ARP table. There can be more than 28 000 IP-MAC entries in our distribution switches.

Note! Cisco Nexus 9500 Series Switches have tested support for 60,000 IPv4 ARPs and 30,000 IPv6 NDs. The corresponding figures for Nexus 9300 series switches are 45,000 (IPv4 ARP) and 20,000 (IPv6 ND).

Spanning-tree (STP): In the traditional layer 2 networks, the Control Plane protocol is STP, which provides a loop-free L2 topology for the hosts. Data Plane is formed by "Flood and Learn" principle, where switches learn the mac addresses from the received Ethernet frames and flood the BUM traffic (Broadcast, Unknown unicast and Multicast). Without the STP-like loop prevention mechanism, the network could choke excessive broadcast messages. Since STP does not support load balancing between links, some of the links may not be actively utilized for traffic transfer. However, load balancing can be achieved by using proper STP design where the root switches of different VLANs or MST instances are decentralized or Multichassis EtherChannel technology is used. In principle, the network could also be constructed using a routed-access model (link switches L3), but this would prevent VM machines from being mobilized between the physical hosts located at a different switch.

How VXLAN responds to the challenges

VXLAN is a MAC-over-IP / UDP tunneling mechanism that allows Layer2 network segments to be "stretched" over Layer 3 network. Each stretched Layer 2 network is represented as a VXLAN segment identified by a 24-bit segment-id (VXLAN Network - VNI). With VNI, we can identify 16 million VXLAN segments. Virtual machines belonging to different VXLAN segments may have overlapping mac addresses or VLANs since only hosts inside VXLAN segment can have Layer 2 connection between each other.

Because VXLAN segments are tunneled over the Layer 3 network, no Spanning Tree Protocol is required. In VXLAN technology-based DC, VLANs no longer has global significance since VLANs are a switch or even switch port specific, meaning that host-A, on subnet 192.168.10.0/24 in Leaf switch 101 may belong to VLANs 200 while host B in the same subnet on different Leaf switch 102 may belong to VLAN 201.

In a VXLAN-based DC network, the Leaf (access) switch link to Spine Switches (Distribution + Core) is a Layer 3 connection, so other than the Leaf switches are not aware of mac addresses of Virtual Machines.

VXLAN enables the use of anycast gateway, where the routing of client networks is distributed between Leaf Switches. This means that gateway address of the network 192.168.10.0/24 (192.168.1.1) is found on each Leaf switch. As the virtual machine moves to a new host connected to the different switch, its gateway is still directly connected. The decentralized anycast gateway greatly reduces the number of mac addresses on individual switch ARP table.

Why VXLAN - Mindmap

Figure 1-2: The Mind Map.

Edited February 9.3.2018 | Toni Pasanen CCIE#28158
Next part: VXLAN Part II. The Underlay network – Unicast Routing  (Downloaded)

Sources:

RFC 7348: Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/scalability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_612I21/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_612I12_chapter_01.html