VXLAN Part XI: Using vPC Peer Link as an Underlay Backup Path

Edited: Wednesday, 5 September 2018 | Toni Pasanen

This short post shows how VTEP Leaf switch can use a vPC peer link as a backup path to Spine switch in a situation where the Leaf switch loses connection to the Spine switch. This is recommended redundancy model when using vPC in VXLAN BGP EVPN fabric. Just like in my previous posts, I am using only one Spine switch to keep things as simple as possible.

Figure 11-1: Example Topology and IP addressing

Before setting up the underlay network backup path over the vPC Peer Link, let’s quickly recap what is needed in Underlay network. First, we need an IP address to Inter-Switch links. I have used unnumbered link addresses, where the IP address of Loopback 0 is used. Then we need OSPF to advertise IP address information (link and Loopback information). These are the requirements for Unicast traffic. We also need Multicast for BUM traffic (unless we use Ingres-Replication), which means that PIM is needed on Inter-Switch link. These are the basic requirements for the Underlay network.

Configuration

Configuration is simple; all necessary commands are shown in figure 11-1:

Step 1: Create VLAN 99

Step 2: Create Interface VLAN 99 an assign the IP address to it.

Step 3: Enable ospf and set link type to P2P in VLAN interface (we do not need DR/BDR election here)

Step 4: Enable PIM-SM on Interface VLAN 99 

Figure 11-2: SVI for vPC Backup.

Note! We are using Port-channel 23 (Trunk) as vPC Peer Link and VLANs 1-4094 are allowed in it. This is why there is no need switchport trunk allowed vlan add 99 –command.

Vlan 99  Name Underlay-BUoVPC_Peer-Link ! interface Vlan99   description ** Underlay BU over vPC Peer-Link **   no shutdown   ip address 192.168.99.2/30   ip ospf network point-to-point   ip router ospf UNDERLAY-NET area 0.0.0.0   ip pim sparse-mode

Example 11-1: SVI for Backup over vPC Peer Link in VXLAN fabric.

Note! I am using VLAN 99 to establish the Backup Underlay Network connection over the vPC Peer Link. VLAN 99 is not a client VLAN (not mapped to any L2VNI) but an infra VLAN, which is why the command system nve infra-vlans 99 is required when using physical Nexus switches. I am using Nexus 9000v with NX-OSv and there is no such command in it.

Verification

First, we ping between interface vlan VLAN 99 just to make sure that we have IP connectivity.

Leaf-102# ping 192.168.99.2 PING 192.168.99.2 (192.168.99.2): 56 data bytes 64 bytes from 192.168.99.2: icmp_seq=0 ttl=254 time=59.303 ms 64 bytes from 192.168.99.2: icmp_seq=1 ttl=254 time=47.207 ms 64 bytes from 192.168.99.2: icmp_seq=2 ttl=254 time=65.063 ms 64 bytes from 192.168.99.2: icmp_seq=3 ttl=254 time=46.248 ms 64 bytes from 192.168.99.2: icmp_seq=4 ttl=254 time=32.883 ms --- 192.168.99.2 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 32.883/50.14/65.063 ms

Example 11-2: ping test between vPC peers Leaf-102 and Leaf-103.

Then we verify that the Leaf-102 and Leaf-103 are OSPF and PIM neighbors.

Leaf-103# sh ip ospf neighbors  OSPF Process ID UNDERLAY-NET VRF default  Total number of neighbors: 2  Neighbor ID     Pri State            Up Time  Address         Interface  192.168.0.11      1 FULL/ -          03:03:07 192.168.0.11    Eth1/1  192.168.0.102     1 FULL/ -          00:02:39 192.168.99.1    Vlan99

Example 11-3: OSPF neighbors.

Leaf-103# sh ip pim neighbor PIM Neighbor Status for VRF "default" Neighbor        Interface            Uptime    Expires   DR       Bidir-  BFD                                                          Priority Capable State 192.168.0.11    Ethernet1/1          03:04:36  00:01:42  1        yes     n/a 192.168.99.1    Vlan99               01:20:16  00:01:36  1        yes     n/a

Example 11-4: PIM neighbors.

Now I am going to tear down the link between Ethernet Switch and Leaf-102 as well as the link between Leaf-103 and Spine-11 by shutting down the interfaces on both ends. After these operations, there is only one possible path from the host Cafe to the host Beef shown in Figure 11-2.

Figure 11-3: Backup path over vPC Peer Link

First, I am going to test DataPlane by pinging from host Cafe to host Beef. As we can see from the example 11-5, DataPlane is Ok.

Cafe#ping 192.168.11.12      Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.11.12, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 12/22/30 ms

Example 11-5: ping from Cafe to Beef.

Then we check the mac address-tables. The mac address-table of Leaf-103 shows that it has learned the mac address of host Cafe through its interface Po10, which leads to Ethernet Switch.

Leaf-103# show system internal l2fwder mac Legend:         * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC         age - seconds since last seen,+ - primary entry using vPC Peer-Link,         (T) - True, (F) - False, C - ControlPlane MAC    VLAN     MAC Address      Type      age     Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ <snipped> *    10    1000.0010.cafe   dynamic   00:04:38   F     F       Po10

Example 11-6: Mac address-table of Leaf-103.

While the switch Leaf-102 has learned the mac address of host Cafe via Po23, which is the vPC peer Link to Leaf-103. This is just basic mac address learning process based on flood & learn process.

Leaf-102#  show system internal l2fwder mac Legend:         * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC         age - seconds since last seen,+ - primary entry using vPC Peer-Link,         (T) - True, (F) - False, C - ControlPlane MAC    VLAN     MAC Address      Type      age     Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ <snipped> +    10    1000.0010.cafe   dynamic   00:00:23   F     F       Po23

Example 11-7: Mac address-table of Leaf-102.

Since both vPC peer switches Leaf-103 and Leaf-102 has learned and installed the mac address of host Cafe to their mac address-table, they will also send a BGP update to Spine-11 (example 11-7). Note that Leaf-103 still has IP connectivity and BGP peering with Spine-11 over the vPC peer link.

Spine-11# sh bgp l2vpn evpn <snipped> Route Distinguisher: 192.168.77.102:32777 *>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[0]:[0.0.0.0]/216                       192.168.100.23                    100          0 i *>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272                       192.168.100.23                    100          0 <snipped> Route Distinguisher: 192.168.77.103:32777 *>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[0]:[0.0.0.0]/216                       192.168.100.23                    100          0 i *>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272                       192.168.100.23                    100          0 i

Example 11-8: Host Cafe mac and mac-ip routes in Spine-11.

Leaf-101 has received this routing information from Spine-11. Note that the next-hop is set to vPC VIP address instead of PIP.

Leaf-101# sh bgp l2vpn evpn vni-id 10000 <snippde>    Network            Next Hop            Metric     LocPrf     Weight Path Route Distinguisher: 192.168.77.101:32777    (L2VNI 10000) <snipped> *>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[0]:[0.0.0.0]/216                       192.168.100.23                    100          0 i * i                   192.168.100.23                    100          0 i <snipped> *>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272                       192.168.100.23                    100          0 i * i                   192.168.100.23                    100          0 i

Example 11-9: Host Cafe related information in Leaf-101.

It installs the mac information in its L2RIB.

Leaf-101# show l2route mac all Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link (Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending (S):Stale (C):Clear, (Ps):Peer Sync (O):Re-Originated (Nho):NH-Override (Pf):Permanently-Frozen Topology    Mac Address    Prod   Flags         Seq No     Next-Hops      ----------- -------------- ------ ------------- ---------- ---------------- 10          1000.0010.beef Local  L,            0          Eth1/4         10          1000.0010.cafe BGP    SplRcv        0          192.168.100.23 <snipped>

Example 11-10: L2 RIBin Leaf-101.

From L2RIB the mac address 1000.0010.cafe is stored into the mac address-table. Now all three switches have mac address 1000.0010.cafe in the mac address-table.

Leaf-101# show system internal l2fwder mac Legend:         * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC         age - seconds since last seen,+ - primary entry using vPC Peer-Link,         (T) - True, (F) - False, C - ControlPlane MAC    VLAN     MAC Address      Type      age     Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ *    10    1000.0010.cafe    static   -          F     F  (0x47000001) nve-peer1 192.168  <snipped>

Example 11-11: Mac address-table Leaf-101.

This was a short description of how the vPC peer link can be used as an Underlay network Backup.

Author: Toni Pasanen CCIE#28158

Published: 4.9.2018

-------------------------------------------------

References:

VXLAN/EVPN Configuration Example (N9k / p2p) by Lukas Krattiger: https://community.cisco.com/t5/data-center-blogs/vxlan-evpn-configuration-example-n9k-p2p/ba-p/3663830

Configuring VXLAN BGP EVPN: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x_chapter_0100.pdf