Tuesday 4 September 2018

VXLAN Part XI: Using vPC Peer Link as an Underlay Backup Path

Edited: Wednesday, 5 September 2018 | Toni Pasanen


This short post shows how VTEP Leaf switch can use a vPC peer link as a backup path to Spine switch in a situation where the Leaf switch loses connection to the Spine switch. This is recommended redundancy model when using vPC in VXLAN BGP EVPN fabric. Just like in my previous posts, I am using only one Spine switch to keep things as simple as possible.


Figure 11-1: Example Topology and IP addressing


Before setting up the underlay network backup path over the vPC Peer Link, let’s quickly recap what is needed in Underlay network. First, we need an IP address to Inter-Switch links. I have used unnumbered link addresses, where the IP address of Loopback 0 is used. Then we need OSPF to advertise IP address information (link and Loopback information). These are the requirements for Unicast traffic. We also need Multicast for BUM traffic (unless we use Ingres-Replication), which means that PIM is needed on Inter-Switch link. These are the basic requirements for the Underlay network.


Configuration

Configuration is simple; all necessary commands are shown in figure 11-1:

Step 1: Create VLAN 99
Step 2: Create Interface VLAN 99 an assign the IP address to it.
Step 3: Enable ospf and set link type to P2P in VLAN interface (we do not need DR/BDR election here)
Step 4: Enable PIM-SM on Interface VLAN 99 


Figure 11-2: SVI for vPC Backup.

Note! We are using Port-channel 23 (Trunk) as vPC Peer Link and VLANs 1-4094 are allowed in it. This is why there is no need switchport trunk allowed vlan add 99 –command.

Vlan 99
 Name Underlay-BUoVPC_Peer-Link
!
interface Vlan99
  description ** Underlay BU over vPC Peer-Link **
  no shutdown
  ip address 192.168.99.2/30
  ip ospf network point-to-point
  ip router ospf UNDERLAY-NET area 0.0.0.0
  ip pim sparse-mode
Example 11-1: SVI for Backup over vPC Peer Link in VXLAN fabric.


Note! I am using VLAN 99 to establish the Backup Underlay Network connection over the vPC Peer Link. VLAN 99 is not a client VLAN (not mapped to any L2VNI) but an infra VLAN, which is why the command system nve infra-vlans 99 is required when using physical Nexus switches. I am using Nexus 9000v with NX-OSv and there is no such command in it.



Verification

First, we ping between interface vlan VLAN 99 just to make sure that we have IP connectivity.

Leaf-102# ping 192.168.99.2
PING 192.168.99.2 (192.168.99.2): 56 data bytes
64 bytes from 192.168.99.2: icmp_seq=0 ttl=254 time=59.303 ms
64 bytes from 192.168.99.2: icmp_seq=1 ttl=254 time=47.207 ms
64 bytes from 192.168.99.2: icmp_seq=2 ttl=254 time=65.063 ms
64 bytes from 192.168.99.2: icmp_seq=3 ttl=254 time=46.248 ms
64 bytes from 192.168.99.2: icmp_seq=4 ttl=254 time=32.883 ms

--- 192.168.99.2 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 32.883/50.14/65.063 ms
Example 11-2: ping test between vPC peers Leaf-102 and Leaf-103.

Then we verify that the Leaf-102 and Leaf-103 are OSPF and PIM neighbors.

Leaf-103# sh ip ospf neighbors
 OSPF Process ID UNDERLAY-NET VRF default
 Total number of neighbors: 2
 Neighbor ID     Pri State            Up Time  Address         Interface
 192.168.0.11      1 FULL/ -          03:03:07 192.168.0.11    Eth1/1
 192.168.0.102     1 FULL/ -          00:02:39 192.168.99.1    Vlan99
Example 11-3: OSPF neighbors.

Leaf-103# sh ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor        Interface            Uptime    Expires   DR       Bidir-  BFD
                                                         Priority Capable State
192.168.0.11    Ethernet1/1          03:04:36  00:01:42  1        yes     n/a
192.168.99.1    Vlan99               01:20:16  00:01:36  1        yes     n/a
Example 11-4: PIM neighbors.

Now I am going to tear down the link between Ethernet Switch and Leaf-102 as well as the link between Leaf-103 and Spine-11 by shutting down the interfaces on both ends. After these operations, there is only one possible path from the host Cafe to the host Beef shown in Figure 11-2.

Figure 11-3: Backup path over vPC Peer Link

First, I am going to test DataPlane by pinging from host Cafe to host Beef. As we can see from the example 11-5, DataPlane is Ok.

Cafe#ping 192.168.11.12     
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.11.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/22/30 ms
Example 11-5: ping from Cafe to Beef.

Then we check the mac address-tables. The mac address-table of Leaf-103 shows that it has learned the mac address of host Cafe through its interface Po10, which leads to Ethernet Switch.


Leaf-103# show system internal l2fwder mac
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
<snipped>
*    10    1000.0010.cafe   dynamic   00:04:38   F     F       Po10 
Example 11-6: Mac address-table of Leaf-103.

While the switch Leaf-102 has learned the mac address of host Cafe via Po23, which is the vPC peer Link to Leaf-103. This is just basic mac address learning process based on flood & learn process.


Leaf-102#  show system internal l2fwder mac
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
<snipped>
+    10    1000.0010.cafe   dynamic   00:00:23   F     F       Po23 
Example 11-7: Mac address-table of Leaf-102.

Since both vPC peer switches Leaf-103 and Leaf-102 has learned and installed the mac address of host Cafe to their mac address-table, they will also send a BGP update to Spine-11 (example 11-7). Note that Leaf-103 still has IP connectivity and BGP peering with Spine-11 over the vPC peer link.


Spine-11# sh bgp l2vpn evpn
<snipped>
Route Distinguisher: 192.168.77.102:32777
*>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[0]:[0.0.0.0]/216
                      192.168.100.23                    100          0 i
*>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272
                      192.168.100.23                    100          0
<snipped>
Route Distinguisher: 192.168.77.103:32777
*>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[0]:[0.0.0.0]/216
                      192.168.100.23                    100          0 i
*>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272
                      192.168.100.23                    100          0 i
Example 11-8: Host Cafe mac and mac-ip routes in Spine-11.

Leaf-101 has received this routing information from Spine-11. Note that the next-hop is set to vPC VIP address instead of PIP.


Leaf-101# sh bgp l2vpn evpn vni-id 10000
<snippde>
   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 192.168.77.101:32777    (L2VNI 10000)
<snipped>
*>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[0]:[0.0.0.0]/216
                      192.168.100.23                    100          0 i
* i                   192.168.100.23                    100          0 i
<snipped>
*>i[2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272
                      192.168.100.23                    100          0 i
* i                   192.168.100.23                    100          0 i
Example 11-9: Host Cafe related information in Leaf-101.

It installs the mac information in its L2RIB.


Leaf-101# show l2route mac all

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending
(S):Stale (C):Clear, (Ps):Peer Sync (O):Re-Originated (Nho):NH-Override
(Pf):Permanently-Frozen

Topology    Mac Address    Prod   Flags         Seq No     Next-Hops     
----------- -------------- ------ ------------- ---------- ----------------
10          1000.0010.beef Local  L,            0          Eth1/4        
10          1000.0010.cafe BGP    SplRcv        0          192.168.100.23
<snipped>
Example 11-10: L2 RIBin Leaf-101.

From L2RIB the mac address 1000.0010.cafe is stored into the mac address-table. Now all three switches have mac address 1000.0010.cafe in the mac address-table.
Leaf-101# show system internal l2fwder mac
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*    10    1000.0010.cafe    static   -          F     F  (0x47000001) nve-peer1 192.168 
<snipped>
Example 11-11: Mac address-table Leaf-101.

This was a short description of how the vPC peer link can be used as an Underlay network Backup.


Author: Toni Pasanen CCIE#28158
Published: 4.9.2018
-------------------------------------------------
References:


1 comment:

  1. Hello,

    I am running Nexus 9000v as well [ 9.3(1) ] but the SVI would not come up. The VLAN is allowed and active on the peer-link.

    Interface Role Sts Cost Prio.Nbr Type
    ---------------- ---- --- --------- -------- --------------------------------
    Po1 Desg BKN*4 128.4096 (vPC peer-link) Network P2p *BA, vPC_PL_Inc

    vPC Peer-link status
    ---------------------------------------------------------------------
    id Port Status Active vlans
    -- ---- ------ -------------------------------------------------
    1 Po1 up 1,10,777

    ReplyDelete

Note: only a member of this blog may post a comment.