Remote VTEP Leaf-102: Low-Level Control Plane Analysis
In this section, we will first examine the update process of the BGP tables on the VTEP switch Leaf-102 when it receives a BGP Update message from Spine-11. After that, we will go through the update processes for the MAC-VRF and the MAC Address Table. Finally, we will examine how the VXLAN manager on Leaf-102 learns the IP address of Leaf-10's NVE interface and creates a unidirectional NVE peer record in the NVE Peer Database based on this information.
Remote Learning: BGP Processes
We have configured switches Leaf-101 and Leaf-102 as Route Reflector Clients on the Spine-11 switch. Spine-11 has stored the content of the BGP Update message sent by Leaf-101 in the neighbor-specific Adj-RIB-In of Leaf-101. Spine-11 does not import this information in its local BGP Loc-RIB because we have not defined a BGP import policy. Since Leaf-102 is an RR Client, the BGP process on Spine-11 copies this information in the neighbor-specific Adj-RIB-Out table for Leaf-102 and sends the information to Leaf-102 in a BGP Update message. The BGP process on Leaf-102 stores the received information from the Adj-RIB-In table to the BGP Loc-RIB according to the import policy of EVPN Instance 10010 (import RT 65000:10010). During the import process, the Route Distinguisher values are also modified to match the configuration of Leaf-102: change the RD value from 192.168.10.101:32777 (received RD) to 192.168.10.102:32777 (local RD).
Figure 3-13: MAC Address Propagation Process – From BGP Adj-RIB-Out.
Example 3-15 shows a summary of the BGP tables on Leaf-102. In the BGP Adj-RIB-In table, under RD 192.168.10.101:32777, there are two valid (*) MAC Advertisement routes learned from an internal BGP peer (i), both with the same next-hop IP address pointing to NVE1 interface on Leaf-101. The first of these routes has been selected as the best route (>) and imported into the BGP Loc-RIB with RD 192.168.10.102:32777 and L2VNI 10010.
Leaf-102# sh bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 99, Local Router ID is 192.168.10.102
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.10.101:32777
*>i[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216
192.168.20.101 100 0 i
* i 192.168.20.101 100 0 i
Route Distinguisher: 192.168.10.102:32777 (L2VNI 10010)
*>i[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216
192.168.20.101 100 0 i
Example 3-15: Remote Leaf-102 BGP Entries - Summary.
Example 3-16 shows the BGP Adj-RIB-In and Loc-RIB tables of switch Leaf-102 for the MAC address 0050.7966.6806. Under Route Distinguisher 192.168.10.101:32777, there is an EVPN MAC Advertisement Route [2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, which has been learned from both Spine-11 and Spine-12. The BGP Path Attributes of the BGP Update messages are the same, but the path advertised by Spine-11 is selected as the best path based on the smaller BGP Router ID, and thus it is imported into the BGP Loc-RIB (Imported to 1 destination).
The route under the Comment 2 has been imported into the BGP Loc-RIB from the Adj-RIB-In (Imported from). During the process the RD is changed based on the EVI configuration. As an observant reader, you probably noticed that the BGP multipath solution for eBGP and iBGP is enabled in the Leaf-102 BGP process (Multipath: eBGP iBGP). Due to BGP's loop prevention mechanism, the solution does not apply to Layer 2 MAC addresses learned from iBGP peers. However, this does not mean that our EVPN fabric does not support Equal-Cost Multipath (ECMP) flow-based load balancing. I will explain it after the BGP process description.
Leaf-102# show bgp l2vpn evpn 0050.7966.6806
Comment 1: Original information received from EVPN peers
and stored into in Adj-Rib-In
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.168.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, version 98
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Multipath: eBGP iBGP
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-10010
AS-Path: NONE, path sourced internal to AS
192.168.20.101 (metric 81) from 192.168.10.11 (192.168.10.11)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10010
Extcommunity: RT:65000:10010 ENCAP:8
Originator: 192.168.10.101 Cluster list: 192.168.10.11
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
192.168.20.101 (metric 81) from 192.168.10.12 (192.168.10.12)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10010
Extcommunity: RT:65000:10010 ENCAP:8
Originator: 192.168.10.101 Cluster list: 192.168.10.12
Path-id 1 not advertised to any peer
Comment 2: Routing Entry Imported from Adj-RIB-In to Loc-RIB.
Route Distinguisher: 192.168.10.102:32777 (L2VNI 10010)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, version 99
Paths: (1 available, best #1)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW
Multipath: eBGP iBGP
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop, in rib
Imported from 192.168.10.101:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216
AS-Path: NONE, path sourced internal to AS
192.168.20.101 (metric 81) from 192.168.10.11 (192.168.10.11)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10010
Extcommunity: RT:65000:10010 ENCAP:8
Originator: 192.168.10.101 Cluster list: 192.168.10.11
Path-id 1 not advertised to any peer
Example 3-16: Remote Leaf-102 BGP Entries - Summary.
Example 3-17 shows the BGP process regarding the EVPN MAC Advertisement route information sent by the switches Spine-11 and Spine-12 to the switch Leaf-102. The first two processing stages of the route information sent by both spine switches are identical. We see how the route information is first received into Adj-RIB-In. From there, after RD modification, the information is encoded into BGP Loc-RIB. After this, we see that only the information advertised by Spine-11 is installed into the L2RIB.
Leaf-102# show bgp internal event-history events | i 0050.7966.6806
#Comment 1: BGP Process concerning BGP Update received from Spine-11
Installed into RIB and downloaded to L2RIB.
[bgp_l2vpn_evpn_l2rib_send_mac_route:3251] (default) RIB: [L2VPN EVPN] Suppressing 192.168.10.102:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216 download to L2RIB
[bgp_l2vpn_evpn_rib_add_or_delete_route:4004] (default) RIB: [L2VPN EVPN] Add/delete 192.168.10.102:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, flags=0x210, in_rib: yes
[bgp_tbl_ctx_import:3890] (default) IMP: [L2VPN EVPN] Importing prefix 192.168.10.101:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216 to <default> RD 192.168.10.102:32777
[bgp_l2vpn_evpn_rib_add_or_delete_route:4004] (default) RIB: [L2VPN EVPN] Add/delete 192.168.10.101:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, flags=0x200, evi_ctx invalid, in_rib: no
#Comment 2: BGP Process concerning BGP Update received from Spine-12
Not installed into RIB.
[bgp_l2vpn_evpn_rib_add_or_delete_route:4004] (default) RIB: [L2VPN EVPN] Add/delete 192.168.10.102:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, flags=0x200, in_rib: no
[bgp_tbl_ctx_import:3918] (default) IMP: [L2VPN EVPN] Created import destination entry for 192.168.10.102:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216
[bgp_tbl_ctx_import:3890] (default) IMP: [L2VPN EVPN] Importing prefix 192.168.10.101:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216 to <default> RD 192.168.10.102:32777
[bgp_l2vpn_evpn_rib_add_or_delete_route:4004] (default) RIB: [L2VPN EVPN] Add/delete 192.168.10.101:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, flags=0x200, evi_ctx invalid, in_rib: no
Example 3-17: Remote Leaf-102 BGP MAC Advertisement Route Import Process.
Example 3-18 shows the steps where a MAC Advertisement route is exported from the BGP Loc-RIB to the L2RIB.
Leaf-102# show bgp event-history l2rib | i 0050.7966.6806
[bgp_l2vpn_evpn_calculate_nh_list:2064] (default) L2RIB: Adding path to 192.168.10.102:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216 via 192.168.20.101, label 10010, tags 0, ip_pctag 0, flags2 0x400, l2r_nh_flags 0x0
[bgp_l2vpn_evpn_l2rib_send_mac_route:3513] (default) L2RIB: (EVI L2-10010) Sent add MAC route 192.168.10.102:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216, topoid 10, nhs 1, encap 8, flags 0x0, if 0x0
[bgp_l2vpn_evpn_calculate_nh_list:2064] (default) L2RIB: Adding path to 192.168.10.102:32777:[2]:[0]:[0]:[48]:[0050.7966.6806]:[0]:[0.0.0.0]/216 via 192.168.20.101, label 10010, tags 0, ip_pctag 0, flags2 0x400, l2r_nh_flags 0x0
Example 3-18: Remote Leaf-102 BGP Process - MAC Advertisement Route into L2RIB.
ECMP for MAC Addresses
I previously noted that the iBGP multipath solution does not apply to Layer 2 traffic. Instead, Flow-Based ECMP is based on IGP routing. Example 3-18 shows that there are two equal-cost routes to the IP address 192.168.20.101 on the NVE interface of Leaf-101.
Leaf-102# sh ip route 192.168.20.101
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.20.101/32, ubest/mbest: 2/0
*via 192.168.0.11, Eth1/1, [110/81], 5d01h, ospf-UNDERLAY-NET, intra
*via 192.168.0.12, Eth1/2, [110/81], 5d01h, ospf-UNDERLAY-NET, intra
Leaf-102#
Example 3-18: Remote Leaf-102 BGP Entries - Summary.
To verify that Flow-Based load balancing is occurring, we will ping from TS1 (Leaf-101) to TS2 (Leaf-102) while simultaneously capturing ICMP packets on both spine switches using the built-in Ethanalyzer tool on the Nexus switches. Ethanalyzer only shows traffic that is directed to the CPU, and therefore, we configure a monitor session on the spine switches as shown in Example 3-19.
monitor session 1
source interface Ethernet1/1 both
source interface Ethernet1/2 both
destination interface sup-eth0
no shut
Example 3-19: Monitor Session on Spine Switches.
In the ICMP packet capture 3-2, lines 1 and 2, we see that Leaf-101 has sent an ICMP Request message with ID 0xc0b (seq=1/256) to the Spine-11 switch. In the ICMP packet capture 3-3, lines 5 and 6, we see that Leaf-101 has sent an ICMP Request message with ID 0xc2b (seq=3/768) to the Spine-12 switch. Note that each sequence is listed twice because the monitor session records inbound traffic from both uplink interfaces.
Spine-11# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'
1. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc0b9, seq=1/256
2. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc0b9, seq=1/256
3. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc1b9, seq=2/512
4. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc1b9, seq=2/512
5. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc2b9, seq=3/768
6. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc2b9, seq=3/768
7. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc3b9, seq=4/1024
8. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc3b9, seq=4/1024
Capture 3-2: Packet Capture from Spine-11 – Ping from TS2 to TS.
Spine-12# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'
1. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc0b9, seq=1/256
2. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc0b9, seq=1/256
3. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc1b9, seq=2/512
4. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc1b9, seq=2/512
5. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc2b9, seq=3/768
6. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc2b9, seq=3/768
7. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc3b9, seq=4/1024
8. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc3b9, seq=4/1024
9. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc4b9, seq=5/1280
10. 172.16.10.10 → 172.16.10.20 ICMP 148 Echo (ping) request id=0xc4b9, seq=5/1280
11. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc4b9, seq=5/1280
12. 172.16.10.20 → 172.16.10.10 ICMP 148 Echo (ping) reply id=0xc4b9, seq=5/1280
Capture 3-3: Packet Capture from Spine-12 – Ping from TS2 to TS.
Remote Learning: L2RIB Update
After Leaf-102's BGP process has selected the best path for the MAC address 0050.7966.6805 and imported it into the BGP Loc-RIB, the routing information is stored in the L2RIB.
Figure 3-14: MAC Address Propagation Process – From BGP Loc-RIB to L2RIB.
Example 3-21 shows the L2RIB update process (read from bottom to top). The route information is received and marked with the flag "Received". After this, a new MAC route is created with the Next-Hop IP address 192.168.20.101 and L2VNI 10010.
Leaf-102# show system internal l2rib event-history mac | i 0050.7966.6806
[l2rib_show_mac_rt:2948] (10,0050.7966.6806,5):
NH[0]: 192.168.20.101 (Label: 10010) egressVNI: 0
<snipped for better readability>
[l2rib_obj_mac_route_create:3721] (10,0050.7966.6806,5): NH[0]: 192.168.20.101 (Label: 10010)
[l2rib_obj_mac_route_create:3703] (10,0050.7966.6806,5): MAC route created with seqNum: 0, flags: (Rcv),
[l2rib_obj_mac_route_create:3597] (10,0050.7966.6806,5): Setting receive flag
[l2rib_client_show_route_msg:1787] Rcvd MAC ROUTE msg: (10, 0050.7966.6806), vni 0, admin_dist 0, seq 0, soo 0,
Example 3-20: Storing MAC Advertisement Route from BGP Loc-RIB to L2RIB.
Example 3-21 illustrates the MAC address stored in L2RIB with its reachability information. The MAC address belongs to VLAN 10 (Topology 10), and information is received from BGP (Protocol: BGP, Flags: Received). The Next-Hop is the NVE1 IP address of Leaf-101. The "Label: 10010" specifies the L2VNI value in the VXLAN header when data is sent to TS1. We can also see that the information is sent to L2FM, which eventually records it to the MAC Address Table.
Leaf-102# sh l2route mac topology 10 detail
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending
(S):Stale (C):Clear, (Ps):Peer Sync (O):Re-Originated (Nho):NH-Override
(Asy):Asymmetric (Gw):Gateway
(Bh):Blackhole, (Dum):Dummy
(Pf):Permanently-Frozen, (Orp): Orphan
(PipOrp): Directly connected Orphan to PIP based vPC BGW
(PipPeerOrp): Orphan connected to peer of PIP based vPC BGW
Topology Mac Address Prod Flags Seq No Next-Hops
----------- -------------- ------ ---------- ---------- -----------------------------
10 0050.7966.6806 BGP Rcv 0 192.168.20.101 (Label: 10010)
Route Resolution Type: Regular
Forwarding State: Resolved (PeerID: 1)
Sent To: L2FM
Encap: 1
Example 3-21: MAC-VRF (L2RIB) Entry About MAC Address 0050.7966.6806.
Remote VTEP Leaf-102: MAC Table
After the MAC Advertisement route information is stored in the L2RIB, it is also recorded in the MAC Address table as shown in Figure 3-15.
Figure 3-15: MAC Address Propagation Process – From L2RIB to MAC Table.
Example 3-22 shows the MAC table update process for the TS1 MAC address. The VXLAN Network Identifier (VNI) for the MAC address is presented in HEX format as 0x271a, which is 10010 in decimal format. Compare this to the VNI value in Capture 3-1.
Leaf-102# sh system internal l2fm event-history detail | include 0050.7966.6806
[l2fm_mac_regist_notify_detailed:7774] mnode not found in add or del MAC reg db for mac: 0050.7966.6806 vlan_id: 10
[l2fm_mac_regist_notify_detailed:7702] Searching reg db to see if MAC is registered MAC 0050.7966.6806, mac_ntfn_flags=0 always_notify: 0
[l2fm_add_to_mcast_mac_ins_msg:2328] Install idx 0 mac 0050.7966.6806 vlan 10 if 0x49080001 everywhere
[l2fm_l2rib_batch_remote_mac_entry:737] curr_idx: 0, VLAN:10, i/f:0x49080001 mac-flags = 7, mac=0050.7966.6806, VNI = 0x271a
Example 3-22: Monitor Session on Spine Switches.
From Example 3-24, we can see that the MAC address 0050.7966.6806 has been installed (Op:INSERT) into the MAC address table (Db: 0-MACDB) from the L2RIB (Src: 23-L2RIB). The example also shows the Interface Leaf-101 interface ID.
Leaf-102# show system internal l2fm l2dbg macdb address 0050.7966.6806 vlan 10
Legend
------
Db: 0-MACDB, 1-GWMACDB, 2-SMACDB, 3-RMDB, 4-SECMACDB 5-STAGEDB
Db: 6-MACFAILDB, 7-PEER_SYNC_DB, 8-CACHE_DB, 9-HOLD_DB
Src: 0-UNKNOWN, 1-L2FM, 2-PEER, 3-LC, 4-HSRP
5-GLBP, 6-VRRP, 7-STP, 8-DOTX, 9-PSEC 10-CLI 11-PVLAN
12-ETHPM, 13-ALW_LRN, 14-Non_PI_MOD, 15-MCT_DOWN, 16 - SDB
17-OTV, 18-Deounce Timer, 19-AM, 20-PCM_DOWN, 21 - MCT_UP
22-VxLAN, 23-L2RIB 24-CTRL, 25-UFDM 26-VRRPV3 27-VIM 28-DEJAVU 29-SMAC_MV
30-ARP, 31-DHCP
Slot:0 based for LCS 31-MCEC 20-OTV/ORIB
VLAN: 10 MAC: 0050.7966.6806 FE ID: 0
Time If/swid Db Op Src Slot FE-BMP Count Detail
May 6 07:04:41 2024:49882 0x49080001 0 INSERT 23 0 0xffff
Example 3-24: L2FM – Inserting MAC Address from L2RIB to MAC Address Table.
Example 3-25 shows how the MAC address of TS1 is stored in the MAC address table of VLAN 10 on Leaf-102. The information is learned dynamically (Type: Dynamic) through the Control Plane (C-Control Plane). The Next-Hop IP address is the NVE interface of Leaf-101.
Leaf-102# show mac address-table vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable A – ESI Active Path, S – ESI Standby Path
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C 10 0050.7966.6806 dynamic NA F F nve1(192.168.20.101)
Example 3-25: VLAN 10 MAC Address Table on Leaf-102.
Remote VTEP Leaf-102: NVE Peering Based on BGP Update
Leaf-102 learns the MAC address's physical (Leaf-101) and logical (VLAN 10/VNI 10010) location from the received BGP Update message's EVPN NLRI and stores them from the BGP table to the L2RIB and MAC address table. Additionally, Leaf-102's VXLAN Manager stores Leaf-101's VTEP peering parameters, which inform it about the configured VNIs on the peer and the tunnel IP address used for sending traffic. Note that the VXLAN Manager creates a VTEP peering entry only if the peer has a configured local VNI. In our example, VNI 10010 is present on both leaf switches. The VXLAN Manager can also identify a new VTEP peering through the Data Plane based on tunneling headers (IP source UDP dst port, and VXLAN Network ID).
Figure 3-16: New VTEP resolution Based of the BGP Update Message about EVPN RT 2.
Example 3-26 shows the information related to VXLAN tunneling stored by the VXLAN Manager on the VTEP switch Leaf-101. Because the BGP Update message's EVPN NLRI MAC Advertisement Route did not contain IP address information (VLAN 10 does not have a routing interface), the NLRI did not include the Router MAC address. For this reason, it is marked as N/A.
Leaf-102# sh nve peers detail
Details of nve Peers:
----------------------------------------
Peer-Ip: 192.168.20.101
NVE Interface : nve1
Peer State : Up
Peer Uptime : 00:02:06
Router-Mac : n/a
Peer First VNI : 10010
Time since Create : 00:02:06
Configured VNIs : 10010
Provision State : peer-add-complete
Learnt CP VNIs : 10010
vni assignment mode : SYMMETRIC
Peer Location : N/A
Group policy capable: no
----------------------------------------
Example 3-26: New NVE/VTEP Peer info Related to Leaf-101.
No comments:
Post a Comment