Introduction
The previous chapter explained how to build
an IPv6-only multi-ASN Clos fabric where IPv4 prefixes can be advertised with
IPv6 link-local next hops.
In this chapter, we extend VLAN 10 between
Leaf-101 and Leaf-102 using BGP EVPN as the control plane and VXLAN as the data
plane.
Figure 7-1 shows the main building blocks
required to provide this functionality. The IP underlay, where leaf and spine
switches use IPv6 link-local addresses for eBGP peering, has already been
configured in the previous chapter. Loopback0 is used only as the BGP Router ID
and is not advertised to any peer.
The IP address of Loopback1, in turn, is used
as the VXLAN Tunnel Endpoint (VTEP) address. Each switch therefore advertises
its Loopback1 IP address using BGP IPv4 Unicast so that the VTEP addresses are
reachable across the fabric. This address identifies the local VTEP in the EVPN
control plane and is used as the BGP next hop for EVPN routes. A VXLAN packet
sent to a remote VTEP uses the remote VTEP address as the destination address
in the outer IP header.
Each extended VLAN is mapped to an L2VNI
(Layer-2 VXLAN Network Identifier). In Figure 7-1, VLAN 10 is mapped to L2VNI
10010. Leaf switches use the MP_REACH_NLRI path attribute in BGP Update
messages to advertise EVPN NLRIs. The EVPN routes carry information such as the
MAC address and the VTEP next hop, while the VNI identifies the VXLAN segment
associated with the Layer-2 service.
In the VXLAN data plane, the remote VTEP
address learned through BGP EVPN is used as the destination address in the
outer IP header, while the L2VNI is carried in the VXLAN header.
The key roles can roughly be summarized as
follows:
o
VXLAN carries Ethernet frames
between VTEPs.
o
L2VNI identifies the Layer-2
segment being extended.
o
BGP EVPN provides the overlay
control-plane information needed for forwarding.
o
IPv4 Unicast provides
reachability for the VTEP addresses.
o
IPv6 Unicast provides the
underlying IP connectivity.
VXLAN can transport both known Unicast and
BUM traffic: Broadcast, Unknown Unicast, and Multicast. Known unicast traffic
is forwarded toward a specific remote VTEP based on the destination MAC
information learned through the EVPN control plane. BUM traffic is handled
differently because it is not associated with a single destination MAC address.
We will examine both forwarding cases in this
chapter. The known Unicast path shows how an EVPN-learned remote MAC address
results in forwarding toward a specific remote VTEP through the VXLAN tunnel.
The BUM path is explained using an ARP Request, which demonstrates how a
broadcast Ethernet frame is replicated across the VXLAN overlay.
Anycast Gateway is present on the VTEPs but
is not involved when traffic remains within the same L2VNI. It becomes relevant
when a host sends traffic toward a destination in another subnet, which we will
examine later in this chapter.
Figure 7-1: BGP
EVPN with VXLAN: Example Setup.
Configuration
Configure and Advertise the VTEP
Address
Figure 7-2 shows how to configure and advertise the Loopback address
used for the VTEP. The configuration is straightforward: create a Loopback
interface and assign it an IP address with a host subnet mask. Then, under the
global BGP process, enable the IPv4 address family and advertise the Loopback
address.
Note that enabling the IPv4 address family globally is not enough to
advertise the address to a BGP peer. IPv4 Unicast must also be activated for
each peer-specific neighbor.
In this example, both Leaf-101 and Leaf-102 have established
unnumbered eBGP peering with Spine-11 over Ethernet0. IPv4 Unicast is therefore
activated for these BGP neighbors on the leaf switches. The same configuration
is required on Spine-11 for its eBGP neighbors.
Figure 7-2: Advertising
VTEP IP Addresses.
Configure
VLAN and L2VNI Mapping
The next step
is to create VLAN 10 and add the host-facing Ethernet2 interface as an untagged
member. This creates the local Layer-2 segment that will later be mapped to
L2VNI 10010.
The Linux
Bash example uses the SONiC config command to create VLAN 10 and add Ethernet2
as an untagged VLAN member. The same VLAN membership must be represented in the
SONiC configuration so that SONiC can propagate the configuration to the
hardware. In the Sonic CLI, Ethernet2 is therefore configured as an access port
for VLAN 10.
Next, we configure the VXLAN VTEP. The VXLAN
interface uses Loopback1 as the VTEP source IP address and maps L2VNI 10010 to
VLAN 10. This mapping associates the local VLAN with the VXLAN segment that is
extended between the VTEPs. A single VXLAN VTEP can contain multiple
VLAN-to-L2VNI mappings, allowing multiple Layer-2 segments to be extended over
the same VTEP.
Figure 7-3: VLAN
Creation, Interface to VLAN and VLAN-to-VNI Bindings.
Enable
L2VPN EVPN for BGP
At this
stage, all switches know the VTEP addresses and have the required VLAN-to-VNI
mappings. The final configuration step is to enable EVPN for the BGP instance.
SONiC uses
FRRouting (FRR) as its routing stack. In FRR, EVPN is enabled with the
advertise-all-vni command under the BGP instance's l2vpn evpn address-family.
The l2vpn evpn address-family must also be activated for each BGP peer with
which EVPN routes are exchanged.
FRR uses a
wildcard Route Target for automatic EVPN import. A wildcard Route Target has
the form *:VNI and matches a received Route Target with the same Local
Administrator value, regardless of its Global Administrator value. In the
typical EVPN deployment, the Local Administrator value represents the VNI.
This is
particularly useful in an eBGP fabric where each leaf has a unique BGP ASN. FRR
automatically derives the export Route Target from the local ASN and VNI, so
different leaf switches can advertise the same EVPN NLRI while attaching
different Route Targets to the BGP route. The wildcard import *:VNI allows each
leaf to import EVPN routes for the same VNI without having to explicitly
configure every remote leaf's Route Target.
Without
wildcard Route Targets, the receiving leaf could instead explicitly configure
the Route Targets of the remote leaves or use the same manually configured
Route Target across the fabric. FRR's wildcard import simplifies this
configuration and makes the design easier to automate as the number of leaf
switches grows.
Figure 7-4: Enabling
EVPN in BGP Instance.
Complete BGP Configuration
Leaf-101 & Leaf-102
The switch-specific configuration parameters are highlighted to show
which parameters differ when using configuration templates, for example. The BGP
configuration can be simplified further by using BGP peer-groups to group neighbors
that share the same configuration.
|
Leaf-101 Interface Loopback 0 description **BGP_RID** ip address 1.1.1.101/32 ! interface Loopback 1 description **VTEP** ip address 1.1.101.101/32 ! interface Ethernet0 no shutdown ipv6 enable no ipv6 nd suppress-ra ipv6 nd ra-interval 5 ! interface Ethernet2 no shutdown switchport access Vlan 10 ! router bgp 65101 router-id 1.1.1.101 log-neighbor-changes timers 60 180 ! address-family ipv4 unicast maximum-paths 1 maximum-paths ibgp 1 network 1.1.101.101/32 ! address-family l2vpn evpn advertise-all-vni dup-addr-detection ! neighbor interface Ethernet0 v6only remote-as external capability extended-nexthop ! address-family ipv4 unicast activate ! address-family ipv6 unicast activate ! address-family l2vpn evpn activate ! interface vxlan vtep_l-101 source-ip 1.1.101.101 qos-mode pipe dscp 0 map vni 10010 vlan 10 |
Leaf-102 interface Loopback 0 description **BGP_RID** ip address 1.1.1.102/32 ! interface Loopback 1 description **VTEP** ip address 1.1.102.102/32 ! interface Ethernet0 no shutdown ipv6 enable no ipv6 nd suppress-ra ipv6 nd ra-interval 5 ! interface Ethernet2 no shutdown switchport access Vlan 10 ! router bgp 65102 router-id 1.1.1.102 log-neighbor-changes timers 60 180 ! address-family ipv4 unicast maximum-paths 1 maximum-paths ibgp 1 network 1.1.102.102/32 ! address-family l2vpn evpn advertise-all-vni dup-addr-detection ! neighbor interface Ethernet0 v6only remote-as external capability extended-nexthop ! address-family ipv4 unicast activate ! address-family ipv6 unicast activate ! address-family l2vpn evpn activate ! interface vxlan vtep_l-102 source-ip 1.1.102.102 qos-mode pipe dscp 0 map vni 10010 vlan 10 |
Example 7-1:
Leaf-101 & Leaf-102 BGP EVPN Configuration.
Spine-11
interface
Loopback 0
description **BGP_RID**
ip address 1.1.1.11/32
!
interface
Loopback 100
description **reachability-test**
ip address 192.168.1.11/32
!
interface
Ethernet0
no shutdown
ipv6 enable
no ipv6 nd suppress-ra
ipv6 nd ra-interval 5
!
interface
Ethernet1
no shutdown
ipv6 enable
no ipv6 nd suppress-ra
ipv6 nd ra-interval 5
!
router bgp 65011
router-id 1.1.1.11
log-neighbor-changes
timers 60 180
!
address-family ipv4 unicast
maximum-paths 1
maximum-paths ibgp 1
!
address-family l2vpn evpn
dup-addr-detection
!
neighbor interface Ethernet0
v6only
remote-as external
capability extended-nexthop
!
address-family ipv4 unicast
activate
send-community both
!
address-family ipv6 unicast
activate
!
address-family l2vpn evpn
activate
!
neighbor interface Ethernet1
v6only
remote-as external
capability extended-nexthop
!
address-family ipv4 unicast
activate
send-community both
!
address-family ipv6 unicast
activate
!
address-family l2vpn evpn
activate
Example 7-2:
Spine-11 BGP EVPN Configuration.
Verification
Control Plane
BGP IPv4 Unicast - VTEP Reachability
As the next step after verifying VTEP reachability, we can check the
VXLAN interfaces on Leaf-101 and Leaf-102. The following examples show the
basic VXLAN interface parameters, including the VTEP name, source IP address,
EVPN NVO name, and source interface.
Leaf-101# show
bgp ipv4 unicast
BGP routing table
information for VRF default
Router identifier
1.1.1.101, local AS number 65101
Status codes: R -
removed, S - stale, s - suppressed, * - valid
h - history, d - damped, > -
best, = - multipath, q - queued, r - RIB-failure
Origin codes: i -
IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPref Weight Path
*> 1.1.101.101/32 0.0.0.0 0 32768 i
*>
1.1.102.102/32
fe80::e22:34ff:feb6:a 0
65011 65102 i
Example 7-3:
Leaf-101 - BGP Table: IPv4 unicast.
Example 7-4
verifies that the selected BGP route is also installed in the local routing
table.
Leaf-101# show
ip route
Codes: K - kernel route, C - connected, S - static,
B - BGP, O - OSPF
> - selected route, * - FIB route, q
- queued route, r - rejected route
Destination Gateway Dist/Metric
Last Update
-------------------------------------------------------------------------------------
C>*
1.1.1.101/32 Direct Loopback0 0/0
00:09:33 ago
C>*
1.1.101.101/32 Direct Loopback1 0/0
00:09:33 ago
B>* 1.1.102.102/32 via fe80::e22:34ff:feb6:a Ethernet0
20/0 00:07:52 ago
C>*
10.0.10.0/24 Direct Vlan10 0/0
00:09:25 ago
C>*
192.168.1.101/32 Direct Loopback100 0/0
00:09:33 ago
Leaf-101#
Example 7-4:
Leaf-101 - Route Table: IPv4 unicast.
The following four examples verify the same
process from the Spine-11 and Leaf-102 perspectives. The BGP routes for the
VTEP addresses are installed in the BGP table, and the selected routes are then
installed in the local RIB.
Spine-11# show
bgp ipv4 unicast
BGP routing table
information for VRF default
Router identifier
1.1.1.11, local AS number 65011
Status codes: R -
removed, S - stale, s - suppressed, * - valid
h - history, d - damped, > -
best, = - multipath, q - queued, r - RIB-failure
Origin codes: i -
IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPref
Weight Path
*> 1.1.101.101/32 fe80::ecb:81ff:fe19:a 0
0 65101 i
*> 1.1.102.102/32 fe80::ee8:78ff:fe22:a 0 0 65102
i
Example 7-5:
Spine-11 - BGP Table: IPv4 unicast.
Spine-11# show
ip route
Codes: K - kernel route, C - connected, S - static,
B - BGP, O - OSPF
> - selected route, * - FIB route, q
- queued route, r - rejected route
Destination Gateway
Dist/Metric Last Update
--------------------------------------------------------------------------------------
C>*
1.1.1.11/32 Direct Loopback0 0/0 00:08:53 ago
B>*
1.1.101.101/32 via
fe80::ecb:81ff:fe19:a Ethernet0 20/0 00:03:50 ago
B>*
1.1.102.102/32 via
fe80::ee8:78ff:fe22:a Ethernet1 20/0 00:08:15 ago
C>*
192.168.1.11/32 Direct Loopback100 0/0 00:08:53
ago
Spine-11#
Example 7-6:
Spine-11 - Route Table: IPv4 unicast.
Leaf-102# show
bgp ipv4 unicast
BGP routing table
information for VRF default
Router identifier
1.1.1.102, local AS number 65102
Status codes: R -
removed, S - stale, s - suppressed, * - valid
h - history, d - damped, > -
best, = - multipath, q - queued, r - RIB-failure
Origin codes: i -
IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPref
Weight Path
*> 1.1.101.101/32 fe80::e22:34ff:feb6:a 0
65011 65101 i
*> 1.1.102.102/32 0.0.0.0 0 32768
i
Leaf-102#
Example 7-7:
Leaf-102 - BGP Table: IPv4 unicast.
Leaf-102# show ip
route
Codes: K - kernel route, C - connected, S - static,
B - BGP, O - OSPF
> - selected route, * - FIB route, q
- queued route, r - rejected route
Destination Gateway Dist/Metric Last
Update
------------------------------------------------------------------------------------
C>*
1.1.1.102/32 Direct Loopback 0/0
00:09:34 ago
B>*
1.1.101.101/32 via
fe80::e22:34ff:feb6:a Ethernet0 20/0
00:08:43 ago
C>*
1.1.102.102/32 Direct Loopback1 0/0
00:09:34 ago
C>*
10.0.10.0/24 Direct Vlan10 0/0
00:09:33 ago
Leaf-102#
Example 7-8:
Leaf-102 - Route Table: IPv4 unicast.
VTEP Interfaces
As the next logical step after verifying VTEP
reachability, we can check the status of vxlan interfaces on VTEP switches
Leaf-10 and Leaf-102. Example below shows the basic vxlan interface options
such as VTEP name and source address.
|
Leaf-101# show
vxlan interface VTEP Name :
vtep_l-101 VTEP Source IP :
1.1.101.101 QoS Mode :
pipe(dscp:0) EVPN NVO Name : nvo1 EVPN VTEP :
vtep_l-101 Source Interface :
Loopback1 Primary IP Interface : Not
Configured External IP Interface : Not Configured Leaf-101#
|
Leaf-102# show
vxlan interface VTEP Name :
vtep_l-102 VTEP Source IP : 1.1.102.102 QoS Mode :
pipe(dscp:0) EVPN NVO Name : nvo1 EVPN VTEP :
vtep_l-102 Source Interface :
Loopback1 Primary IP Interface : Not
Configured External IP Interface : Not Configured Leaf-101#
|
Example 7-9:
Leaf-101 & Leaf-102- VXLAN Interfaces.
The EVPN NVO Name identifies the EVPN Network Virtualization Overlay
(NVO) configuration associated with the VTEP. In this example, both VTEPs use
the NVO named nvo1. The EVPN VTEP field identifies the local VTEP associated
with that NVO. For example, on Leaf-101, nvo1 uses vtep_l-101, whose source IP
address is 1.1.101.101 and whose source interface is Loopback1.
The following example shows the configuration and operational state
of EVPN VNI 10010 on both VTEPs. The VNI is an L2 VNI associated with VLAN 10.
The output also shows the local and remote VTEP addresses. The flood: HER entry
indicates that BUM traffic for VNI 10010 uses Head-End Replication. For
example, when Leaf-101 receives a broadcast frame on VLAN 10, it can replicate
the frame toward the remote VTEP 1.1.102.102 using VNI 10010. The resulting
VXLAN packet uses 1.1.101.101 as the outer source IP address and 1.1.102.102 as
the outer destination IP address.
|
Leaf-101# show
evpn vni detail VNI: 10010 Type: L2 Tenant VRF: default Client State: Up VxLAN interface:
vtep_l-101-10 VxLAN ifIndex: 79 SVI interface: Vlan10 SVI ifIndex: 54 Local VTEP IP: 1.1.101.101 Local external VTEP IP:
0.0.0.0 VxLAN external interface:
unknown Mcast group: 0.0.0.0 Remote VTEPs for this VNI: 1.1.102.102 flood: HER External: 0, Label: 10010 Kernel Add: Success, Add
ReAttempt:0 Number of MACs (local and
remote) known for this VNI: 0 Number of ARPs (IPv4 and
IPv6, local and remote) known for this VNI: 0 Advertise-gw-macip: No Advertise-svi-macip: No
|
Leaf-102# show
evpn vni detail VNI: 10010 Type: L2 Tenant VRF: default Client State: Up VxLAN interface:
vtep_l-102-10 VxLAN ifIndex: 79 SVI interface: Vlan10 SVI ifIndex: 76 Local VTEP IP: 1.1.102.102 Local external VTEP IP:
0.0.0.0 VxLAN external interface:
unknown Mcast group: 0.0.0.0 Remote VTEPs for this VNI: 1.1.101.101 flood: HER External: 0, Label: 10010 Kernel Add: Success, Add
ReAttempt:0 Number of MACs (local and
remote) known for this VNI: 0 Number of ARPs (IPv4 and
IPv6, local and remote) known for this VNI: 0 Advertise-gw-macip: No Advertise-svi-macip: No |
Example 7-10: Leaf-101 & Leaf-102- EVPN 10010.
Example 7-11
verifies that Leaf-101 has established an operational VXLAN tunnel to the
remote VTEP at 1.1.102.102.
Leaf-101# show
vxlan tunnel
Name SIP
DIP source
Group D-VNI operstatus
====
=== === ======
===== ===== ==========
EVPN_1.1.102.102 1.1.101.101
1.1.102.102 EVPN internal
no oper_up
Example 7-11: Leaf-101 - VXLAN Tunnel with VTEP Leaf-102.
Finally, Example
7-12 verifies the VLAN-to-VNI mapping on Leaf-101. VLAN 10 is mapped to VNI
10010. The same mapping is configured on Leaf-102.
Leaf-101# show
vxlan vlanvnimap
VLAN
VNI
====
===
Vlan10
10010
Example 7-10: Leaf-102 & Leaf-102- EVPN 10010.
BGP L2VPN EVPN - VNI Advertisement
Next, we verify the reachability
information pipeline for Host-1 (MAC 00:50:79:66:68:00, IP 10.0.10.101) from
the local VTEP, Leaf-101, toward the remote VTEP, Leaf-102. We start by
verifying the BGP EVPN session that provides the control-plane connection
between the VTEPs.
Example 7-11
shows that Leaf-101 has an established BGP session with Spine-11 for exchanging
L2VPN EVPN NLRI. The session uses Ethernet0 as the underlying interface, and
the peer ASN is 65011. The PfxRcd value of 1 indicates that Leaf-101 has
received one EVPN NLRI from the peer.
Leaf-101# show
bgp l2vpn evpn summary
BGP router
identifier 1.1.1.101, local AS number 65101 VRF default
Neighbor V
AS MsgRcvd MsgSent
InQ OutQ Up/Down State/PfxRcd
Ethernet0 4
65011 128 130
0 0 00:09:50 1
Total number of
neighbors 1
Total number of
neighbors established 1
Example 7-11: Leaf-101 - BGP EVPN neighbors.
Example 7-12 verifies that Leaf-101 has learned
the local Host-1 MAC address 00:50:79:66:68:00 in the MAC address table for
VLAN 10. The MAC address was learned dynamically from a frame received on the
local interface Ethernet2.
The table
also contains an entry for the remote Host-2 MAC address, but at this stage we
focus on Host-1. When Host-1 starts, it may transmit an ARP probe to check
whether its IPv4 address is already in use. This provides an example of a frame
from which Leaf-101 can learn Host-1's source MAC address. The MAC-learning
process itself is independent of the purpose of the ARP frame: Leaf-101 learns
the MAC address from the source MAC address of the ingress Ethernet frame and
associates it with Ethernet2 in VLAN 10.
Leaf-101# show
mac address-table
-----------------------------------------------------------
VLAN MAC-ADDRESS TYPE INTERFACE
-----------------------------------------------------------
10 00:50:79:66:68:01 DYNAMIC
VxLAN DIP: 1.1.102.102
10 00:50:79:66:68:00 DYNAMIC
Ethernet2
Leaf-101#
Example 7-12: Leaf-101 - MAC Address Table.
The ARP table is
also updated with the IP-to-MAC information for the local and remote hosts.
Leaf-101# show
ip arp
Type: R - Remote
Neighbor entries (EVPN or MCLAG Separate IP)
--------------------------------------------------------------------------------------
Address Hardware address Interface Egress Interface Type Action
--------------------------------------------------------------------------------------
10.0.10.101 00:50:79:66:68:00 Vlan10
Ethernet2 Dynamic Fwd
10.0.10.102 00:50:79:66:68:01 Vlan10
VxLAN DIP: 1.1.102.102 Dynamic(R) Fwd
169.254.0.1 0c:22:34:b6:00:0a Ethernet0 - Static Fwd
Example 7-13
Leaf-101 - IP ARP Address Table.
Example 7-14 shows the EVPN routes currently
present in the BGP table on Leaf-101. The entries are grouped by BGP Route
Distinguisher (RD). Leaf-101 uses 1.1.1.101:10 for EVPN routes originated
locally, while routes received from Leaf-102 are grouped under 1.1.1.102:10.
In this SONiC implementation, the RD is
derived from the BGP router ID. The RD makes otherwise identical EVPN routes
unique in the BGP table. This allows the same MAC or IP information to be used
in different EVPN instances without the routes being treated as the same BGP
route.
The output contains two EVPN route types
relevant to this example. Route Type 2 is a MAC/IP Advertisement route and
carries the reachability information for an individual MAC address, optionally
together with its IP address. Route Type 3 is an Inclusive Multicast Ethernet
Tag (IMET) route. It advertises the information required to establish the
replication list for BUM traffic.
In this example, each VTEP has originated a
Type-3 route. Leaf-101 advertises a Type-3 route with origin IP 1.1.101.101,
while Leaf-102 advertises one with origin IP 1.1.102.102. Because the VTEPs use
Head-End Replication, these routes provide the EVPN control-plane information
used to identify the remote VTEPs that participate in BUM forwarding for VNI
10010.
The PMSI Tunnel Attribute associated with an
EVPN Type-3 route specifies how the receiving VTEP should reach the originating
VTEP for multi-destination traffic. With ingress replication, the receiving
VTEP uses the originating VTEP as a replication destination and sends the BUM
frame as a separate VXLAN packet.
For example, Leaf-101 receives the Type-3
route originated by Leaf-102. Together with the associated EVPN information,
this tells Leaf-101 that Leaf-102 participates in the BUM forwarding domain for
VNI 10010. This corresponds to the 1.1.102.102 flood: HER entry previously
shown by show evpn vni detail.
Leaf-101# show
bgp l2vpn evpn route
BGP table version
is 3, local router ID is 1.1.1.101
Status codes: s
suppressed, d damped, h history, * valid, > best, i - internal, q queued, r
RIB-failure
Origin codes: i -
IGP, e - EGP, ? - incomplete
EVPN type-1
prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2
prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3
prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4
prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5
prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 1.1.1.101:10
*> [2]:[0]:[48]:[00:50:79:66:68:00]
1.1.101.101 32768 i
ET:8 RT:65101:10010
*>
[2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101]
1.1.101.101 32768 i
ET:8 RT:65101:10010
*> [3]:[0]:[32]:[1.1.101.101]
1.1.101.101 32768 i
ET:8 RT:65101:10010
Route Distinguisher: 1.1.1.102:10
*> [2]:[0]:[48]:[00:50:79:66:68:01]
1.1.102.102 0 65011
65102 i
RT:65102:10010 ET:8
*>
[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102]
1.1.102.102 0 65011
65102 i
RT:65102:10010 ET:8
*> [3]:[0]:[32]:[1.1.102.102]
1.1.102.102 0 65011
65102 i
RT:65102:10010 ET:8
Displayed 6
prefixes (6 paths)
Leaf-101#
Example 7-14
Leaf-101 - BGP Table.
EVPN Type-2 MAC/IP Advertisement Routes
Example 7-15 shows each entry in detail, including the EVPN route
type, the advertised MAC and IP information, the VNI, the BGP next hop, and the
extended communities.
The first Host-1 entry is [2]:[0]:[48]:[00:50:79:66:68:00]. This is
an EVPN Type-2 MAC Advertisement route containing only the MAC address. The [2]
identifies the route as Type 2, [0] is the Ethernet Tag ID, [48] specifies a
48-bit MAC address, and 00:50:79:66:68:00 is the MAC address of Host-1. The
route is originated locally by Leaf-101. Its BGP next hop is 1.1.101.101, which
is the VTEP address of Leaf-101. The output also shows VNI 10010, associating
the MAC address with the VXLAN segment corresponding to VLAN 10.
The second Host-1 entry is [2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101].
This is an EVPN Type-2 MAC/IP Advertisement route. It contains the same MAC
address together with Host-1's IPv4 address. The [32] specifies a 32-bit IP
address, followed by 10.0.10.101.
The MAC/IP route provides the IP-to-MAC binding that a remote VTEP
can use for ARP suppression. Instead of flooding an ARP request through the
VXLAN fabric, a VTEP can use the EVPN-learned binding to respond locally when
the requested IP address is known.
Therefore, Leaf-101 has two Type-2 routes for Host-1: one advertises
the MAC address alone, while the other associates the MAC address with Host-1's
IPv4 address. Both routes are locally originated and use 1.1.101.101 as the BGP
next hop.
The extended community contains RT:65101:10010, which is the route
target associated with this EVPN route, and ET:8, which identifies VXLAN as the
encapsulation type.
EVPN Type-3 IMET Routes
The Type-3 route is [3]:[0]:[32]:[1.1.101.101].
The [3] identifies an EVPN Type-3 Inclusive Multicast Ethernet Tag (IMET)
route. The origin IP 1.1.101.101 identifies Leaf-101 as the originating
VTEP. Unlike the Type-2 routes, this route does not identify an individual
host. Instead, it advertises the VTEP's participation in the EVPN segment and
provides information used for multi-destination forwarding.
The IMET route carries a PMSI
Tunnel Attribute, which specifies how multi-destination traffic should
reach the originating VTEP. The detailed output shows: PMSI Tunnel Type:
Ingress Replication, label: 10010
This indicates that Leaf-101 uses
Ingress Replication for VNI 10010. In this lab, this corresponds to the
previously observed flood: HER setting. Leaf-101 can therefore replicate a BUM
frame to the remote VTEPs participating in VNI 10010.
Leaf-102 advertises the
corresponding Type-3 IMET route [3]:[0]:[32]:[1.1.102.102]. Here, 1.1.102.102
identifies Leaf-102 as the originating VTEP. Its IMET route also carries a PMSI
Tunnel Attribute specifying Ingress Replication with label 10010. Leaf-101 can
therefore use this EVPN information to include Leaf-102 as a replication destination
for BUM traffic in VNI 10010. The remaining Type-2 routes in the output follow
the same format. The routes under RD 1.1.1.102:10 describe MAC addresses
advertised by Leaf-102. For example, the route containing 00:50:79:66:68:01
identifies Host-2, with 1.1.102.102 as the BGP next hop. These routes are
received by Leaf-101 through Spine-11.
Leaf-101# show
bgp l2vpn evpn route detail
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]
Paths: (0
available, no best path)
Not advertised to any peer
Route
Distinguisher: 1.1.1.101:10
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[00:50:79:66:68:00] VNI
10010
Local
1.1.101.101 from 0.0.0.0 (1.1.1.101)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65101:10010
SubType: 1 Last update: Thu Sep 10
05:41:42 2026
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route
[2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101] VNI 10010
Local
1.1.101.101 from 0.0.0.0 (1.1.1.101)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65101:10010
SubType: 1 Last update: Thu Sep 10
05:41:42 2026
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[0c:cb:81:19:00:03]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[0c:cb:81:19:00:03] VNI
10010
Local
1.1.101.101 from 0.0.0.0 (1.1.1.101)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65101:10010
SubType: 1 Last update: Thu Sep 10
05:41:42 2026
BGP routing table
entry for 1.1.1.101:10:[3]:[0]:[32]:[1.1.101.101]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [3]:[0]:[32]:[1.1.101.101] VNI 10010
Local
1.1.101.101 from 0.0.0.0 (1.1.1.101)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65101:10010
SubType: 1 Last update: Thu Sep 10
05:41:42 2026
PMSI Tunnel Type: Ingress Replication,
label: 10010
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]
Paths: (0
available, no best path)
Not advertised to any peer
Route
Distinguisher: 1.1.1.102:10
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[00:50:79:66:68:01] VNI
10010
65011 65102
1.1.102.102 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:29 2026
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route
[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102] VNI 10010
65011 65102
1.1.102.102 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:29 2026
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[0c:e8:78:22:00:03]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[0c:e8:78:22:00:03] VNI
10010
65011 65102
1.1.102.102 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:29 2026
BGP routing table
entry for 1.1.1.102:10:[3]:[0]:[32]:[1.1.102.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [3]:[0]:[32]:[1.1.102.102]
65011 65102
1.1.102.102 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:29 2026
PMSI Tunnel Type: Ingress Replication,
label: 10010
Displayed 8
prefixes (8 paths)
Leaf-101#
Example 7-15: Leaf-101 BGP Table.
Example 7-16 shows all EVPN routes
in the BGP Loc-RIB on Spine-11. We keep our focus on Host-1 reachability
information. For the Host-1 MAC/IP Advertisement route, Spine-11 has learned
the route from Leaf-101 over Ethernet0. The BGP entry shows the following AS
path 65101 and EVPN next hop 1.1.101.101 from Ethernet0 (1.1.1.101)
The EVPN
next hop 1.1.101.101 is the VTEP address of the originating Leaf-101. The EVPN
NLRI identifies Host-1 by its MAC address 00:50:79:66:68:00, IPv4 address 10.0.10.101,
and VNI 10010.
Spine-11# show
bgp l2vpn evpn route detail
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]
Paths: (0
available, no best path)
Not advertised to any peer
Route
Distinguisher: 1.1.1.101:10
BGP routing table entry for
1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]
Paths: (1 available, best #1)
Advertised to
non peer-group peers:
Ethernet0
Ethernet1
Route
[2]:[0]:[48]:[00:50:79:66:68:00] VNI 10010
65101
1.1.101.101
from Ethernet0 (1.1.1.101)
Origin
IGP, valid, external, best (First path received)
Extended
Community: RT:65101:10010 ET:8
SubType:
0 Last update: Thu Sep 10 05:49:30 2026
BGP routing table entry for
1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101]
Paths: (1 available, best #1)
Advertised to
non peer-group peers:
Ethernet0
Ethernet1
Route
[2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101] VNI 10010
65101
1.1.101.101
from Ethernet0 (1.1.1.101)
Origin
IGP, valid, external, best (First path received)
Extended
Community: RT:65101:10010 ET:8
SubType:
0 Last update: Thu Sep 10 05:49:30 2026
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[0c:cb:81:19:00:03]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0 Ethernet1
Route [2]:[0]:[48]:[0c:cb:81:19:00:03] VNI
10010
65101
1.1.101.101 from Ethernet0 (1.1.1.101)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65101:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:30 2026
BGP routing table entry for
1.1.1.101:10:[3]:[0]:[32]:[1.1.101.101]
Paths: (1 available, best #1)
Advertised to
non peer-group peers:
Ethernet0
Ethernet1
Route
[3]:[0]:[32]:[1.1.101.101]
65101
1.1.101.101
from Ethernet0 (1.1.1.101)
Origin
IGP, valid, external, best (First path received)
Extended
Community: RT:65101:10010 ET:8
SubType:
0 Last update: Thu Sep 10 05:49:30 2026
PMSI
Tunnel Type: Ingress Replication, label: 10010
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]
Paths: (0
available, no best path)
Not advertised to any peer
Route
Distinguisher: 1.1.1.102:10
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0 Ethernet1
Route [2]:[0]:[48]:[00:50:79:66:68:01] VNI
10010
65102
1.1.102.102 from Ethernet1 (1.1.1.102)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:30 2026
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0 Ethernet1
Route
[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102] VNI 10010
65102
1.1.102.102 from Ethernet1 (1.1.1.102)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:30 2026
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[0c:e8:78:22:00:03]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0 Ethernet1
Route [2]:[0]:[48]:[0c:e8:78:22:00:03] VNI
10010
65102
1.1.102.102 from Ethernet1 (1.1.1.102)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:30 2026
BGP routing table
entry for 1.1.1.102:10:[3]:[0]:[32]:[1.1.102.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0 Ethernet1
Route [3]:[0]:[32]:[1.1.102.102]
65102
1.1.102.102 from Ethernet1 (1.1.1.102)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:30 2026
PMSI Tunnel Type: Ingress Replication,
label: 10010
Displayed 8
prefixes (8 paths)
Spine-11#
Example 7-16: Spine-11 BGP Table.
Example 7-17 shows the EVPN routes
in the BGP Loc-RIB on Leaf-102. We again focus on the Host-1 MAC/IP
Advertisement route. Leaf-102 receives the route from Spine-11 over Ethernet0.
The BGP AS_PATH Path Attribute now contains both AS 65011, added by Spine-11,
and the originating AS 65101 of Leaf-101. The route is learned from the BGP UPDATE
message received from the BGP peer over Ethernet0. However, the EVPN next hop
remains 1.1.101.101, the VTEP address of Leaf-101. This address is then used as
the VXLAN tunnel destination and appears as the destination address in the
outer IP header when Leaf-102 sends traffic to Host-1.
Leaf-102# show
bgp l2vpn evpn route detail
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]
Paths: (0
available, no best path)
Not advertised to any peer
Route
Distinguisher: 1.1.1.101:10
BGP routing table entry for
1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]
Paths: (1 available, best #1)
Advertised to
non peer-group peers:
Ethernet0
Route
[2]:[0]:[48]:[00:50:79:66:68:00] VNI 10010
65011 65101
1.1.101.101
from Ethernet0 (1.1.1.11)
Origin
IGP, valid, external, best (First path received)
Extended
Community: RT:65101:10010 ET:8
SubType:
0 Last update: Thu Sep 10 05:49:29 2026
BGP routing table entry for
1.1.1.101:10:[2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101]
Paths: (1 available, best #1)
Advertised to
non peer-group peers:
Ethernet0
Route
[2]:[0]:[48]:[00:50:79:66:68:00]:[32]:[10.0.10.101] VNI 10010
65011 65101
1.1.101.101
from Ethernet0 (1.1.1.11)
Origin
IGP, valid, external, best (First path received)
Extended
Community: RT:65101:10010 ET:8
SubType:
0 Last update: Thu Sep 10 05:49:29 2026
BGP routing table
entry for 1.1.1.101:10:[2]:[0]:[48]:[0c:cb:81:19:00:03]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[0c:cb:81:19:00:03] VNI
10010
65011 65101
1.1.101.101 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65101:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:29 2026
BGP routing table
entry for 1.1.1.101:10:[3]:[0]:[32]:[1.1.101.101]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [3]:[0]:[32]:[1.1.101.101]
65011 65101
1.1.101.101 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65101:10010 ET:8
SubType: 0 Last update: Thu Sep 10
05:49:29 2026
PMSI Tunnel Type: Ingress Replication,
label: 10010
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:00:00:00:00:00]
Paths: (0
available, no best path)
Not advertised to any peer
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]
Paths: (0
available, no best path)
Not advertised to any peer
Route
Distinguisher: 1.1.1.102:10
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[00:50:79:66:68:01] VNI
10010
Local
1.1.102.102 from 0.0.0.0 (1.1.1.102)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65102:10010
SubType: 1 Last update: Thu Sep 10
05:42:36 2026
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route
[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102] VNI 10010
Local
1.1.102.102 from 0.0.0.0 (1.1.1.102)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65102:10010
SubType: 1 Last update: Thu Sep 10
05:42:36 2026
BGP routing table
entry for 1.1.1.102:10:[2]:[0]:[48]:[0c:e8:78:22:00:03]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[0c:e8:78:22:00:03] VNI
10010
Local
1.1.102.102 from 0.0.0.0 (1.1.1.102)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65102:10010
SubType: 1 Last update: Thu Sep 10
05:42:36 2026
BGP routing table
entry for 1.1.1.102:10:[3]:[0]:[32]:[1.1.102.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [3]:[0]:[32]:[1.1.102.102] VNI 10010
Local
1.1.102.102 from 0.0.0.0 (1.1.1.102)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8 RT:65102:10010
SubType: 1 Last update: Thu Sep 10
05:42:36 2026
PMSI Tunnel Type: Ingress Replication,
label: 10010
Displayed 8
prefixes (8 paths)
Leaf-102#
Example 7-17: Leaf-102 BGP Table.
Example 7-18
shows that the MAC address learned through the EVPN Type-2 MAC Advertisement
route is installed as a remote MAC entry in VLAN 10 on Leaf-102. The EVPN route
carries VNI 10010, but it does not contain the VLAN identifier Vlan10. Leaf-102
resolves the VNI to the local VLAN using the VLAN-to-VNI mapping shown in
Example 7-19.
The MAC table
shows Host-1's MAC address as a remote dynamic entry. Its egress interface is
the VXLAN interface, with the remote VTEP 1.1.101.101 as the VXLAN destination.
Leaf-102# show
mac address-table
-----------------------------------------------------------
VLAN MAC-ADDRESS TYPE INTERFACE
-----------------------------------------------------------
10 00:50:79:66:68:00 DYNAMIC
VxLAN DIP: 1.1.101.101
10 00:50:79:66:68:01 DYNAMIC
Ethernet2
Example 7-18: Leaf-102 VLAN 10 MAC Address Table.
Leaf-101# show
vxlan vlanvnimap
VLAN
VNI
====
===
Vlan10
10010
Example 7-19: Leaf-102 VLAN-to-VNI Mapping.
Example 7-20
verifies that the IP-to-MAC information from the EVPN MAC/IP Advertisement
route is also present in the ARP table.
Leaf-102# show ip
arp
Type: R - Remote
Neighbor entries (EVPN or MCLAG Separate IP)
--------------------------------------------------------------------------------------
Address Hardware address Interface Egress Interface Type Action
-------------------------------------------------------------------------------------
10.0.10.101 00:50:79:66:68:00 Vlan10
VxLAN DIP: 1.1.101.101 Dynamic(R)
Fwd
10.0.10.102 00:50:79:66:68:01 Vlan10
Ethernet2 Dynamic Fwd
169.254.0.1 0c:22:34:b6:00:0a Ethernet0 - Static Fwd
Leaf-102#
Example 7-19: Leaf-102 IP ARP Table.
Data Plane
The following
four examples show the packet exchange when Host-1 sends an ICMP echo request
to Host-2. The captures show both the ARP resolution and the subsequent ICMP
exchange carried over VXLAN VNI 10010. Note, only relevant fields are shown.
1. ARP request
Host-1 sends
an ARP request for Host-2's IP address. Because the destination MAC address of
the original Ethernet frame is the Ethernet broadcast address
(ff:ff:ff:ff:ff:ff), the request is BUM traffic. Leaf-101 therefore uses
ingress replication, also called head-end replication (HER), to forward the
frame to the remote VTEPs participating in VNI 10010.
Example 7-10
on page 190 shows that Leaf-101 has 1.1.102.102 as a remote VTEP for VNI 10010,
with flood: HER. Leaf-102 has the corresponding entry for 1.1.101.101. In this
two-VTEP example, Leaf-101 therefore creates a VXLAN packet with VNI 10010 and
sends the ARP broadcast to the Leaf-102 VTEP at 1.1.102.102.
Although the
original Ethernet frame is a broadcast frame, the VXLAN packet carrying it is
sent across the fabric as a unicast packet between the VTEPs. The outer IP
header therefore contains 1.1.101.101 as the source and 1.1.102.102 as the
destination, while the broadcast destination is preserved in the encapsulated
Ethernet frame.
When Leaf-102
receives the VXLAN packet, it decapsulates the outer Ethernet, IP, UDP, and
VXLAN headers. The VNI identifies the VXLAN segment, which is mapped to VLAN 10
on Leaf-102. The original Ethernet broadcast frame is then forwarded to the
local ports belonging to VLAN 10.
Ethernet II
Destination: 0c:22:34:b6:00:0a
Source: 0c:cb:81:19:00:0a
Type: IPv4 (0x0800)
Internet Protocol
Version 4
Source: 1.1.101.101
Destination: 1.1.102.102
Version: 4
User Datagram
Protocol
Source Port: 54679
Destination Port: 4789
VXLAN
VNI: 10010
Ethernet II
Destination: ff:ff:ff:ff:ff:ff
Source: 00:50:79:66:68:00
Type: ARP (0x0806)
Address
Resolution Protocol
Hardware Type: Ethernet (1)
Protocol Type: IPv4 (0x0800)
Opcode: Request (1)
Sender MAC: 00:50:79:66:68:00
Sender IP: 10.0.10.101
Target MAC: ff:ff:ff:ff:ff:ff
Target IP: 10.0.10.102
Example 7-20: ARP Request from Host-1.
Figure 7-6 illustrates how VTEP Leaf-101
processes the ARP request received on Ethernet2, which is bound to VLAN 10.
Based on the VLAN-to-VNI mapping, Leaf-101 determines that VLAN 10 is
associated with VNI 10010. Because the destination MAC address is the Ethernet
broadcast address, Leaf-101 uses the HER VTEP list for VNI 10010 to determine
the remote VTEPs to which the frame must be replicated. It then encapsulates
the original Ethernet frame with the outer Ethernet, IP, UDP, and VXLAN headers
and sends the resulting VXLAN packet to each remote VTEP.
Figure 7-5: ARP
Request from Host-1.
2. ARP reply
Host-2
responds with its MAC address. The response is encapsulated by Leaf-102 and
sent back to the Leaf-101 VTEP.
Ethernet II
Destination: 0c:cb:81:19:00:0a
Source: 0c:22:34:b6:00:0a
Type: IPv4 (0x0800)
Internet Protocol
Version 4
Source: 1.1.102.102
Destination: 1.1.101.101
Version: 4
User Datagram
Protocol
Source Port: 54770
Destination Port: 4789
VXLAN
VNI: 10010
Ethernet II
Destination: 00:50:79:66:68:00
Source: 00:50:79:66:68:01
Type: ARP (0x0806)
Address
Resolution Protocol
Hardware Type: Ethernet (1)
Protocol Type: IPv4 (0x0800)
Opcode: Reply (2)
Sender MAC: 00:50:79:66:68:01
Sender IP: 10.0.10.102
Target MAC: 00:50:79:66:68:00
Target IP: 10.0.10.101
Example 7-21: ARP Reply from Host-2.
3. ICMP echo request
After the ARP exchange, Host-1 sends an ICMP
echo request to Host-2. The destination MAC address is now known, so the
Ethernet frame is unicast. Leaf-101 forwards the frame through the VXLAN tunnel
to Leaf-102 using VNI 10010.
Ethernet II
Destination: 0c:22:34:b6:00:0a
Source: 0c:cb:81:19:00:0a
Type: IPv4 (0x0800)
Internet Protocol
Version 4
Source: 1.1.101.101
Destination: 1.1.102.102
Version: 4
User Datagram
Protocol
Source Port: 39133
Destination Port: 4789
VXLAN
VNI: 10010
Ethernet II
Destination: 00:50:79:66:68:01
Source: 00:50:79:66:68:00
Type: IPv4 (0x0800)
Internet Protocol
Version 4
Source: 10.0.10.101
Destination: 10.0.10.102
Version: 4
Internet Control
Message Protocol
Type: Echo request (8)
Identifier: 28443
Sequence number: 1
Example 7-22: ICMP Echo Request from Host-1 to Host-2.
4. ICMP echo reply
Host-2 responds with an ICMP echo reply.
Leaf-102 sends the unicast frame through the VXLAN tunnel to Leaf-101 using VNI
10010.
Ethernet II
Destination: 0c:cb:81:19:00:0a
Source: 0c:22:34:b6:00:0a
Type: IPv4 (0x0800)
Internet Protocol
Version 4
Source: 1.1.102.102
Destination: 1.1.101.101
Version: 4
User Datagram
Protocol
Source Port: 43054
Destination Port: 4789
VXLAN
VNI: 10010
Ethernet II
Destination: 00:50:79:66:68:00
Source: 00:50:79:66:68:01
Type: IPv4 (0x0800)
Internet Protocol
Version 4
Source: 10.0.10.102
Destination: 10.0.10.101
Version: 4
Internet Control
Message Protocol
Type: Echo reply (0)
Identifier: 28443
Sequence number: 1
Example 7-23: ICMP Echo Reply from Host-2 to Host-1.
Database Updates
CONFIG_DB
Figure 7-6 shows an example sonic-cli configuration for enabling the
BGP EVPN address family and configuring the local VXLAN VTEP. The BGP
configuration creates the BGP_GLOBALS_AF and BGP_NEIGHBOR_AF entries in
CONFIG_DB, including the l2vpn_evpn address family and their field/value pairs
shown in the figure. The frrcfgd daemon consumes these entries and translates
them into FRR configuration commands, which are applied to the running FRR
routing stack. These BGP-specific configuration entries are not copied to
APPL_DB.
The VXLAN_EVPN_NVO entry identifies the Network Virtualization
Overlay (NVO) and specifies its source VTEP through the source_vtep field. The
referenced VXLAN_TUNNEL entry defines the local VTEP source IP address, while
the VXLAN_TUNNEL_MAP entry defines the local mapping between VLAN 10 and VNI
10010.
The VLAN-to-VNI mapping is local to each VTEP. Different VTEPs can
therefore use different local VLAN identifiers for the same Layer-2 segment,
provided that the local VLANs are mapped to the same VNI. The VXLAN-specific
configuration entries are consumed by VxlanMgr, which produces the
corresponding entries in APPL_DB.
APPL_DB - Local Upates
Figure 7-6 also shows the corresponding VXLAN entries created in
APPL_DB. VxlanMgr, running in the SWSS container, consumes the VXLAN_EVPN_NVO,
VXLAN_TUNNEL, and VXLAN_TUNNEL_MAP entries from CONFIG_DB and produces their
corresponding entries in APPL_DB. The entries describe the local NVO, the local
VTEP tunnel, and the VLAN-to-VNI mapping.
Figure 7-7 also illustrates how the corresponding APPL_DB VXLAN
tables are consumed by VxlanOrch and FdbOrch, which produce the required SAI
objects in ASIC_DB. VxlanOrch processes the VXLAN tunnel and remote VNI
information, while FdbOrch processes the remote MAC information. These
components are included in Figure 7-7 to show the complete Redis database
pipeline for BGP EVPN VXLAN configuration.
Figure 7-6: BGP
EVPN w. VXLAN: CONFIG_DB & APPL_DB Pipeline.
APPL_DB - EVPN Route
Updates
In addition to the locally
configured VXLAN_EVPN_NVO_TABLE, VXLAN_TUNNEL_TABLE, and VXLAN_TUNNEL_MAP_TABLE
entries, APPL_DB is populated with remote VXLAN information derived from
received BGP EVPN routes. Figure 7-7 shows an example in which Leaf-101 receives
a BGP UPDATE from its EVPN peer, Spine-11. The UPDATE contains three EVPN NLRIs
originated by Leaf-102: an EVPN Type-3 Inclusive Multicast Ethernet Tag (IMET)
route, a Type-2 MAC Advertisement route, and a Type-2 MAC-IP Advertisement
route.
The
Type-3 route:
Route [3]:[0]:[32]:[1.1.102.102]
PMSI Tunnel Type: Ingress
Replication, label: 10010
identifies Leaf-102's VTEP address
1.1.102.102 and carries VNI 10010 in the PMSI Tunnel attribute. The route is
used to establish the remote VTEP information required for BUM traffic
replication.
The
Type-2 MAC route identifies Host-2's MAC address:
Route [2]:[0]:[48]:[00:50:79:66:68:01] VNI 10010
The
Type-2 MAC-IP route carries the same MAC address together with Host-2's IPv4
address:
Route [2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.102] VNI 10010
After BGP path selection, Zebra
installs the selected remote EVPN information into the Linux kernel. The IMET
and MAC information is reflected in the Linux forwarding database, while the
MAC-IP binding is installed in the Linux neighbor table. fdbsyncd monitors the
relevant Linux FDB information and synchronizes the remote VTEP and MAC
information to APPL_DB. The IMET route therefore results in a VXLAN_REMOTE_VNI_TABLE
entry, while the MAC route results in a VXLAN_FDB_TABLE entry. The SONiC
EVPN/VXLAN HLD defines fdbsyncd as the producer of both tables.
The MAC-IP binding follows a
separate path. Zebra installs the remote MAC-IP binding into the Linux neighbor
table, and the Linux kernel generates a Netlink neighbor notification.
neighsyncd receives the notification and synchronizes the binding to NEIGH_TABLE
in APPL_DB. The resulting entry associates Host-2's IP address 10.0.10.102 with
MAC address 00:50:79:66:68:01.
The neigh field binds the remote IP address to Host-2's MAC address.
The family field identifies IPv4, while proto value 11 identifies Zebra as the
source of the kernel entry. The remote field indicates that the neighbor was
learned remotely.
Figure 7-7: BGP
EVPN w. VXLAN: CONFIG_DB & APPL_DB Pipeline.
ASIC_DB: Packet Processing Pipeline for Received Frames
Instead of describing every ASIC_DB object and attribute related to
VXLAN tunneling separately, this section focuses on the relationships and
dependencies between the objects. The complete ASIC_DB entries are shown at the
end of the section (Example 7-24) to allow the relationships described here to
be mapped to the actual objects and attributes.
For better readability, the common ASIC_STATE:SAI_OBJECT_TYPE_ prefix and object identifiers (oid:*) are omitted. Object attributes are also shown without the
object-specific prefix. For example, ASIC_STATE:SAI_OBJECT_ TYPE_TUNNEL and its SAI_TUNNEL_ATTR_DECAP_MAPPERS attribute are referred to as the TUNNEL object and its DECAP_MAPPERS
attribute.
We start by examining the ASIC_DB objects
involved in VXLAN tunnel decapsulation when Leaf-101 receives an Ethernet frame
from remote Host-2 (00:50:79:66:68:01) destined
for local Host-1 (00:50:79:66:68:00).
The TUNNEL
object defines the tunnel as VXLAN through its TUNNEL_TYPE attribute. Its DECAP_MAPPERS
attribute references two TUNNEL_MAP
objects that provide the mappings required during decapsulation.
The first TUNNEL_MAP object provides the VNI-to-VLAN mapping. Its TYPE attribute defines the mapping operation as VNI_TO_VLAN_ID. The
corresponding TUNNEL_MAP_ENTRY object contains
the actual mapping through its VLAN_ID_VALUE
and VNI_KEY attributes. The TUNNEL_MAP attribute of the entry associates it with the TUNNEL_MAP object.
The second TUNNEL_MAP object provides the VNI-to-virtual-router mapping. Its TYPE attribute defines this mapping operation. This mapping associates
the incoming VXLAN VNI with the virtual router context used for further
processing.
The TUNNEL
object's UNDERLAY_INTERFACE attribute references
to a ROUTER_INTERFACE object. This
interface is a loopback interface with an MTU of 9100. Its VIRTUAL_ROUTER_ID attribute references the corresponding VIRTUAL_ROUTER object.
The BRIDGE_PORT object, associated with the TUNNEL object through its TUNNEL_ID attribute, also defines that the
source MAC address carried in the inner Ethernet frame is not learned in the
VLAN 10 MAC address table through the FDB_LEARNING_MODE attribute. This is the standard MAC learning behavior for the VXLAN
configuration used here: remote MAC addresses are learned through the control
plane from BGP EVPN updates rather than through data-plane MAC learning.
After the VXLAN tunnel headers are
removed—the outer Ethernet and IP headers, UDP header, and VXLAN header—the
ASIC uses the Layer-2 forwarding information represented by the FDB_ENTRY object. In addition to the destination MAC address, the FDB_ENTRY object references the VLAN object through its bvid entry (\"bvid\":). The VLAN
object identifies the corresponding VLAN through its VLAN_ID attribute.
The egress interface for the destination MAC
address is resolved through the PORT_ID
attribute of the FDB_ENTRY object. This
attribute references the PORT object
representing the switch port. The associated HOSTIF object identifies the interface as Ethernet2 through its NAME attribute and is associated with the PORT object through its OBJ_ID
attribute. The VLAN_MEMBER object defines
how frames are transmitted through this port. Its VLAN_TAGGING_MODE attribute is set to UNTAGGED, indicating that Ethernet2 operates as
an access port and the VLAN tag is removed before the frame is transmitted to
Host-1.
"ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL:oid:0x2a000000000a77":
{
"value": {
"SAI_TUNNEL_ATTR_DECAP_MAPPERS":
"2:oid:0x29000000000a6d,oid:0x29000000000a6f",
"SAI_TUNNEL_ATTR_ENCAP_DST_IP":
"1.1.102.102",
"SAI_TUNNEL_ATTR_ENCAP_MAPPERS":
"2:oid:0x29000000000a6e,oid:0x29000000000a70",
"SAI_TUNNEL_ATTR_ENCAP_SRC_IP":
"1.1.101.101",
"SAI_TUNNEL_ATTR_PEER_MODE":
"SAI_TUNNEL_PEER_MODE_P2P",
"SAI_TUNNEL_ATTR_TYPE":
"SAI_TUNNEL_TYPE_VXLAN",
"SAI_TUNNEL_ATTR_UNDERLAY_INTERFACE":
"oid:0x60000000009c7"
"ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE_PORT:oid:0x3a000000000a78":
{
"value": {
"SAI_BRIDGE_PORT_ATTR_ADMIN_STATE": "true",
"SAI_BRIDGE_PORT_ATTR_BRIDGE_ID":
"oid:0x3900000000098e",
"SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE":
"SAI_BRIDGE_PORT_FDB_LEARNING_MODE_DISABLE",
"SAI_BRIDGE_PORT_ATTR_TUNNEL_ID":
"oid:0x2a000000000a77",
"SAI_BRIDGE_PORT_ATTR_TYPE":
"SAI_BRIDGE_PORT_TYPE_TUNNEL"
"ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY:oid:0x3b000000000a73":
{
"value": {
"SAI_TUNNEL_MAP_ENTRY_ATTR_TUNNEL_MAP":
"oid:0x29000000000a6d",
"SAI_TUNNEL_MAP_ENTRY_ATTR_TUNNEL_MAP_TYPE":
"SAI_TUNNEL_MAP_TYPE_VNI_TO_VLAN_ID",
"SAI_TUNNEL_MAP_ENTRY_ATTR_VLAN_ID_VALUE": "10",
"SAI_TUNNEL_MAP_ENTRY_ATTR_VNI_ID_KEY": "10010"
"ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL_MAP:oid:0x29000000000a6d":
{
"value": {
"SAI_TUNNEL_MAP_ATTR_TYPE":
"SAI_TUNNEL_MAP_TYPE_VNI_TO_VLAN_ID"
"ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE:oid:0x3900000000098e":
{
"value": {
"NULL":
"NULL"
"ASIC_STATE:SAI_OBJECT_TYPE_ROUTER_INTERFACE:oid:0x60000000009c7":
{
"value": {
"SAI_ROUTER_INTERFACE_ATTR_MTU": "9100",
"SAI_ROUTER_INTERFACE_ATTR_TYPE":
"SAI_ROUTER_INTERFACE_TYPE_LOOPBACK",
"SAI_ROUTER_INTERFACE_ATTR_VIRTUAL_ROUTER_ID":
"oid:0x300000000003a"
"ASIC_STATE:SAI_OBJECT_TYPE_VIRTUAL_ROUTER:oid:0x300000000003a":
{
"value": {
"NULL":
"NULL"
"ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL_MAP:oid:0x29000000000a6f":
{
"value": {
"SAI_TUNNEL_MAP_ATTR_TYPE":
"SAI_TUNNEL_MAP_TYPE_VNI_TO_VIRTUAL_ROUTER_ID"
Example 7-7:
ASIC_DB Objects for Ingress Packet Processing Pipeline.
ASIC_DB: Packet Processing Pipeline for Transmitted Frames
The egress
packet-processing pipeline is examined from the point where Leaf-101 has
determined that the destination MAC address belongs to a remote host and the
frame must be forwarded through the VXLAN tunnel to Leaf-102.
The FDB_ENTRY object for the remote Host-2 MAC
address (00:50:79:66:68:01) provides the Layer-2 forwarding information. Its bvid identifies the VLAN through the referenced VLAN
object, which defines VLAN 10 through its VLAN_ID attribute. The BRIDGE_PORT_ID attribute references a BRIDGE_PORT object that identifies the VXLAN
tunnel as the egress interface.
The BRIDGE_PORT object is associated with the VXLAN
TUNNEL object
through its TUNNEL_ID attribute. Its TYPE attribute identifies it as a tunnel bridge
port, and its FDB_LEARNING_MODE is disabled. The FDB_ENTRY also contains the
remote VTEP address 1.1.102.102 in its ENDPOINT_IP attribute. The PACKET_ACTION attribute is set to FORWARD,
allowing the frame to be forwarded through the tunnel.
The
TUNNEL object defines
the tunnel as VXLAN through its TYPE attribute and as a point-to-point tunnel
through PEER_MODE. The ENCAP_SRC_IP and ENCAP_DST_IP attributes define the outer VXLAN
source and destination addresses as 1.1.101.101 and 1.1.102.102, respectively.
The UNDERLAY_INTERFACE attribute references the loopback router
interface used to reach the remote VTEP.
The TUNNEL
object's ENCAP_MAPPERS attribute references two TUNNEL_MAP objects that provide the information required
to construct the VXLAN header. The first TUNNEL_MAP object defines a VLAN-to-VNI
mapping through its TYPE attribute, VLAN_ID_TO_VNI. This mapping converts the local VLAN 10 into
the corresponding VXLAN VNI 10010. The second TUNNEL_MAP object defines a
virtual-router-to-VNI mapping through its TYPE attribute,
VIRTUAL_ROUTER_ID_TO_VNI. Together, these mappings provide the VXLAN VNI
associated with the Layer-2 forwarding domain and virtual-router context.
Based on
these objects, the ASIC encapsulates the Ethernet frame with the VXLAN tunnel
headers. The original Ethernet frame is retained as the inner frame, while an
outer IP/UDP/VXLAN header is added with source VTEP 1.1.101.101, destination
VTEP 1.1.102.102, and VNI 10010. The resulting VXLAN packet is then transmitted
through the underlay toward Leaf-102.
"ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY: {\"bvid\":\"oid:0x26000000000a48\",\"mac\":\"00:50:79:66:68:01\",\"switch_id\":\"oid:0x21000000000000\"}":
{
"value": {
"SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID":
"oid:0x3a000000000a78",
"SAI_FDB_ENTRY_ATTR_ENDPOINT_IP": "1.1.102.102",
"SAI_FDB_ENTRY_ATTR_PACKET_ACTION":
"SAI_PACKET_ACTION_FORWARD",
"SAI_FDB_ENTRY_ATTR_TYPE":
"SAI_FDB_ENTRY_TYPE_STATIC_MACMOVE"’
"ASIC_STATE:SAI_OBJECT_TYPE_VLAN:oid:0x26000000000a48":
{
"value": {
"SAI_VLAN_ATTR_VLAN_ID":
"10"
"ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE_PORT:oid:0x3a000000000a78":
{
"value": {
"SAI_BRIDGE_PORT_ATTR_ADMIN_STATE": "true",
"SAI_BRIDGE_PORT_ATTR_BRIDGE_ID":
"oid:0x3900000000098e",
"SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE":
"SAI_BRIDGE_PORT_FDB_LEARNING_MODE_DISABLE",
"SAI_BRIDGE_PORT_ATTR_TUNNEL_ID":
"oid:0x2a000000000a77",
"SAI_BRIDGE_PORT_ATTR_TYPE":
"SAI_BRIDGE_PORT_TYPE_TUNNEL"
"ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL:oid:0x2a000000000a77":
{
"value": {
"SAI_TUNNEL_ATTR_DECAP_MAPPERS":
"2:oid:0x29000000000a6d,oid:0x29000000000a6f",
"SAI_TUNNEL_ATTR_ENCAP_DST_IP":
"1.1.102.102",
"SAI_TUNNEL_ATTR_ENCAP_MAPPERS":
"2:oid:0x29000000000a6e,oid:0x29000000000a70",
"SAI_TUNNEL_ATTR_ENCAP_SRC_IP":
"1.1.101.101",
"SAI_TUNNEL_ATTR_PEER_MODE":
"SAI_TUNNEL_PEER_MODE_P2P",
"SAI_TUNNEL_ATTR_TYPE":
"SAI_TUNNEL_TYPE_VXLAN",
"SAI_TUNNEL_ATTR_UNDERLAY_INTERFACE":
"oid:0x60000000009c7"
"ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL_MAP:oid:0x29000000000a6e":
{
"value": {
"SAI_TUNNEL_MAP_ATTR_TYPE":
"SAI_TUNNEL_MAP_TYPE_VLAN_ID_TO_VNI"
"ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL_MAP:oid:0x29000000000a70":
{
"value": {
"SAI_TUNNEL_MAP_ATTR_TYPE":
"SAI_TUNNEL_MAP_TYPE_VIRTUAL_ROUTER_ID_TO_VNI"
Example 7-8:
ASIC_DB Objects for Egress Packet Processing Pipeline.
