This chapter introduces EVPN multihoming in
SONiC. It examines five issues that arise when a host or downstream switch is
multihomed to multiple VTEPs: Ethernet Segment configuration, Ethernet Segment
discovery, Designated Forwarder election, aliasing and load balancing, and fast
convergence. For each issue, the chapter explains the EVPN mechanisms used to
address it and examines their implementation in SONiC.
Multi-Chassis Link Aggregation
Figure 9-1 represents the first issue. ASW-104 is dual-homed to VTEP
switches Leaf-102 and Leaf-103. From ASW-104's perspective, the multichassis
LAG configuration is straightforward. First, a logical PortChannel is created,
and then interfaces Ethernet0 and Ethernet1 are added to the PortChannel. This
bundles the two physical interfaces into a single logical interface.
After the LACP configuration is enabled, ASW-104 exchanges LACP
frames with Leaf-102 and Leaf-103 using the link-local multicast MAC address
01:80:c2:00:00:02. The LACP frames advertise the system identity and port
information used by the switches to negotiate the LAG.
Figure 9-1 shows the complete PortChannel configuration on all three
switches and the common System MAC configuration on the leaf switches. It also
shows the configuration used to create the EVPN Ethernet Segment and associate
the PortChannel with it. However, at this point, we focus on the PortChannel
and LACP operation. The Ethernet Segment configuration is examined next.
Figure 9-1: LACP
and EVPN ES Configuration
Port-Channel verification
Examples 9-1, 9-2, and 9-3 verify that the
configured interfaces are members of PortChannel234 and that the PortChannel is
operational (U).
Leaf-102# show
PortChannel summary
Flags(oper-status): D - Down U - Up (portchannel) P - Up in
portchannel (members)
I - LACP
individual
-----------------------------------------------------------------------------------
Group PortChannel Type Protocol Member Ports
-----------------------------------------------------------------------------------
234 PortChannel234 (U) Eth
LACP Ethernet4(P)
Leaf-102#
Example 9-1:
PortChannel Verification: Leaf-102.
Leaf-103# show
PortChannel summary
Flags(oper-status): D - Down U - Up (portchannel) P - Up in
portchannel (members)
I - LACP
individual
-----------------------------------------------------------------------------------
Group PortChannel Type Protocol Member Ports
-----------------------------------------------------------------------------------
234 PortChannel234 (U) Eth
LACP Ethernet4(P)
Leaf-103#
Example 9-2:
PortChannel Verification: Leaf-103.
ASW-104# show
PortChannel summary
Flags(oper-status): D - Down U - Up (portchannel) P - Up in
portchannel (members)
I - LACP
individual
-----------------------------------------------------------------------------------
Group PortChannel Type Protocol Member Ports
-----------------------------------------------------------------------------------
234 PortChannel234 (U) Eth
LACP Ethernet0(P)
Ethernet1(P)
Example 9-3:
PortChannel Verification: ASW-104.
The role of the configured System MAC can be seen in the LACP
messages exchanged between the leaf switches and ASW-104. In an LACP message,
the Actor is the local device transmitting the LACP information, while the
Partner is the device at the other end of the link. The partial tcpdump outputs
in Example 9-4 show the LACP information transmitted by Leaf-102 and Leaf-103
toward ASW-104. The LACP frames are sent to the IEEE-reserved link-local
multicast MAC address 01:80:c2:00:00:02, which is used by LACP and other Slow
Protocols.
On both leaf switches, the Actor Information TLV reports the same
LACP System MAC, 02:02:01:03:01:04, and the same LACP Key, 234. The Key
corresponds to PortChannel234 in this configuration. The state flags include
Synchronization, Collecting, and Distributing, indicating that the links are
synchronized and actively participating in the LAG. The Partner Information TLV
identifies ASW-104 with System MAC 0c:b3:04:8f:00:0a.
Leaf-102:
05:37:24.900360
02:02:01:03:01:04 > 01:80:c2:00:00:02,
ethertype Slow Protocols (0x8809), length
124: LACPv1, length 110
Actor Information TLV (0x01), length 20
System 02:02:01:03:01:04, System Priority 65535,
Key 234, Port 5, Port Priority 255
State Flags [Activity, Aggregation,
Synchronization,
Collecting, Distributing]
Partner Information TLV (0x02), length 20
System 0c:b3:04:8f:00:0a, System Priority
65535,
Key 234, Port 1, Port Priority 255
State Flags [Activity, Aggregation,
Synchronization,
Collecting, Distributing]
Leaf-103:
05:37:25.302036
02:02:01:03:01:04 > 01:80:c2:00:00:02,
ethertype Slow Protocols (0x8809), length
124: LACPv1, length 110
Actor Information TLV (0x01), length 20
System 02:02:01:03:01:04, System Priority 65535,
Key 234, Port 5, Port Priority 255
State Flags [Activity, Aggregation,
Synchronization,
Collecting, Distributing]
Partner Information TLV (0x02), length 20
System 0c:b3:04:8f:00:0a, System Priority
65535,
Key 234, Port 2, Port Priority 255
State Flags [Activity, Aggregation,
Synchronization,
Collecting, Distributing]
Example 9-4: Partial LACP packet captures on Leaf-102 and Leaf-103
Split Horizon
Figure 9-2: Split
Horizon.
Figure 9-2 above depicts the second issue,
Split Horizon. Host-5 sends an ARP message, which the ASW-104 hashing algorithm
selects for forwarding over the PortChannel234 member interface Ethernet0.
Leaf-103 receives the ARP message and forwards it over the ingress replication
PMSI tunnels established for BUM traffic toward the VTEP switches Leaf-102 and
Leaf-101. Leaf-101 processes the received VXLAN-encapsulated ARP message and
forwards the ARP frame to Host-1, as expected. However, Leaf-102, as the
multihoming peer of Leaf-103, must not forward the ARP message back to the
multihomed ASW-104.
EVPN Ethernet Segment (ES) architecture
provides the mechanism to solve this problem. No additional configuration is
required beyond what was shown in Figure 9-1. The command evpn ethernet-segment
00:00:00:00:00:00:00:00:12:34 is configured under the PortChannel234 interface
on both Leaf-102 and Leaf-103. This configuration creates an Ethernet Segment
to which the VLANs carried over the PortChannel are associated.
After the Ethernet Segment is configured, the
switches advertise an EVPN Ethernet Segment Route, EVPN Route-Type 4, through
BGP. The Ethernet Segment identifier is carried in the route, together with
Route Target extended communities for the associated EVIs/VNIs. The receiving
VTEPs use this information to discover the remote Ethernet Segment and its
multihoming peers. The next section explains the Ethernet Segment and Ethernet
Segment Route in detail.
Ethernet Segment Route
The EVPN Ethernet Segment Route, Route Type 4, is advertised by each
VTEP attached to an Ethernet Segment. It identifies the Ethernet Segment
through its ESI (ES identifier) and provides the information required by the
participating VTEPs to discover each other and perform Designated Forwarder
(DF) election. The DF is the VTEP responsible for forwarding BUM (Broadcast,
Unknown Unicast, and Multicast) traffic toward a multihomed Ethernet Segment.
In Single-Active multihoming, the Ethernet Segment routes also participate in
the procedures that determine the active PE for the multihomed segment.
Examples 9-5 and 9-6 show the Route Type 4 entries in the BGP EVPN
tables of Leaf-102 and Leaf-103. Figure 9-3 shows the first Ethernet Segment
Route entry from the Leaf-102 BGP table.
The Route Distinguisher (RD) is 1.1.1.102:6097. The first field,
1.1.1.102, is the Administrator field of the RD, while 6097 is the Assigned
Number field. The field [4], Route Type 4, identifies the NLRI as an Ethernet
Segment Route.
The next field contains the complete 10-octet Ethernet Segment
Identifier, 00:00:00:00:00:00:00:00:12:34. The first octet, 00, identifies this
as an ESI Type 0 ESI, which is manually configured by the operator. The
remaining nine octets, together with the ESI type, form the complete Ethernet
Segment Identifier.
The next field, [32], specifies the length of the Originator IP
Address. The final field, [1.1.102.102], is the Originating Router's IP
Address.
The ESI, IP Address Length, and Originating Router's IP Address
identify the Ethernet Segment Route in the EVPN NLRI.
Figure 9-3: Ethernet
Segment Route Example.
In addition to the mandatory regular BGP path attributes, the Route
Type 4 advertisements carry three extended communities relevant to Ethernet
Segment operation and DF election.
The Tunnel Encapsulation extended community ET:8, depicts the VXLAN tunnel
encapsulation.
The ES-Import Route Target, displayed as
ES-Import-Rt:00:00:00:00:00:00, is used to restrict import of the Ethernet
Segment route only to VTEPs attached to the same Ethernet Segment. Each VTEP
connected to the ES derives an import rule from the ESI and uses the ES-Import
Route Target to import the Type 4 routes originated by the other VTEPs attached
to that ES. This allows the VTEPs participating in the same Ethernet Segment to
discover one another and participate in the common DF-election process.
The DF Election Extended Community, displayed as DF: (alg: 2, pref:
32767), carries the parameters used for DF election. In this topology, Alg: 2
identifies the Highest-Preference Algorithm, defined by RFC 9785, and Pref:
32767 is the DF preference advertised by the VTEP. RFC 9785 defines the
preference as a 16-bit value in the range 0–65535, with 32767 as the default
preference. With the Highest-Preference Algorithm, the PE advertising the
highest preference is selected as the DF.
Both Leaf-102 and Leaf-103 advertise the same preference, 32767, as
shown in Examples 9-5 and 9-6. When the preference is equal, tie-breaking is based
on the numerically lowest VTEP IP address, and the VTEP Leaf-102 is selected as
the DF for the Ethernet Segment.
The two Route Type 4 entries shown in each BGP table also illustrate
the difference between the locally originated and remotely received Ethernet
Segment routes. On Leaf-102, the route with RD 1.1.1.102:6097 is locally
originated, while the route with RD 1.1.1.103:6097 was received from Leaf-103
through the EVPN BGP session. Leaf-103 has the corresponding opposite view: its
own Route Type 4 is locally originated (SubType: 1), while the Leaf-102 route
is received remotely (SubType: 0).
Leaf-102# show
bgp l2vpn evpn route detail type es
Route
Distinguisher: 1.1.1.102:6097
BGP routing table
entry for 1.1.1.102:6097:[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.102.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route
[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.102.102]
Local
1.1.102.102 from 0.0.0.0 (1.1.1.102)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8
ES-Import-Rt:00:00:00:00:00:00 DF: (alg: 2, pref: 32767)
SubType: 1 Last update: Tue Sep 29
05:46:39 2026
Route
Distinguisher: 1.1.1.103:6097
BGP routing table
entry for 1.1.1.103:6097:[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.103.103]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route
[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.103.103]
65011 65103
1.1.103.103 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: ET:8
ES-Import-Rt:00:00:00:00:00:00 DF: (alg: 2, pref: 32767)
SubType: 0 Last update: Tue Sep 29
05:58:02 2026
Displayed 2
prefixes (2 paths) (of requested type)
Leaf-102#
Example 9-5: EVPN Route Type 4 BGP Table Entries - Leaf-102.
Leaf-103# show
bgp l2vpn evpn route detail type es
Route
Distinguisher: 1.1.1.102:6097
BGP routing table
entry for 1.1.1.102:6097:[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.102.102]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route
[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.102.102]
65011 65102
1.1.102.102 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: ET:8
ES-Import-Rt:00:00:00:00:00:00 DF: (alg: 2, pref: 32767)
SubType: 0 Last update: Tue Sep 29
05:58:04 2026
Route
Distinguisher: 1.1.1.103:6097
BGP routing table
entry for 1.1.1.103:6097:[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.103.103]
Paths: (1
available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route
[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.103.103]
Local
1.1.103.103 from 0.0.0.0 (1.1.1.103)
Origin IGP, weight 32768, valid, sourced,
local, best (First path received)
Extended Community: ET:8
ES-Import-Rt:00:00:00:00:00:00 DF: (alg: 2, pref: 32767)
SubType: 1 Last update: Tue Sep 29
05:47:10 2026
Displayed 2
prefixes (2 paths) (of requested type)
Leaf-103#
Example 9-6: EVPN Route Type 4 BGP Table Entries - Leaf-103.
Ethernet Segment Verification
Examples 9-7 and 9-8 provide detailed
Ethernet Segment information from Leaf-102 and Leaf-103. The outputs of the
command show evpn es detail provide an
operational view of the Ethernet Segment, including its state, DF status, DF
preference, VNI information, and the remote VTEP participating in the same
Ethernet Segment.
On both switches, the same ESI,
00:00:00:00:00:00:00:00:12:34, is associated with PortChannel234. The Ethernet
Segment is operationally up, is associated with a bridge port, and is ready for
BGP. Each switch reports one associated VNI, corresponding to VNI 10010 used by
VLAN 10 in this topology. The MAC Count shown in the output reflects the
current MAC state associated with the Ethernet Segment and does not affect the
DF election.
The difference between the two outputs is the
DF status. As shown in Example 9-7, Leaf-102 reports DF status: df, while
Example 9-8 shows DF status: non-df on Leaf-103. Leaf-102 is therefore the DF
for this Ethernet Segment, while Leaf-103 is the non-DF PE.
The outputs also confirm that both PEs use
the same DF preference, 32767, and identify each other as the remote VTEP
associated with the Ethernet Segment. This is consistent with the Route Type 4
advertisements examined in the previous section. Because the configured DF
preferences are equal, the DF election proceeds to the tie-breaking rules
described earlier, resulting in Leaf-102 being selected as the DF.
The show bgp l2vpn evpn es detail output
provides additional BGP-level information about the Ethernet Segment. Each PE
has its own ES fragment, identified by its local Route Distinguisher and
Originator-IP address. Leaf-102 therefore shows RD 1.1.1.102:6097 and
Originator-IP 1.1.102.102, while Leaf-103 shows RD 1.1.1.103:6098 and
Originator-IP 1.1.103.103.
The same output also shows one local VNI and
one remote VNI on each PE, as well as one associated VRF. This confirms that
the Ethernet Segment information and its associated VNI are consistent between
the two multihoming VTEPs. The remote VTEP entries additionally show the
preference-based DF-election algorithm and the common preference value of
32767.
Finally, the show bgp l2vpn
evpn es-evi detail output associates the ESI with VNI
10010 on both switches. Each PE shows its own ES fragment and the other PE as
the remote VTEP, confirming that both Leaf-102 and Leaf-103 have consistent
information about the same Ethernet Segment and its VNI.
Together, Examples 9-7 and 9-8 provide three complementary views of
the Ethernet Segment: show evpn es detail shows the operational state and DF result, show bgp l2vpn
evpn es detail
shows the BGP-level ES information and DF-election
parameters, and show bgp l2vpn evpn es-evi detail shows
the ESI-to-VNI association.
The outputs are consistent with the Route Type 4 advertisements
examined in the previous section: both PEs advertise and discover the same ESI,
use the same DF-election algorithm and preference, and independently arrive at
the same DF-election result, with Leaf-102 as DF and Leaf-103 as non-DF.
Leaf-102# show
evpn es detail
ESI:
00:00:00:00:00:00:00:00:12:34
Type: Local,Remote
Interface: PortChannel234
State: up
Bridge port: yes
Ready for BGP: yes
VNI Count: 1
MAC Count: 0
DF status: df
DF preference: 32767
Nexthop group: 536870913
VTEPs:
1.1.103.103 df_alg: preference df_pref:
32767 nh: 268435458
Leaf-102# show
bgp l2vpn evpn es detail
ESI:
00:00:00:00:00:00:00:00:12:34
Type: LR
RD: 1.1.1.102:6097
Originator-IP: 1.1.102.102
Local ES DF
preference: 32767
VNI Count: 1
Remote VNI Count: 1
VRF Count: 1
MACIP EVI Path Count: 0
MACIP Global Path Count: 0
Inconsistent VNI VTEP Count: 0
Inconsistencies: -
Fragments:
1.1.1.102:6097 EVIs: 1
VTEPs:
1.1.103.103 flags: EA df_alg: preference df_pref: 32767
Leaf-102#
Leaf-102# show
bgp l2vpn evpn es-evi detail
VNI: 10010 ESI:
00:00:00:00:00:00:00:00:12:34
Type: LR
ES fragment RD: 1.1.1.102:6097
Inconsistencies: -
VTEPs: 1.1.103.103(EV)
Leaf-102#
Example 9-7: EVPN Route Type 4 ES Verification - Leaf-102.
Leaf-103# show
evpn es detail
ESI:
00:00:00:00:00:00:00:00:12:34
Type: Local,Remote
Interface: PortChannel234
State: up
Bridge port: yes
Ready for BGP: yes
VNI Count: 1
MAC Count: 0
DF status:
non-df
DF preference: 32767
Nexthop group: 536870913
VTEPs:
1.1.102.102 df_alg: preference df_pref:
32767 nh: 268435458
Leaf-103#
Leaf-103# show
bgp l2vpn evpn es detail
ESI:
00:00:00:00:00:00:00:00:12:34
Type: LR
RD: 1.1.1.103:6098
Originator-IP: 1.1.103.103
Local ES DF
preference: 32767
VNI Count: 1
Remote VNI Count: 1
VRF Count: 1
MACIP EVI Path Count: 0
MACIP Global Path Count: 0
Inconsistent VNI VTEP Count: 0
Inconsistencies: -
Fragments:
1.1.1.103:6098 EVIs: 1
VTEPs:
1.1.102.102 flags: EA df_alg: preference df_pref: 32767
Leaf-103#
Leaf-103# show
bgp l2vpn evpn es-evi detail
VNI: 10010 ESI:
00:00:00:00:00:00:00:00:12:34
Type: LR
ES fragment RD: 1.1.1.103:6098
Inconsistencies: -
VTEPs: 1.1.102.102(EV)
Example 9-8: EVPN Route Type 4 ES Verification - Leaf-103.
Now back to our original problem: How does Leaf-102 prevent an ARP
message received over the ingress replication tunnel from Leaf-103, originally
sent by Host-5, from being forwarded back to the dual-homed ASW-104?
The VXLAN encapsulation does not carry the ESI. However, the outer
IP header contains Leaf-103's VTEP IP address as the source IP address, and
Leaf-103 is identified as an ES peer of Leaf-102 (see Example 9-7). This
provides the first piece of information needed for split-horizon processing.
The VNI 10010 carried in the VXLAN header is also associated with the Ethernet
Segment 00:00:00:00:00:00:00:00:12:34 (see Example 9-7).
Using the remote VTEP information and the VNI-to-ES association,
Leaf-102 determines that the received BUM frame originated from the same
Ethernet Segment toward which it would otherwise forward the frame. Leaf-102
therefore drops the ARP message instead of forwarding it back toward ASW-104,
even though Leaf-102 is the DF for the Ethernet Segment.
DF-Based BUM Traffic Filtering
The Designated Forwarder (DF) for the
Ethernet Segment also resolves the third issue, which is related to BUM traffic
but occurs when the traffic is received over an ingress replication PMSI tunnel
from a remote VTEP, Leaf-101.
In Figure 9-4, Host-1 sends an ARP Request, which Leaf-101 forwards
to both Leaf-102 and Leaf-103. In this case, the forwarding decision toward the
multihomed Ethernet Segment is based on the DF role. Because Leaf-102 is the
DF, it forwards the ARP Request toward ASW-104. Leaf-103, as the non-DF VTEP,
drops the received ARP Request.
Figure 9-4: DF-Based
BUM Filtering.
Load-Balancing and Aliasing
Figure 9-5 depicts the fourth issue related
to EVPN ESI Multihoming. In this topology, VLANs 10 (VNI 10010) and 20 (VNI
10020) are configured on all VTEPs. Host-5 and Host-6 are connected to ASW-104,
which is dual-homed to Leaf-102 and Leaf-103. Because both VLANs are allowed on
the logical PortChannel234 interface, they are also associated with Ethernet
Segment 00:00:00:00:00:00:00:00:12:34.
The LACP hashing algorithm on ASW-104 selects
Ethernet0 for both flows originated by Host-5 and Host-6. As a result, both MAC
addresses are learned initially only by Leaf-103. Leaf-103 advertises the
corresponding MAC/IP Advertisement routes to remote VTEPs. Leaf-102 does not
independently learn these MAC addresses from the hosts because its PortChannel
member link is not selected by the ASW-104 hashing algorithm.
This creates a load-balancing problem for
remote VTEPs. For example, Leaf-101 learns the MAC reachability information for
Host-5 and Host-6 through Leaf-103, but the Ethernet Segment is also reachable
through Leaf-102. Without additional EVPN signaling, Leaf-101 has no
information indicating that Leaf-102 can also forward traffic toward the same
multihomed Ethernet Segment for these MAC addresses.
EVPN solves this problem through aliasing.
Ethernet A-D per EVI routes advertise the reachability of the Ethernet Segment
for a specific EVI, while the ESI carried in the MAC/IP Advertisement Route
associates the MAC address with that Ethernet Segment. A remote VTEP can
therefore use the MAC route together with the Ethernet A-D route to identify
multiple VTEPs as valid forwarding paths toward the same multihomed Ethernet
Segment, even when only one of the VTEPs has locally learned the MAC address.
Figure 9-5: Load
Balancing Problem.
Ethernet A-D per EVI (Route type 1)
Example 9-9 shows that Leaf-101 has imported Ethernet A-D per-EVI
routes received from both Leaf-102 and Leaf-103. The routes carry Route Target
extended communities 65102:10010 and 65103:10010, respectively. Leaf-101's EVPN
import policy matches these Route Targets and therefore imports both routes for
VNI 10010.
Both routes contain the same ESI, 00:00:00:00:00:00:00:00:12:34, in
the Ethernet A-D per-EVI NLRI. This tells Leaf-101 that both Leaf-102 and
Leaf-103 have reachability to VNI 10010 through the same Ethernet Segment.
Leaf-101#
show bgp l2vpn evpn route detail type ead
Route
Distinguisher: 1.1.1.102:10
BGP routing table entry for
1.1.1.102:10:[1]:[0]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
Paths:
(1 available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [1]:[0]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
VNI 10010
65011 65102
1.1.102.102 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65102:10010 ET:8
SubType: 0 Last update: Tue Sep 29
05:58:03 2026
Route
Distinguisher: 1.1.1.103:10
BGP routing table entry for
1.1.1.103:10:[1]:[0]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
Paths:
(1 available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [1]:[0]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
VNI 10010
65011 65103
1.1.103.103 from Ethernet0 (1.1.1.11)
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65103:10010 ET:8
SubType: 0 Last update: Tue Sep 29
05:58:03 2026
Example 9-9: EVPN Route Type 1: Ethernet A-D per-EVI on Leaf-101.
The ESI is also carried in the ESI-aware MAC Advertisement Route, as
shown in Example 9-10. The first entry advertises the MAC address
00:50:79:66:68:01, while the second entry also advertises its IP address
10.0.10.105. Both entries contain the same ESI, 00:00:00:00:00:00:00:00:12:34.
The ESI therefore provides the common identifier that allows
Leaf-101 to associate the MAC advertisement with the Ethernet Segment. By
combining the ESI carried in the Type-2 MAC Advertisement Route with the
Ethernet A-D per-EVI routes for the same ESI, Leaf-101 can determine that
Leaf-102 and Leaf-103 are both valid VTEP paths toward the multihomed Ethernet
Segment, even though Leaf-103 is the VTEP that originally learned the MAC
address from ASW-104.
This is the basis of EVPN aliasing. The remote VTEP does not require
each ES peer to independently learn and advertise the MAC address. Instead, the
Ethernet A-D per-EVI routes provide the additional ES-peer reachability needed
to use both VTEPs as forwarding paths toward the Ethernet Segment..
Leaf-101#
show bgp l2vpn evpn route detail type macip
Route
Distinguisher: 1.1.1.103:10
BGP routing table entry for
1.1.1.103:10:[2]:[0]:[48]:[00:50:79:66:68:01]
Paths:
(1 available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[00:50:79:66:68:01] VNI
10010
65011 65103
1.1.103.103 from Ethernet0 (1.1.1.11)
ESI 00:00:00:00:00:00:00:00:12:34
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65103:10010 ET:8
SubType: 0 Last update: Tue Sep 29
06:12:54 2026
BGP
routing table entry for
1.1.1.103:10:[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.105]
Paths:
(1 available, best #1)
Advertised to non peer-group peers:
Ethernet0
Route [2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.105] VNI
10010/28158
65011 65103
1.1.103.103 from Ethernet0 (1.1.1.11)
ESI 00:00:00:00:00:00:00:00:12:34
Origin IGP, valid, external, best (First
path received)
Extended Community: RT:65103:10010
RT:65103:28158 ET:8 Rmac:0c:38:66:11:00:0a
SubType: 0 Last update: Tue Sep 29
06:12:54 2026
Example 9-10: EVPN Route Type 2: ESI-Aware MAC Advertisement on
Leaf-101.
Fast Convergence: Ethernet A-D per ES
The last EVPN multihoming issue introduced in this chapter is an ES
member-link failure. Figure 9-6 shows the failure of the Ethernet0 member link
on Leaf-102 and the resulting withdrawal of the Ethernet A-D per-ES route.
Before the failure, Leaf-102 has advertised an Ethernet A-D per-ES
(EVPN Route Type 1) route to Leaf-101 (1). Unlike the Ethernet A-D per-EVI
route discussed in the aliasing section, the per-ES route is not associated
with an individual EVI. The route identifies the Ethernet Segment and carries
the Route Targets for the EVPN instances associated with that Ethernet Segment.
In the SONiC/FRR output, the associated VNI is therefore shown as 0.
The route also carries the ESI Label Extended Community. In the FRR
output, this is displayed as ESI-label-Rt:AA. The AA indicates All-Active
redundancy. In RFC 7432 terminology, this corresponds to the Single-Active bit
in the ESI Label Extended Community being cleared (0). The ESI Label itself is
used for split-horizon filtering, while the Ethernet A-D per-ES route also
provides the Ethernet Segment reachability information used for fast
convergence.
The Route Target 65102:10010 identifies the EVPN instance associated
with VNI 10010. If the Ethernet Segment carries multiple EVIs, the set of
Ethernet A-D per-ES routes carries the Route Targets for those EVIs, allowing
the routes to be imported by remote PEs participating in those EVPN instances.
When the Ethernet0 ES member link fails (2), Leaf-102 withdraws its
Ethernet A-D per-ES route by sending a BGP UPDATE containing an
MP_UNREACH_NLRI. Figure 9-6 shows selected fields from the withdrawal message,
decoded from the raw tcpdump hexadecimal representation in Example 9-11.
The withdrawal identifies the Ethernet Segment through its ESI,
00:00:00:00:00:00:00:00:12:34, and identifies Leaf-102 as the originating VTEP.
Upon receiving the withdrawal, Leaf-101 removes Leaf-102 as a forwarding path
toward the Ethernet Segment. This also removes the corresponding aliasing path
for MAC addresses associated with the Ethernet Segment, leaving Leaf-103 as the
remaining active path. Examples 9-12 and 9-13 show the BGP table on Leaf-101
before and after the withdrawal.
Figure 9-6: ES
member Link Failure and Mass-Withdrawn.
admin@Leaf-101:~$ sudo tcpdump
-i Ethernet0 -nn -e -vvv -t 'tcp port 179'
tcpdump: listening on Ethernet0,
link-type EN10MB (Ethernet), snapshot length 262144 bytes
0c:22:34:b6:00:0a >
0c:cb:81:19:00:0a, ethertype IPv6 (0x86dd), length 283: (class 0xc0, flowlabel
0x7f57a, hlim 1, next-header TCP (6) payload length: 229)
fe80::e22:34ff:feb6:a.179 > fe80::ecb:81ff:fe19:a.44198: Flags [P.], cksum
0x071d (correct), seq 2973481895:2973482092, ack 523212514, win 121, options
[nop,nop,TS val 2158920453 ecr 1641476305], length 197: BGP
Update Message (2), length: 82
Multi-Protocol Unreach NLRI (15), length: 55, Flags [OE]:
AFI: VPLS (25), SAFI: EVPN (70)
no AFI 25 / SAFI 70 decoder
0x0000: 0019 4604 1700 0101 0101 6617 d100 0000
0x0010: 0000 0000 0012 3420 0101 6666 0119 0001
0x0020: 0101 0166 17d1 0000 0000 0000 0000 1234
0x0030: ffff ffff 0000 00
Update Message (2), length: 115
Multi-Protocol Reach NLRI (14),
length: 51, Flags [OE]:
AFI: VPLS (25), SAFI: EVPN (70)
no AFI 25 / SAFI 70 decoder
0x0000: 0019 4604 0101 6666 0002 2800 0101 0101
0x0010: 6600 0a00 0000 0000 0000 0012 3400 0000
0x0020: 0030 0050 7966 6801 200a 000a 6900 271a
0x0030: 006d fe
Origin (1), length: 1, Flags [T]: IGP
0x0000: 00
AS Path (2), length: 10, Flags [TE]:
65011 65102
0x0000: 0202 0000 fdf3 0000 fe4e
Extended Community (16), length: 16,
Flags [OT]:
target (0x0002), Flags [none]:
65102:10010 (= 0.0.39.26)
encapsulation (0x030c), Flags
[none]: Tunnel type: VXLAN
0x0000: 0002 fe4e 0000 271a 030c 0000 0000 0008
Example 9-11: Tcpdump - Leaf-101.
Leaf-101# show bgp l2vpn evpn
route
BGP table version is 9, local
router ID is 1.1.1.101
Status codes: s suppressed, d
damped, h history, * valid, > best, i - internal, q queued, r RIB-failure
Origin codes: i - IGP, e - EGP, ?
- incomplete
EVPN type-1 prefix:
[1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix:
[2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix:
[3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix:
[4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix:
[5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 1.1.1.101:10
*> [2]:[0]:[48]:[00:50:79:66:68:03]
1.1.101.101 32768 i
ET:8 RT:65101:10010
*>
[2]:[0]:[48]:[00:50:79:66:68:03]:[32]:[10.0.10.101]
1.1.101.101 32768 i
ET:8 RT:65101:10010
RT:65101:28158 Rmac:0c:cb:81:19:00:0a
*> [3]:[0]:[32]:[1.1.101.101]
1.1.101.101 32768 i
ET:8 RT:65101:10010
Route Distinguisher: 1.1.1.101:20
*> [3]:[0]:[32]:[1.1.101.101]
1.1.101.101 32768 i
ET:8 RT:65101:10020
Route Distinguisher: 1.1.1.102:10
*>
[1]:[0]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
1.1.102.102 0 65011
65102 i
RT:65102:10010 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:01]
1.1.102.102 0 65011
65102 i
ESI:00:00:00:00:00:00:00:00:12:34
RT:65102:10010 ET:8
ND:Proxy
*>
[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.105]
1.1.102.102 0 65011
65102 i
ESI:00:00:00:00:00:00:00:00:12:34
RT:65102:10010
RT:65102:28158 ET:8 Rmac:0c:e8:78:22:00:0a
*> [3]:[0]:[32]:[1.1.102.102]
1.1.102.102 0 65011
65102 i
RT:65102:10010 ET:8
Route Distinguisher: 1.1.1.102:20
*> [3]:[0]:[32]:[1.1.102.102]
1.1.102.102 0 65011
65102 i
RT:65102:10020 ET:8
Route Distinguisher: 1.1.1.102:6097
*>
[1]:[4294967295]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
1.1.102.102 0 65011
65102 i
RT:65102:10010 ET:8
ESI-label-Rt:AA
*>
[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.102.102]
1.1.102.102 0 65011
65102 i
ET:8
ES-Import-Rt:00:00:00:00:00:00 DF: (alg: 2, pref: 32767)
Route Distinguisher: 1.1.1.103:10
*>
[1]:[0]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
1.1.103.103 0 65011
65103 i
RT:65103:10010 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:01]
1.1.103.103 0 65011
65103 i
ESI:00:00:00:00:00:00:00:00:12:34
RT:65103:10010 ET:8
*>
[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.105]
1.1.103.103 0 65011
65103 i
ESI:00:00:00:00:00:00:00:00:12:34
RT:65103:10010
RT:65103:28158 ET:8 Rmac:0c:38:66:11:00:0a
*> [3]:[0]:[32]:[1.1.103.103]
1.1.103.103 0 65011
65103 i
RT:65103:10010 ET:8
Route Distinguisher:
1.1.1.103:6097
*>
[1]:[4294967295]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
1.1.103.103 0 65011
65103 i
RT:65103:10010 ET:8
ESI-label-Rt:AA
*>
[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.103.103]
1.1.103.103 0 65011
65103 i
ET:8
ES-Import-Rt:00:00:00:00:00:00 DF: (alg: 2, pref: 32767)
Displayed 17 prefixes (17 paths)
Example 9-12: BGP Table of Leaf-101
Before Link Failure.
Leaf-101# show
bgp l2vpn evpn route
BGP table version
is 17, local router ID is 1.1.1.101
Status codes: s
suppressed, d damped, h history, * valid, > best, i - internal, q queued, r
RIB-failure
Origin codes: i -
IGP, e - EGP, ? - incomplete
EVPN type-1
prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2
prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3
prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4
prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5
prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Extended Community
Route
Distinguisher: 1.1.1.101:10
*> [2]:[0]:[48]:[00:50:79:66:68:03]
1.1.101.101 32768 i
ET:8 RT:65101:10010
*>
[2]:[0]:[48]:[00:50:79:66:68:03]:[32]:[10.0.10.101]
1.1.101.101 32768 i
ET:8 RT:65101:10010
RT:65101:28158 Rmac:0c:cb:81:19:00:0a
*> [3]:[0]:[32]:[1.1.101.101]
1.1.101.101 32768 i
ET:8 RT:65101:10010
Route
Distinguisher: 1.1.1.101:20
*> [3]:[0]:[32]:[1.1.101.101]
1.1.101.101 32768 i
ET:8 RT:65101:10020
Route
Distinguisher: 1.1.1.103:10
*>
[1]:[0]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
1.1.103.103 0 65011
65103 i
RT:65103:10010 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:01]
1.1.103.103 0 65011
65103 i
ESI:00:00:00:00:00:00:00:00:12:34
RT:65103:10010 ET:8
*>
[2]:[0]:[48]:[00:50:79:66:68:01]:[32]:[10.0.10.105]
1.1.103.103 0 65011
65103 i
ESI:00:00:00:00:00:00:00:00:12:34
RT:65103:10010
RT:65103:28158 ET:8 Rmac:0c:38:66:11:00:0a
*> [3]:[0]:[32]:[1.1.103.103]
1.1.103.103 0 65011
65103 i
RT:65103:10010 ET:8
Route
Distinguisher: 1.1.1.103:6097
*>
[1]:[4294967295]:[00:00:00:00:00:00:00:00:12:34]:[128]:[::]:[0]
1.1.103.103 0 65011
65103 i
RT:65103:10010 ET:8
ESI-label-Rt:AA
*>
[4]:[00:00:00:00:00:00:00:00:12:34]:[32]:[1.1.103.103]
1.1.103.103 0 65011
65103 i
ET:8
ES-Import-Rt:00:00:00:00:00:00 DF: (alg: 2, pref: 32767)
Displayed 10
prefixes (10 paths)
Leaf-101#
Example 9-13: BGP Table of Leaf-101
After Link Failure.
Database Updates
CONFIG_DB and APPL_DB
Figure 9-7 depicts how the sonic-cli ESI
Multihoming configuration is reflected in CONFIG_DB. We first examine the
EVPN_ETHERNET_SEGMENT|PortChannel234 entry and its corresponding APPL_DB
entries. The CONFIG_DB entry defines the ESI, its type, and the attached
interface PortChannel234. The related PortChannel, member-port, and VLAN
configuration entries are examined in the following section.
The vxlanmgr daemon inside the swss container
translates the ESI configuration from CONFIG_DB into the APPL_DB entries
required for EVPN Multihoming. The L2NHG_TABLE:536870913 entry identifies the
ESI and its associated child next-hop group.
Additionally, the ESI configuration results
in three operational entries for PortChannel234: EVPN_ES_TABLE,
EVPN_ES_BACKUP_NHG_TABLE, and EVPN_SPLIT_HORIZON_TABLE. EVPN_ES_TABLE tracks
the operational state of the local Ethernet Segment, while EVPN_ES_BACKUP_NHG_TABLE
associates the Ethernet Segment with the next-hop group used for Ethernet
Segment fast recovery.
The EVPN_SPLIT_HORIZON_TABLE entry represents
the ESI split-horizon filtering policy. In this example, it identifies remote
ES peer VTEP 1.1.103.103. When a frame is received over a VXLAN tunnel from
this VTEP, the split-horizon policy prevents the frame from being forwarded out
local PortChannel234, avoiding a loop through the multihomed Ethernet Segment.
The remote ES peer information is learned through BGP EVPN Ethernet Segment
Routes (Route Type 4) and reflected in APPL_DB by the EVPN control-plane
components.
The EVPN_ES_TABLE and
EVPN_ES_BACKUP_NHG_TABLE entries are examined in more detail in the APPL_DB
section.
Figure 9-7: CONFIG_DB and APPL_DB: EVPN
ES - Leaf-102.
Figure 9-8 shows how the PORTCHANNEL|PortChannel234 and
PORTCHANNEL_MEMBER|PortChannel234|Ethernet4
entries in CONFIG_DB define the PortChannel and its member interface. In this
context, CONFIG_DB represents the intended configuration, the state we want
SONiC to establish. The PORTCHANNEL
entry sets the administrative state, LACP system MAC, and VLAN 10 membership,
while the PORTCHANNEL_MEMBER entry
associates Ethernet4 with PortChannel234.
The SONiC teamdmanager daemon in the
swss/teamd subsystem processes the PortChannel configuration from CONFIG_DB and
configures and manages the corresponding Linux teamd instance. It provides
teamd with the LAG and LACP configuration derived from the PortChannel and its
member interface (1a). teamd manages the Linux team interface and provides the
LACP control-plane functionality, including LACP negotiation with the connected
device (2).
The LACP process also depends on the runtime
state of the member interface. teamd monitors the member interfaces through the
Linux networking stack. For Ethernet4, a carrier-state change (3) is reported
by the Linux kernel through a netlink notification (4). teamd uses the
member-interface state together with the LACP state to maintain the operational
state of the LAG. In parallel, portsyncd receives the kernel notification for
SONiC port-state synchronization.
The resulting LAG and member operational
state is monitored by teamsyncd, which communicates with the teamd instances
and reflects LAG state changes into APPL_DB’s (5) entries LAG_TABLE:PortChannel234 and LAG_MEMBER_TABLE:PortChannel234:Ethernet4. Figure 9-8 illustrates this operational-state update path.
In contrast to CONFIG_DB, APPL_DB represents
the resulting operational state, the state that SONiC has actually established.
In our example, LAG_TABLE:PortChannel234
reports the PortChannel as operationally up (oper_status: up) with a speed of
25 Gbit/s and an MTU of 9100. The reason attribute indicates OPER_UP,
while active: true indicates that the LAG is active. The LAG_MEMBER_TABLE:PortChannel234:Ethernet4 entry reports the member interface as enabled, and LAG_APP_STATUS_TABLE:
PortChannel234 reports the interface-tracking status as
up.
The traffic_disable_evpnmh attribute
in LAG_TABLE represents the interaction between LAG operation and EVPN
Multihoming. Its value false
indicates that traffic on the PortChannel is not currently disabled by the EVPN
Multihoming control logic. This state can be coordinated with EVPN Multihoming
events, such as Designated Forwarder election changes or Ethernet Segment fast
recovery, to temporarily suppress or restore traffic when required.
Note: system_mac:02:02:01:03:01:04 in the PORTCHANNEL|PortChannel234 CONFIG_DB entry is not replicated as an attribute in the corresponding
APPL_DB LAG entries. Instead, the PortChannel configuration is used by
teamdmanager to configure the Linux teamd instance, where the system MAC is
used as the LACP system identifier for the PortChannel. Both Leaf-102 and
Leaf-103 use the same system MAC toward ASW-104, allowing the two VTEPs
participating in the Ethernet Segment to present a common LACP system identity
to the multihomed device.
Figure 9-8: CONFIG_DB and APPL_DB:
PORTCHANNEL - Leaf-102.
The VLAN configuration is also translated from CONFIG_DB into
APPL_DB. The VLAN|Vlan10 entry in CONFIG_DB defines VLAN 10 and lists Ethernet2
and PortChannel234 as its members. The PORTCHANNEL|PortChannel234 entry
independently specifies VLAN 10 in its tagged_vlans@ attribute, establishing
the PortChannel's VLAN membership from the PortChannel side.
The SONiC vlanmgrd daemon processes the VLAN
and VLAN-member configuration from CONFIG_DB and creates the corresponding
application state in APPL_DB. For PortChannel234, this results in the
VLAN_MEMBER_TABLE:Vlan10:PortChannel234 entry. The dynamic: no attribute
indicates that the membership is statically configured rather than dynamically
learned, while tagging_mode: tagged indicates that frames transmitted through
the PortChannel retain the VLAN 10 tag.
Thus, the CONFIG_DB entries define the
intended VLAN membership, while the APPL_DB entry represents the VLAN-member
state that vlanmgrd has established for the PortChannel. The resulting
VLAN_MEMBER_TABLE entry is subsequently consumed by the SWSS orchestration
layer to program the corresponding VLAN membership in the switch ASIC.
Figure 9-9: CONFIG_DB and APPL_DB:
VLAN - Leaf-102.
The EVPN_ES_BACKUP_NHG_TABLE:PortChannel234 entry
associates the local Ethernet Segment with the L2 next-hop group used for
Ethernet Segment fast recovery. Its nexthop_group value, 536870913, identifies
the next-hop group associated with the ESI of PortChannel234.
The corresponding L2NHG_TABLE:536870913 entry represents the Ethernet Segment-oriented next-hop-group
hierarchy. Its esi attribute identifies the local ESI, while nexthop_group:
268435458 identifies the next-hop group used to reach a remote Ethernet Segment
peer. The L2NHG_TABLE:268435458 entry then
resolves this next hop to VTEP 1.1.103.103, which is Leaf-103 in this topology.
These L2NHG_TABLE entries represent EVPN-MH control-plane state derived from the
remote Ethernet Segment information processed by FRR. The remote Ethernet
Segment is advertised through BGP EVPN, bgpd and Zebra process the
advertisement and derive the corresponding EVPN-MH L2 next-hop-group hierarchy.
In this example, the resulting hierarchy associates ESI
00:00:00:00:00:00:00:00:12:34 with NHG 536870913, which contains the
VTEP-specific next hop 268435458 toward 1.1.103.103.
A related representation of this next-hop
hierarchy appears in L2_NEXTHOP_GROUP_TABLE. The L2_NEXTHOP_GROUP_TABLE:268435458 entry identifies 1.1.103.103 as the remote VTEP. This table is
populated through a different path: Zebra programs the corresponding Linux
nexthop information in the kernel, and the kernel notifies fpmsyncd of nexthop
changes through rtnetlink RTM_NEWNEXTHOP or RTM_DELNEXTHOP messages. fpmsyncd
then reflects the kernel-derived L2 next-hop information into APPL_DB.
The two tables therefore represent related
but different aspects of the same forwarding hierarchy. L2NHG_TABLE provides
the EVPN-MH Ethernet Segment-oriented association between the ESI, the next-hop
group, and the remote VTEP, while L2_NEXTHOP_GROUP_TABLE represents the L2
next-hop information derived from the Linux kernel. L2NhgOrch consumes the
latter to program the corresponding forwarding objects in the ASIC.
In our example topology, the hierarchy
contains a single remote VTEP because Leaf-102 has one remote Ethernet Segment
peer, Leaf-103. The ES-level NHG can, however, contain multiple remote VTEP
next hops when multiple remote Ethernet Segment peers are available. This provides
the forwarding information required for Ethernet Segment aliasing and fast
recovery when a local Ethernet Segment member becomes unavailable.
The EVPN_ES_BACKUP_NHG_TABLE:PortChannel234 entry provides the association between the
local Ethernet Segment and the ES backup next-hop group, completing the link
between the EVPN-MH control-plane state and the forwarding hierarchy used for
fast recovery.
Figure 9-10: APPL_DB: L2NHG and
L2_NEXTHOP_GROUP_TABLE - Leaf-102.
Although SONiC maintains an L2 next-hop group
that includes the remote ES peer, this does not imply that traffic normally
traverses the peer VTEP. EVPN control-plane convergence provides the primary
recovery mechanism: when an ES member fails, the associated Ethernet A-D route is
withdrawn, allowing remote VTEPs to redirect traffic to the surviving ES
member. The L2NHG may provide an additional forwarding option during a
transient failure condition, but its use depends on the SONiC/SAI forwarding
implementation.
No comments:
Post a Comment