VXLAN Part XII: Routing Exchange: intra/inter-L2VNI, EVPN-to-IP, EVPN-to-VPNv4

Edited: 25.9.2018 | Toni Pasanen

We are using BGP EVPN (MP-BGP AFI25/SAFI70-EVPN) to exchange MAC-IP (Type-2) and Prefix (Type-5) reachability information inside the VXLAN fabric between the VTEPs. Each BGP UPDATE message sent by VTEP includes L2VNI/L3VNI specific Route-Target (RT) Extended Community Path-Attribute. Based on these RTs, routes are imported to correct L2VNI/L3VNIs. Each L2VNI has VNI-specific RT, which is used for intra-VNI communication. Inside the Tenant, there is a common, Tenant specific RT used for inter-L2VNI communication.

The routing information between the external networks cannot rely only on Route-Targets. We could have an external connection over IPv4 networks by using eBGP or connection over the MPLS network by using MP-BGP (AFI1-IPv4/SAFI128-VPNv4). All of these three BGPs (BGP, BGP EVPN, and BGP VPNv4) use dissimilar address representation format in BGP updates. Let’s use the IPv4 address 192.168.100.1/24 as an example.

IPv4:    192.168.100.1/24

VPNv4: [RD]:192.168.100.1/11/112

EVPN:   [RD]:[Route-Type]:[ESI]:[MAC length]:[MAC]:[IP length]:192.168.100.1/272

Because of the different representation mode for the same address, we need to change the address format while exchanging the routing updates between BGP domains over the VXLAN Border-PE.

I am going to use the topology shown in figure 12-1 to do the deep dive to this subject.

Figure 12-1: Example Topology and IP addressing

Intra-L2VNI routing

Intra-L2VNI routing use L2VNI specific Route-Target value defined under EVPN instance in L2VNI specific configuration (example 12-1). We are using auto-generated RT values, which consists of the local BGP AS number and L2VNI- or L3VNI-Id. In our example networks, there are hosts Cafe and Beef attached to VLAN 10, which in turns is attached to VNI 10000. The BGP AS number used in VXLAN fabric is AS65000. This gives the RT 65000:10000 to L2VNI 10000. In figure 12-2, Leaf-102 sends MAC-only and MAC-IP BGP EVPN Updates about host Café MAC/IP addresses. Note that RTs are used only in Control Plane while L2VNI-Id is only used in Data Plane in VXLAN header.

Figure 12-2: intra-L2VNI routing

evpn   vni 10000 l2     rd auto     route-target import auto     route-target export auto

Example 12-1: L2VNI RT

If we look at the capture taken from the link between L-102 and Spine-11 we can verify the RT value of MAC advertisement sent by L-102.

Capture 12-1: Route-Type2 MAC advertisement about host Cafe

We can also verify this from the L-101 (example 12-2). Note that there is also RT 65000:10077 which is used for inter-L2VNI.

Leaf-101# sh bgp L2vpn evpn 192.168.11.11 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 192.168.77.101:32777    (L2VNI 10000) BGP routing table entry for [2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272, version 210 Paths: (1 available, best #1) Flags: (0x000212) on xmit-list, is in l2rib/evpn, is not in HW   Advertised path-id 1   Path type: internal, path is valid, is best path, in rib              Imported from 192.168.77.102:32777:[2]:[0]:[0]:[48]:[1000.0010.cafe]:[32]:[192.168.11.11]/272   AS-Path: NONE, path sourced internal to AS     192.168.100.102 (metric 81) from 192.168.77.11 (192.168.77.111)       Origin IGP, MED not set, localpref 100, weight 0       Received label 10000 10077       Extcommunity: RT:65000:10000 RT:65000:10077 ENCAP:8 Router MAC:5e00.0008.0007       Originator: 192.168.77.102 Cluster list: 192.168.77.111   Path-id 1 not advertised to any peer

Example 12-2: L2VNI RT

Inter-L2VNI routing

Inter-L2VNI routing use RT defined under the VRF Context configuration. Since we are using iBGP inside VXLAN fabric, we can use the auto-generated value. In the case of eBGP, especially when all VTEPs has unique AS number, RT has to be defined manually. 

Figure 12-3: Inter-L2VNI routing

Example 12-3 shows the example configuration. If we remove the command route-target both auto evpn, we will lose the inter-L2VNI connectivity as can be seen from the example 12-4.

Note! Configuration is a platform and OS version specific.

vrf context TENANT77   vni 10077   rd auto   address-family ipv4 unicast     route-target both auto evpn

Example 12-3: L2VNI RT

1)------> Remove RT both auto evpn Leaf-101(config-vrf-af-ipv4)# where   conf; vrf context TENANT77; address-family ipv4 unicast      admin@Leaf-101%default Leaf-101(config-vrf-af-ipv4)# Leaf-101(config-vrf-af-ipv4)# no route-target both auto evpn 2)-----> ping from Cafe to Abba stops working Cafe#ping 192.168.12.11 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.12.11, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) 3)-----> Add RT both auto evpn Leaf-101(config-vrf-af-ipv4)# Leaf-101(config-vrf-af-ipv4)# route-target both auto evpn Leaf-101(config-vrf-af-ipv4)# 4)-----> ping from Cafe to Abba start working again Cafe#ping 192.168.12.11 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.12.11, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 28/38/51 ms

Example 12-4: Removing/Adding RT auto both evpn

As I have explained in my previous posts, there are two BGP updates per host in VXLAN BGP EVPN fabric. First one is only for the MAC information and the second one is about MAC-IP information. Capture 12-2 shows that MAC-IP update carries both L2VNI and L3VNI specific Route-Targets.

Capture 12-2: Route-Type2 MAC-IP advertisement about host Cafe

External BGP to BGP EVPN and another way around

We can see in Figure 12-4, that Ext-Ro02 advertises network 172.16.77.0/24 via eBGP to Border-PE L-103. Interface E1/7 in Leaf-103 is attached to VRF TENANT77 and L-103 installs a route to network 172.16.77.0/24 to vrf TENANT77 RIB. L-103 advertises route as a Route-Type 5 (IP prefix) to BGP EVPN by using RT: 65000:10077. Export concerning to IPv4 routes are exported based on configuration under IPv4 Unicast section under VRF context TENANT77. Note, that we only need to export the route to BGP EVPN. That is why only route-target export 65000:10077 is needed. I have used also command route-target both auto in my previous posts. This is because if we manually specify RT import and export, we have to define the RT target value manually since the option “auto” is only available as an option to RT both –command.

To be able to advertise internal networks of VXLAN fabric to external peer Ext-Ro02, L-103 first import routes based on RT 65000:10077. Then it advertises a route to an external peer.

vrf context TENANT77   vni 10077   rd auto   address-family ipv4 unicast     route-target export 65000:10077     route-target both auto evpn

Example 12-5: Configuring RT for Inter-L2VNI under VRF.

Figure 12-4: External connection using eBGP

We can see from the example 12-6 that by removing command route-target export 65000:10077 we lose connectivity.

1)-------------------------------------> Remove export Leaf-103(config-vrf-af-ipv4)# no route-target export 65000:10077 2)-------------------------------------> Ping stop working Ext-Ro02#ping 192.168.11.12 so 172.16.77.1 <snipped> ..... Success rate is 0 percent (0/5) 3)-------------------------------------> Add export Leaf-103(config-vrf-af-ipv4)# route-target export 65000:10077 4)--------------------------------------> Ping start working Ext-Ro02#ping 192.168.11.12 so 172.16.77.1 <snipped> !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 23/29/36 ms

Example 12-6: Removing/Adding RT

Among the RT export, we need to advertise VXLAN internal networks to an external peer. This is achieved by using the command advertise l2vpn evpn.

router bgp 65000   vrf TENANT77     address-family ipv4 unicast       advertise l2vpn evpn       aggregate-address 192.168.11.0/24 summary-only     neighbor 10.103.77.2       remote-as 64577       address-family ipv4 unicast         send-community

Example 12-7: advertise l2vpn evpn on Leaf-103.

If we remove the command, we lose the connection to the external network as can be seen from example 12-8.

1) -------------------------------> Remove command Leaf-103(config-router-vrf-af)# no advertise l2vpn evpn 2)--------------------------------> ping stop working Ext-Ro02#ping 192.168.11.12 so 172.16.77.1 <snipped> ..... Success rate is 0 percent (0/5) ----------------------------------> Add command Leaf-103(config-router-vrf-af)# advertise l2vpn evpn ----------------------------------> ping start working Ext-Ro02#ping 192.168.11.12 so 172.16.77.1 <snipped> !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 15/19/27 ms

Example 12-8: testing

Example 12-9 shows that Ext-Leaf-103 has a routing entry for network 172.16.77.0/24 in its BGP table.

Leaf-103# sh ip bgp vrf TENANT77 172.16.77.0 BGP routing table information for VRF TENANT77, address family IPv4 Unicast BGP routing table entry for 172.16.77.0/24, version 21 Paths: (1 available, best #1) Flags: (0x80c041a) on xmit-list, is in urib, is best urib route, is in HW, expor ted   vpn: version 20, (0x100002) on xmit-list   Advertised path-id 1, VPN AF advertised path-id 1   Path type: external, path is valid, is best path, in rib   AS-Path: 64577 , path sourced external to AS     10.103.77.2 (metric 0) from 10.103.77.2 (172.16.77.1)       Origin IGP, MED 0, localpref 100, weight 0       Extcommunity: RT:65000:10077   VRF advertise information:   Path-id 1 not advertised to any peer   VPN AF advertise information:   Path-id 1 not advertised to any peer0         32768 i

Example 12-9: Leaf-103 VRF TEANANT77 BGP table.

Example 12-10 shows that route is advertised as BGP EVPN Route-Type 5 to Spine-11 (192.168.77.11) with RT: 65000:10077.

Leaf-103# sh bgp l2vpn evpn 172.16.77.0 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 192.168.77.103:3    (L3VNI 10077) BGP routing table entry for [5]:[0]:[0]:[24]:[172.16.77.0]:[0.0.0.0]/224, version 720 Paths: (1 available, best #1) Flags: (0x000002) on xmit-list, is not in l2rib/evpn   Advertised path-id 1   Path type: local, path is valid, is best path   AS-Path: 64577 , path sourced external to AS     192.168.100.103 (metric 0) from 0.0.0.0 (192.168.77.103)       Origin IGP, MED 0, localpref 100, weight 0       Received label 10077       Extcommunity: RT:65000:10077 ENCAP:8 Router MAC:5e00.0004.0007   Path-id 1 advertised to peers:     192.168.77.11

Example 12-10: BGP L2EVPN

BGP VPNv4 to BGP EVPN and another way around

In figure 12-5, there is an external router Ext-Ro03, which has vrf TEANANT88 (note that I use different VRF name than in VXLAN fabric). Network 172.16.88.0/24 is attached to VRF TENANT88 and advertised by MP-BGP over MPLS network to Leaf-104. Note that Leaf-104 is CSR-1000v. Leaf-104 import VPNv4 routes to EVPN and advertises it to Spine-11, which in turn advertises it to Leaf-101.

Leaf-104 receives Route-Type 5 BGP EVPN Update about IP prefix 192.168.11.0/24. Leaf-104 import EVPN routes to VPNv4 and advertises it to Ext-Ro03.

Figure 12-5: External connection over the MPLS network

We need two commands under the BGP process to importin EVPN routes to VPNv4 and vice versa. Under address-family vpnv4 (1/125) we define that routes received from the EVPN are re-originated to VPNV4 and under address-family l2vpn evpn (afi25/70) we define that routes received from the VPNv4 are re-originated to the EVPN.

Under VRF definition TENANT77 afi IPv4, we have two RT values. First RT 64588:10 is associated with a VPNv4 peer (external). Second RT 65000:10000, with stitching definition, is associated with the BGP EVPN peers (internal). This is how things are done in CSR1000v, in other OS commands might be different.

router bgp 65000  bgp router-id 192.168.77.104  bgp log-neighbor-changes  no bgp default ipv4-unicast  neighbor 10.104.88.3 remote-as 64588  neighbor 192.168.77.11 remote-as 65000  neighbor 192.168.77.11 update-source Loopback77  !  address-family vpnv4   import l2vpn evpn re-originate   neighbor 10.104.88.3 activate   neighbor 10.104.88.3 send-community extended  exit-address-family  !  address-family l2vpn evpn   import vpnv4 unicast re-originate   neighbor 192.168.77.11 activate   neighbor 192.168.77.11 send-community both  exit-address-family ! vrf definition TENANT77  rd 192.168.77.104:3  !  address-family ipv4   route-target export 64588:10   route-target import 64588:10   route-target export 65000:10077 stitching   route-target import 65000:10077 stitching  exit-address-family

Example 12-11: BGP and Export RT configuration on Leaf-104.

In example 12-12, we can see that Leaf-104 has received MP-BGP Update about 172.16.88.0/24 from Ext-Ro03. Note that the Route-Distinguisher value 64588:10 has been configured under the vrf TENANT88 in Ext-Ro03.

Leaf-104#sh ip bgp vpnv4 vrf TENANT77 172.16.88.0 BGP routing table entry for 192.168.77.104:3:172.16.88.0/24, version 3 Paths: (1 available, best #1, table TENANT77)   Not advertised to any peer   Refresh Epoch 1   64588, imported path from 64588:10:172.16.88.0/24 (global)     10.104.88.3 (via default) from 10.104.88.3 (10.104.88.3)       Origin IGP, metric 0, localpref 100, valid, external, best       Extended Community: RT:64588:10       Local vxlan vtep:         vrf:TENANT77, vni:10077         local router mac:0077.0077.0077         encap:8         vtep-ip:192.168.100.104         bdi:BDI77       mpls labels in/out nolabel/16       rx pathid: 0, tx pathid: 0x0

Example 12-12: Leaf-104 afi vpnv4: Network 172.16.88.0/24

In example 12-13, we can see that the routing information is also imported to EVPN.

Leaf-104#sh bgp l2vpn evpn BGP table version is 14, local router ID is 192.168.77.104 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,               x best-external, a additional-path, c RIB-compressed,               t secondary path, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found      Network          Next Hop            Metric LocPrf Weight Path Route Distinguisher: 192.168.77.101:32777  *>i  [2][192.168.77.101:32777][0][48][10000010BEEF][32][192.168.11.12]/24                       192.168.100.101               100      0 i Route Distinguisher: 192.168.77.101:32787  *>i  [2][192.168.77.101:32787][0][48][20000020ABBA][32][192.168.12.11]/24                       192.168.100.101               100      0 i Route Distinguisher: 192.168.77.102:32777  *>i  [2][192.168.77.102:32777][0][48][10000010CAFE][32][192.168.11.11]/24                       192.168.100.102               100      0 i Route Distinguisher: 192.168.77.103:3  *>i  [5][192.168.77.103:3][0][24][172.16.77.0]/17                       192.168.100.103          0    100      0 64577 i  *>i  [5][192.168.77.103:3][0][24][192.168.11.0]/17      Network          Next Hop            Metric LocPrf Weight Path                       192.168.100.103               100      0 i Route Distinguisher: 192.168.77.104:3 (default for vrf TENANT77)  *>   [5][192.168.77.104:3][0][24][172.16.88.0]/17                       10.104.88.3              0             0 64588 i

Example 12-13: Leaf-104 BGP afi EVPN Network 172.16.88.0/24.

In example 12-14, we can that Leaf-101 has received BGP EVPN Route-Type 5 updates about network 172.16.88.0/24, originated by Leaf-104, from Spine-11.

Leaf-101# sh bgp l2vpn evpn 172.16.88.0 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 192.168.77.104:3 BGP routing table entry for [5]:[0]:[0]:[24]:[172.16.88.0]:[0.0.0.0]/224, version 4 Paths: (1 available, best #1) Flags: (0x000002) on xmit-list, is not in l2rib/evpn, is not in HW   Advertised path-id 1   Path type: internal, path is valid, is best path              Imported to 2 destination(s)   AS-Path: 64588 , path sourced external to AS     192.168.100.104 (metric 81) from 192.168.77.11 (192.168.77.111)       Origin IGP, MED 0, localpref 100, weight 0       Received label 10077       Extcommunity: RT:65000:10077 ENCAP:8 Router MAC:0077.0077.0077       Originator: 192.168.77.104 Cluster list: 192.168.77.111   Path-id 1 not advertised to any peer Route Distinguisher: 192.168.77.101:3    (L3VNI 10077) BGP routing table entry for [5]:[0]:[0]:[24]:[172.16.88.0]:[0.0.0.0]/224, version 6 Paths: (1 available, best #1) Flags: (0x000002) on xmit-list, is not in l2rib/evpn, is not in HW   Advertised path-id 1   Path type: internal, path is valid, is best path              Imported from 192.168.77.104:3:[5]:[0]:[0]:[24]:[172.16.88.0]:[0.0.0.0]/224   AS-Path: 64588 , path sourced external to AS     192.168.100.104 (metric 81) from 192.168.77.11 (192.168.77.111)       Origin IGP, MED 0, localpref 100, weight 0       Received label 10077       Extcommunity: RT:65000:10077 ENCAP:8 Router MAC:0077.0077.0077       Originator: 192.168.77.104 Cluster list: 192.168.77.111   Path-id 1 not advertised to any peer

Example 12-14: Leaf-101 BGP afi EVPN Network 172.16.88.0/24

In example 12-15, we can see that Leaf-104 has received BGP EVPN Route-type 5 about subnet 192.168.11.0/24. Note that the output is slightly different in CSR1000v 16.6.1 than in NX-OS 7.0.3.

Leaf-104#sh bgp l2vpn evpn BGP table version is 14, local router ID is 192.168.77.104 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,               x best-external, a additional-path, c RIB-compressed,               t secondary path, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found      Network          Next Hop            Metric LocPrf Weight Path Route Distinguisher: 192.168.77.101:32777  *>i  [2][192.168.77.101:32777][0][48][10000010BEEF][32][192.168.11.12]/24                       192.168.100.101               100      0 i Route Distinguisher: 192.168.77.101:32787  *>i  [2][192.168.77.101:32787][0][48][20000020ABBA][32][192.168.12.11]/24                       192.168.100.101               100      0 i Route Distinguisher: 192.168.77.102:32777  *>i  [2][192.168.77.102:32777][0][48][10000010CAFE][32][192.168.11.11]/24                       192.168.100.102               100      0 i Route Distinguisher: 192.168.77.103:3  *>i  [5][192.168.77.103:3][0][24][172.16.77.0]/17                       192.168.100.103          0    100      0 64577 i  *>i  [5][192.168.77.103:3][0][24][192.168.11.0]/17      Network          Next Hop            Metric LocPrf Weight Path                       192.168.100.103               100      0 i Route Distinguisher: 192.168.77.104:3 (default for vrf TENANT77)  *>   [5][192.168.77.104:3][0][24][172.16.88.0]/17                       10.104.88.3              0             0 64588 i

Example 12-15: Leaf-104 BGP afi EVPN: Network 192.168.11.0/24

In example 12-16, we can see that Leaf-104 has import route 192.168.11.0/24 to VPNV4. The RT 64588:10 is used for VPNv4 as can be seen from the output. Note, I am using Bridge-Domain and Interface BDI77 instead of using VLAN and Interface VLAN in CSR1000v.

Leaf-104#sh ip bgp vpnv4 vrf TENANT77 192.168.11.0 BGP routing table entry for 192.168.77.104:3:192.168.11.0/24, version 15 Paths: (1 available, best #1, table TENANT77)   Advertised to update-groups:      2           Refresh Epoch 1   Local, (aggregated by 0 0.0.0.0), imported path from [5][192.168.77.103:3][0][24][192.168.11.0]/17     192.168.100.103 (metric 42) (via default) from 192.168.77.11 (192.168.77.111)       Origin IGP, localpref 100, valid, internal, atomic-aggregate, best       Extended Community: RT:64588:10 ENCAP:8 Router MAC:5E00.0004.0007       Originator: 192.168.77.103, Cluster list: 192.168.77.111       Local vxlan vtep:         vrf:TENANT77, vni:10077         local router mac:0077.0077.0077         encap:8         vtep-ip:192.168.100.104         bdi:BDI77       Remote VxLAN:         Topoid 0x1(vrf TENANT77)         Remote Router MAC:5E00.0004.0007         Encap 8         Egress VNI 10077         RTEP 192.168.100.103       mpls labels in/out 32/nolabel       rx pathid: 0, tx pathid: 0x0

Example 12-16: Leaf-104 BGP afi VPNV4: Network 192.168.11.0/24

In example 12-17, we can see that Ext-Ro03 has received MP-BGP update and also has imported the route to vrf TENANT88 BGP table.

Ext-Ro03#sh ip bgp vpnv4 vrf TENANT88 192.168.11.0 BGP routing table entry for 64588:10:192.168.11.0/24, version 8 Paths: (1 available, best #1, table TENANT88)   Not advertised to any peer   Refresh Epoch 1   65000, (aggregated by 0 0.0.0.0), imported path from 192.168.77.104:3:192.168.11.0/24 (global)     10.104.88.104 (via default) from 10.104.88.104 (192.168.77.104)       Origin IGP, localpref 100, valid, external, atomic-aggregate, best       Extended Community: RT:64588:10 ENCAP:8 Router MAC:5E00.0004.0007       mpls labels in/out nolabel/32       rx pathid: 0, tx pathid: 0x0

Example 12-17: Ext-Ro03 BGP afi VPNv4: Network 192.168.11.0/24

In example 12-18, we can see that since we have not done any aggregation in Leaf-104, Ext-Ro03 has also received all the host routes from Leaf-104.

Ext-Ro03#sh ip bgp vpnv4 vrf TENANT88             BGP table version is 12, local router ID is 10.104.88.3 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,               r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,               x best-external, a additional-path, c RIB-compressed,               t secondary path, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found      Network          Next Hop            Metric LocPrf Weight Path Route Distinguisher: 64588:10 (default for vrf TENANT88)  *>   172.16.77.0/24   10.104.88.104                          0 65000 64577 i  *>   172.16.88.0/24   0.0.0.0                  0         32768 i  *>   192.168.11.0     10.104.88.104                          0 65000 i  *>   192.168.11.11/32 10.104.88.104                          0 65000 i  *>   192.168.11.12/32 10.104.88.104                          0 65000 i  *>   192.168.12.11/32 10.104.88.104                          0 65000 i

Example 12-18: Ext-Ro03 vrf TENANT88 BGP table.

I am using Bridge Domain with BDI interface in Leaf-104 (CSR1000v has no VLANs) while in Leaf-101 I am using VLAN with VLAN interface. Now if I add command encapsulation dot1q 10 xx to Interface BDI77, I will break the connection as can be seen from example 12-19.

1) ------------------------------------> Add vlan encapsulation Leaf-104(config)#int BDI77 Leaf-104(config-if)#encapsulation dot1Q 77 2) ------------------------------------> Ping stops working Ext-Ro03#ping vrf TENANT88 192.168.11.12 so 172.16.88.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.11.12, timeout is 2 seconds: Packet sent with a source address of 172.16.88.1 ..... Success rate is 0 percent (0/5) 3)-------------------------------------> Remove dot1q encapsulation Leaf-104(config-if)# Leaf-104(config-if)#no encapsulation dot1Q 77 Leaf-104(config-if)# 4)-------------------------------------> Ping starts working again Ext-Ro03#ping vrf TENANT88 192.168.11.12 so 172.16.88.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.11.12, timeout is 2 seconds: Packet sent with a source address of 172.16.88.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 14/19/24 ms

Example 12-19: Interface BDI77 in Leaf-104.

The reason for this can be seen if we compare the ICMP request packets received by Leaf-101 without dot1q encapsulation (figure 12-6) and with dot1q encapsulation (figure 12-7). When we do not have encapsulation definition under BDI interface, the packet is forwarded correctly. 

Figure 12-6: ICMP request from Ext-Ro01 172.16.88.1 to host Beef 192.168.11.12.

Now if we take a capture when dot1q encapsulation is used under BDI interface, we can see that there is an additional 802.1Q TAG with VLAN Id 77 and VTEPs in this lab does not forward frames received from other VTEPS based on VLAN-Id/MAC but L2VNI-Id and MAC.

Figure 12-7: ICMP request from Ext-Ro01 172.16.88.1 to host Beef 192.168.11.12.

And that’s all!

Author: Toni Pasanen CCIE#28158

Published: 25.9.2018

-------------------------------------------------

References:

Building Data Center with VXLAN BGP EVPN – A Cisco NX-OS Perspective

ISBN-10: 1-58714-467-0 – Krattiger Lukas, Shyam Kapadia, and Jansen Davis

Carrier Ethernet Configuration Guide:

Chapter: EVPN VxLAN L3

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/cether/configuration/xe-16/ce-xe-16-book/evpn-vxlan-l3.html#concept_898783137917309440080471753420713

Appendix 1

Device configurations

Note that there is also some configurations that are not related to this lab!

Leaf-103# sh run !Command: show running-config !Time: Tue Sep 25 13:25:24 2018 version 7.0(3)I7(1) hostname Leaf-103 install feature-set mpls vdc Leaf-103 id 1   allow feature-set mpls   limit-resource vlan minimum 16 maximum 4094   limit-resource vrf minimum 2 maximum 4096   limit-resource port-channel minimum 0 maximum 511   limit-resource u4route-mem minimum 248 maximum 248   limit-resource u6route-mem minimum 96 maximum 96   limit-resource m4route-mem minimum 58 maximum 58   limit-resource m6route-mem minimum 8 maximum 8 feature-set mpls nv overlay evpn feature ospf feature bgp feature pim feature fabric forwarding feature interface-vlan feature vn-segment-vlan-based feature nv overlay no password strength-check username admin password 5 $5$XYynpOKA$1s1Y/xDSWW1x48yz8ky//ZYUpjN1Xbkstu5Rzoqkxp0  role network-admin ip domain-lookup snmp-server user admin network-admin auth md5 0x3f07dbb8731ae864dbafa9286555828d priv 0x3f07dbb8731ae864dbafa9286555828d localizedkey rmon event 1 description FATAL(1) owner PMON@FATAL rmon event 2 description CRITICAL(2) owner PMON@CRITICAL rmon event 3 description ERROR(3) owner PMON@ERROR rmon event 4 description WARNING(4) owner PMON@WARNING rmon event 5 description INFORMATION(5) owner PMON@INFO fabric forwarding anycast-gateway-mac 0001.0001.0001 ip pim rp-address 192.168.238.1 group-list 238.0.0.0/24 bidir ip pim ssm range 232.0.0.0/8 vlan 1,10,20,77 vlan 10   name L2VNI-for-VLAN10   vn-segment 10000 vlan 20   name L2VNI-for-VLAN20   vn-segment 20000 vlan 77   name TENANT77   vn-segment 10077 vrf context TENANT77   vni 10077   rd auto   address-family ipv4 unicast     route-target export 65000:10077     route-target both auto evpn vrf context management hardware access-list tcam region racl 512 hardware access-list tcam region arp-ether 256 double-wide interface Vlan1   no shutdown interface Vlan10   no shutdown   vrf member TENANT77   ip address 192.168.11.1/24   fabric forwarding mode anycast-gateway interface Vlan20   no shutdown   vrf member TENANT77   ip address 192.168.12.1/24   fabric forwarding mode anycast-gateway interface Vlan77   no shutdown   vrf member TENANT77   ip forward interface nve1   no shutdown   host-reachability protocol bgp   source-interface loopback100   member vni 10000     suppress-arp     mcast-group 238.0.0.10   member vni 10077 associate-vrf   member vni 20000     suppress-arp     mcast-group 238.0.0.10 interface Ethernet1/1   no switchport   medium p2p   ip unnumbered loopback0   ip ospf network point-to-point   ip router ospf UNDERLAY-NET area 0.0.0.0   ip pim sparse-mode   no shutdown interface Ethernet1/2   no switchport   medium p2p   ip unnumbered loopback0   ip ospf network point-to-point   ip router ospf UNDERLAY-NET area 0.0.0.0   ip pim sparse-mode   no shutdown interface Ethernet1/3   no switchport   no shutdown interface Ethernet1/4 interface Ethernet1/5 interface Ethernet1/6 interface Ethernet1/7   no switchport   vrf member TENANT77   ip address 10.103.77.103/24   no shutdown <interface information snipped> interface mgmt0   vrf member management interface loopback0   description ** RID/Underlay **   ip address 192.168.0.103/32   ip router ospf UNDERLAY-NET area 0.0.0.0   ip pim sparse-mode interface loopback77   description ** BGP peering **   ip address 192.168.77.103/32   ip router ospf UNDERLAY-NET area 0.0.0.0 interface loopback100   description ** VTEP/Overlay **   ip address 192.168.100.103/32   ip router ospf UNDERLAY-NET area 0.0.0.0   ip pim sparse-mode line console line vty router ospf UNDERLAY-NET   router-id 192.168.0.103   name-lookup router bgp 65000   router-id 192.168.77.103   timers bgp 3 9   address-family ipv4 unicast   address-family vpnv4 unicast   address-family l2vpn evpn   neighbor 10.103.78.3     remote-as 64588     description ** Ext-Ro03 **     address-family vpnv4 unicast       send-community       send-community extended       import l2vpn evpn reoriginate   neighbor 192.168.77.11     remote-as 65000     description ** Spine-11 BGP-RR **     update-source loopback77     address-family l2vpn evpn       send-community extended   vrf TENANT   vrf TENANT77     address-family ipv4 unicast       advertise l2vpn evpn       aggregate-address 192.168.11.0/24 summary-only     neighbor 10.103.77.2       remote-as 64577       address-family ipv4 unicast         send-community         send-community extended evpn   vni 10000 l2     rd auto     route-target import auto     route-target export auto   vni 20000 l2     rd auto     route-target import auto     route-target export auto Leaf-103#

Ext-Ro02#sh run Building configuration... Current configuration : 3517 bytes ! ! Last configuration change at 06:23:29 UTC Tue Sep 25 2018 ! version 15.6 service timestamps debug datetime msec service timestamps log datetime msec no service password-encryption ! hostname Ext-Ro02 ! boot-start-marker boot-end-marker ! ! ! no aaa new-model ! ! ! mmi polling-interval 60 no mmi auto-configure no mmi pvc mmi snmp-timeout 180 ! ! ! ! ! ! ! ! ! ! ! ip cef no ipv6 cef ! multilink bundle-name authenticated ! ! ! interface Loopback172  description ** net 172.16.77.0/24 **  ip address 172.16.77.1 255.255.255.0 ! interface GigabitEthernet0/0  ip address 10.255.4.165 255.255.0.0  shutdown  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/1  mac-address 2000.0020.babe  ip address 10.103.77.2 255.255.255.0  duplex auto  speed auto  media-type rj45 ! router bgp 64577  bgp log-neighbor-changes  network 172.16.77.0 mask 255.255.255.0  neighbor 10.103.77.103 remote-as 65000 ! ip forward-protocol nd ! Ext-Ro02#

Leaf-104#sh run Building configuration... Current configuration : 6634 bytes ! ! Last configuration change at 06:31:34 UTC Tue Sep 25 2018 ! version 16.6 service config service timestamps debug datetime msec service timestamps log datetime msec platform qfp utilization monitor load 80 no platform punt-keepalive disable-kernel-core platform console serial ! hostname Leaf-104 ! boot-start-marker boot-end-marker ! ! vrf definition TENANT77  rd 192.168.77.104:3  !  address-family ipv4   route-target export 64588:10   route-target import 64588:10   route-target export 65000:10077 stitching   route-target import 65000:10077 stitching  exit-address-family ! enable secret 5 $1$s8Sc$pvAUHDgeooHUUIkFFtbb80 ! no aaa new-model ! ! ! ! ! ! ! ! ! no ip domain lookup ! !         ! ! ! ! ! ! ! ! subscriber templating ! ! ! ! ! ! ! ! license udi pid CSR1000V sn 9Q262SJZZMA diagnostic bootup level minimal spanning-tree extend system-id !         ! ! username cisco privilege 15 secret 5 $1$AUCD$bD8rjCf9BrKrxjfhLQy4x/ ! redundancy bridge-domain 10  member vni 10000 ! bridge-domain 20  member vni 20000 ! bridge-domain 77  member vni 10077 ! ! ! ! interface Loopback0  description ** RID/Underlay **  ip address 192.168.0.104 255.255.255.255  ip ospf 1 area 0 ! interface Loopback10  no ip address ! interface Loopback77  description ** BGP peering **  ip address 192.168.77.104 255.255.255.255  ip ospf 1 area 0 ! interface Loopback100  description ** VTEP/Overlay **  ip address 192.168.100.104 255.255.255.255  ip pim sparse-mode  ip ospf 1 area 0 ! interface GigabitEthernet1  ip address 10.255.4.162 255.255.0.0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet2  ip unnumbered Loopback0  ip pim sparse-mode  ip ospf network point-to-point  ip ospf 1 area 0  negotiation auto  no mop enabled  no mop sysid ! interface GigabitEthernet3  ip address 10.104.88.104 255.255.255.0  ip ospf 1 area 0  negotiation auto  mpls ip  mpls bgp forwarding  no mop enabled  no mop sysid ! interface nve1  no ip address  source-interface Loopback100  host-reachability protocol bgp  member vni 10077 vrf TENANT77  no mop enabled  no mop sysid ! interface BDI77  mac-address 0077.0077.0077  vrf forwarding TENANT77  ip unnumbered Loopback100  no mop enabled  no mop sysid ! router ospf 1 ! router ospf 2 ! router bgp 65000  bgp router-id 192.168.77.104  bgp log-neighbor-changes  no bgp default ipv4-unicast  neighbor 10.104.88.3 remote-as 64588  neighbor 192.168.77.11 remote-as 65000  neighbor 192.168.77.11 update-source Loopback77  !  address-family vpnv4   import l2vpn evpn re-originate   neighbor 10.104.88.3 activate   neighbor 10.104.88.3 send-community extended  exit-address-family  !  address-family l2vpn evpn   import vpnv4 unicast re-originate   neighbor 192.168.77.11 activate   neighbor 192.168.77.11 send-community both  exit-address-family  !  address-family ipv4 vrf TENANT77   advertise l2vpn evpn  exit-address-family ! ! virtual-service csr_mgmt ! ip forward-protocol nd no ip http server ip http authentication local no ip http secure-server ip http client source-interface GigabitEthernet1 ip pim rp-address 192.168.238.1 1 ip pim ssm range 1 ! ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr ip ssh client algorithm encryption aes128-ctr aes192-ctr aes256-ctr ! !         ip access-list standard group-list access-list 1 permit 232.0.0.0 0.0.0.255 access-list 1 permit 238.0.0.0 0.0.0.255 ! ! control-plane ! ! ! end Leaf-104#

Ext-Ro03#sh run Building configuration... Current configuration : 3576 bytes ! ! Last configuration change at 06:23:15 UTC Tue Sep 25 2018 ! version 15.6 service timestamps debug datetime msec service timestamps log datetime msec no service password-encryption ! hostname Ext-Ro03 ! boot-start-marker boot-end-marker ! ! ! no aaa new-model ! ! ! mmi polling-interval 60 no mmi auto-configure no mmi pvc mmi snmp-timeout 180 ! ! ip vrf TENANT88  rd 64588:10  route-target export 64588:10  route-target import 64588:10 ! ! ! ! ip cef no ipv6 cef ! multilink bundle-name authenticated !         ! ! redundancy ! ! ! ! interface Loopback88  ip vrf forwarding TENANT88  ip address 172.16.88.1 255.255.255.0 !         interface GigabitEthernet0/0  ip address 10.255.4.166 255.255.0.0  shutdown  duplex auto  speed auto  media-type rj45 ! interface GigabitEthernet0/1  ip address 10.104.88.3 255.255.255.0  ip ospf 1 area 0  duplex auto  speed auto  media-type rj45  mpls bgp forwarding  mpls ip ! router ospf 1 ! router bgp 64588  bgp log-neighbor-changes  no bgp default ipv4-unicast  neighbor 10.104.88.104 remote-as 65000  !         address-family vpnv4   neighbor 10.104.88.104 activate   neighbor 10.104.88.104 send-community extended  exit-address-family  !  address-family ipv4 vrf TENANT88   network 172.16.88.0 mask 255.255.255.0   redistribute connected  exit-address-family ! ip forward-protocol nd ! ip bgp-community new-format ! no ip http server no ip http secure-server ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr ip ssh client algorithm encryption aes128-ctr aes192-ctr aes256-ctr ! Ext-Ro03#