SONiC Deep Dive: BGP Update Message Processing

 

BGP Route Advertisement

The next step after configuring BGP peering using BGP Unnumbered with IPv6 link-local addresses is to advertise the host networks. The lower part of Figure 6-10 shows the sonic-cli configuration commands used to advertise the local VLAN subnet on each leaf. On Leaf-101, we advertise 10.0.10.0/24, which is the subnet for VLAN 10. On Leaf-102, we advertise 10.0.20.0/24, which is the subnet for VLAN 20.

By examining the BGP Loc-RIB on Leaf-102, for example, we can see that the next hop for 10.0.10.0/24, originated on Leaf-101, is the IPv6 link-local address of Spine-11's Ethernet0 interface.

Figure 6-10: IPv4 Routes Installed in the BGP Loc-RIB Table.

Figure 6-11 verifies that the host subnets learned through BGP have been installed in the IP routing table on all switches. For example, the destination prefix 10.0.10.0/24 is installed on Leaf-102 with the next hop fe80::e22:34ff:feb6:a and the outgoing interface Ethernet0.

This also illustrates the separation between the BGP control plane and the data plane. BGP uses IPv6 link-local addresses to establish the BGP session and exchange BGP messages carrying IPv4 NLRI, with an IPv6 link-local address specified as the next hop. However, the resulting IPv4 route is used to forward ordinary IPv4 data packets without IPv6 encapsulation. In this example, the routing table resolves the next hop through the physical Ethernet0 interface; no tunnel interface or tunneling mechanism is involved.

Figure 6-11: IPv4 Routes Installed in the Routing Table.

Now that we have seen the result of route advertisement, we can look behind the CLI and follow how SONiC and FRRouting generate and send BGP UPDATE messages, process a received BGP UPDATE, select the best path, and advertise the selected route to another BGP peer.

 

FRRouting - Route Processing

The network 10.0.10.0/24 command under the global address-family ipv4 unicast section updates the BGP_GLOBALS_AF_NETWORK table on CONFIG_DB, as shown in Figure 6-12. Redis notifies frrcfgd about the configuration change. frrcfgd reads the updated BGP configuration and applies it to the running FRRouting software, where the bgpd daemon processes the BGP-specific configuration.

Based on this configuration, bgpd originates a BGP UPDATE message containing the IPv4 prefix 10.0.10.0/24. The UPDATE includes the AS_PATH path attribute, which contains AS 65101. It also includes the MP_REACH_NLRI path attribute, which specifies IPv4 Unicast as the address family, using an Address Family Identifier (AFI) of 1 and a Subsequent Address Family Identifier (SAFI) of 1. The MP_REACH_NLR` attribute specifies the IPv6 link-local address fe80::ecb:81ff:fe19:a as the BGP next hop and carries the IPv4 prefix 10.0.10.0/24 as the Network Layer Reachability Information (NLRI).

Figure 6-12: CONFIG_DB & BGP Update Message from Leaf-101.

 

BGP Update Message Pipeline

After receiving a BGP UPDATE, bgpd processes the advertised NLRI and makes it available for further BGP processing. For advertisement to a BGP peer, the NLRI is installed in the peer-specific Adj-RIB-Out, from which it is sent to the peer.

At the same time, the route is considered together with routes from other sources for the same destination. If it is selected as the best route, it is installed in the local routing system and eventually programmed into the switch ASIC's forwarding information base (FIB).

 

NLRI Update to BGP Peer

 

Figure 6-13 shows what happens when Spine-11 receives the BGP UPDATE message from Leaf-101. Because the destination MAC address belongs to Spine-11, the Ethernet frame is delivered to the Linux networking stack. The bgpd daemon listens on TCP port 179. The Linux networking stack therefore delivers the BGP message carried by the TCP connection to the bgpd daemon.

bgpd first validates the received BGP UPDATE according to the BGP session and message parameters. This includes validating the configured address family (AFI/SAFI) and the AS information according to the eBGP session. The routes received from the peer are then placed in the BGP Adj-RIB-In. Configured BGP ingress policies are applied to the received routes, and the BGP decision process evaluates the eligible paths. The best path is selected and installed in the BGP Loc-RIB.

Spine-11 can now advertise the selected route to Leaf-102. Before sending the route, bgpd applies the configured BGP egress policy for the Leaf-102 peer. The resulting route information is placed in the peer-specific Adj-RIB-Out.

Because Spine-11 and Leaf-102 have an eBGP session, Spine-11 advertises itself as the BGP next hop and adds its AS number to the beginning of the AS_PATH attribute. bgpd then constructs a new BGP UPDATE message for Leaf-102 and sends it over the established BGP session.

Leaf-102 receives the BGP UPDATE and processes it through the same sequence: validation, Adj-RIB-In, BGP policy, and the BGP decision process.

Figure 6-13: BGP Update Message Processing - BGP Message Pipeline.

 

Local Route Processing

After bgpd selects and installs the best route to destination 10.0.10.0/24 in the BGP Loc-RIB, the route is passed to zebra. Zebra installs the route into the Linux kernel routing table and, through its FPM interface, sends the route information to fpmsyncd in an FPM message. Fpmsyncd receives the message and publishes the route information to APPL_DB.

fpmsyncd creates the ROUTE_TABLE:10.0.10.0/24 entry with the nexthop_group field set to 5. The corresponding NEXT_HOP_GROUP_ TABLE:5 entry identifies the next hop through the nexthop field and the outgoing interface through the ifname field. In this case, nexthop is set to the IPv6 link-local address fe80::ecb:81ff:fe19:a, while ifname is set to Ethernet0.

In the SWSS container, the RouteOrch component of orchagent subscribes to APPL_DB and detects the new route entry. RouteOrch processes the route and programs the corresponding route and next-hop objects into ASIC_DB. The syncd daemon in the SyncD container subscribes to ASIC_DB, reads the updated objects, and uses the SAI API to program the corresponding forwarding objects through the vendor SDK. The vendor SDK then programs the forwarding information into the switch ASIC.

Figure 6-14: BGP Update Message Processing - Redis Database Pipeline.

Data Plane Operation

When a packet destined for 10.0.10.101 arrives at Spine-11 from Leaf-102, the ASIC performs a longest-prefix match on the destination IPv4 address. The address 10.0.10.101 matches the 10.0.10.0/24 route. Before the packet can be forwarded, the ASIC must determine the next hop, the egress interface, and the source and destination MAC addresses required to construct the new Ethernet header.

Let's now look at how SONiC represents this forwarding information in ASIC_DB. The following examples show the simplified SAI objects stored in ASIC_DB that are used to program the forwarding behavior. These objects represent the forwarding information in a vendor-independent SAI format, they are not the actual internal hardware tables of the switch ASIC. The vendor SDK translates the SAI objects into the vendor-specific representation used by the hardware.

The route entry programmed for the 10.0.10.0/24 prefix contains a reference to the next-hop object. In the corresponding ASIC_DB ROUTE_ENTRY, the SAI_ROUTE_ENTRY_ATTR_ NEXT_HOP_ID attribute identifies the next-hop object associated with this route. 

ROUTE_ENTRY

  dest: 10.0.10.0/24

  SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID:

      oid:0x4000000000a4a 

The corresponding next-hop object specifies the IPv6 link-local address fe80::ecb:81ff:fe19:a and references the router interface through SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID.

 

NEXT_HOP

  oid: 0x4000000000a4a

  SAI_NEXT_HOP_ATTR_IP:

      fe80::ecb:81ff:fe19:a

  SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID:

      oid:0x6000000000a48

  SAI_NEXT_HOP_ATTR_TYPE:

      SAI_NEXT_HOP_TYPE_IP

 The router interface identifies the egress port through ATTR_PORT_ID and provides the source MAC address used when the packet is transmitted. 

ROUTER_INTERFACE

  oid: 0x6000000000a48

  ATTR_PORT_ID:

      oid:0x1000000000002

  ATTR_SRC_MAC_ADDRESS:

      0C:22:34:B6:00:0A

  ATTR_VIRTUAL_ROUTER_ID:

      oid:0x300000000003a

 The port identified by ATTR_PORT_ID corresponds to Ethernet0, identifying the physical egress interface through which the packet is transmitted. 

PORT

  oid: 0x1000000000002

  SAI_HOSTIF_ATTR_NAME:

      Ethernet0

  SAI_HOSTIF_ATTR_OPER_STATUS:

      true

 The next-hop IPv6 address is resolved to a destination MAC address using the neighbor information. The neighbor entry associates fe80::ecb:81ff:fe19:a with the destination MAC address 0C:CB:81:19:00:0A.

 NEIGHBOR_ENTRY

  ip:  fe80::ecb:81ff:fe19:a

  rif: oid:0x6000000000a48

  ATTR_DST_MAC_ADDRESS:

      0C:CB:81:19:00:0A

 With this information, the ASIC has everything it needs to forward the packet: the destination prefix, the next-hop information, the egress interface, and the source and destination MAC addresses.

Figure 6-15 depicts the relationships between the ASIC_DB entries involved in packet forwarding in original format.

Figure 6-15: Object Relationship in ASIC_DB.

Before closing this chapter, let’s verify that the data plane works. The trace from Host-2 (10.0.20.102, VLAN 20), connected to Leaf-102, to Host-1 (10.0.10.101/24, VLAN 10), connected to Leaf-101, works as expected: the IPv6-only transport network correctly transmits IPv4 packets without tunneling.

Figure 6-16: Data Plane Test - Pinging Host-2 from Host-1.