SONiC Deep Dive: BGP Update Message Processing

 

BGP Route Advertisement

The next step after configuring BGP peering using BGP Unnumbered with IPv6 link-local addresses is to advertise the host networks. The lower part of Figure 6-10 shows the sonic-cli configuration commands used to advertise the local VLAN subnet on each leaf. On Leaf-101, we advertise 10.0.10.0/24, which is the subnet for VLAN 10. On Leaf-102, we advertise 10.0.20.0/24, which is the subnet for VLAN 20.

By examining the BGP Loc-RIB on Leaf-102, for example, we can see that the next hop for 10.0.10.0/24, originated on Leaf-101, is the IPv6 link-local address of Spine-11's Ethernet0 interface.

Figure 6-10: IPv4 Routes Installed in the BGP Loc-RIB Table.

Figure 6-11 verifies that the host subnets learned through BGP have been installed in the IP routing table on all switches. For example, the destination prefix 10.0.10.0/24 is installed on Leaf-102 with the next hop fe80::e22:34ff:feb6:a and the outgoing interface Ethernet0.

This also illustrates the separation between the BGP control plane and the data plane. BGP uses IPv6 link-local addresses to establish the BGP session and exchange BGP messages carrying IPv4 NLRI, with an IPv6 link-local address specified as the next hop. However, the resulting IPv4 route is used to forward ordinary IPv4 data packets without IPv6 encapsulation. In this example, the routing table resolves the next hop through the physical Ethernet0 interface; no tunnel interface or tunneling mechanism is involved.

Figure 6-11: IPv4 Routes Installed in the Routing Table.

Now that we have seen the result of route advertisement, we can look behind the CLI and follow how SONiC and FRRouting generate and send BGP UPDATE messages, process a received BGP UPDATE, select the best path, and advertise the selected route to another BGP peer.

 

FRRouting - Route Processing

The network 10.0.10.0/24 command under the global address-family ipv4 unicast section updates the BGP_GLOBALS_AF_NETWORK table on CONFIG_DB, as shown in Figure 6-12. Redis notifies frrcfgd about the configuration change. frrcfgd reads the updated BGP configuration and applies it to the running FRRouting software, where the bgpd daemon processes the BGP-specific configuration.

Based on this configuration, bgpd originates a BGP UPDATE message containing the IPv4 prefix 10.0.10.0/24. The UPDATE includes the AS_PATH path attribute, which contains AS 65101. It also includes the MP_REACH_NLRI path attribute, which specifies IPv4 Unicast as the address family, using an Address Family Identifier (AFI) of 1 and a Subsequent Address Family Identifier (SAFI) of 1. The MP_REACH_NLR` attribute specifies the IPv6 link-local address fe80::ecb:81ff:fe19:a as the BGP next hop and carries the IPv4 prefix 10.0.10.0/24 as the Network Layer Reachability Information (NLRI).

Figure 6-12: CONFIG_DB & BGP Update Message from Leaf-101.

 

BGP Update Message Pipeline

After receiving a BGP UPDATE, bgpd processes the advertised NLRI and makes it available for further BGP processing. For advertisement to a BGP peer, the NLRI is installed in the peer-specific Adj-RIB-Out, from which it is sent to the peer.

At the same time, the route is considered together with routes from other sources for the same destination. If it is selected as the best route, it is installed in the local routing system and eventually programmed into the switch ASIC's forwarding information base (FIB).

 

NLRI Update to BGP Peer

 

Figure 6-13 shows what happens when Spine-11 receives the BGP UPDATE message from Leaf-101. Because the destination MAC address belongs to Spine-11, the Ethernet frame is delivered to the Linux networking stack. The bgpd daemon listens on TCP port 179. The Linux networking stack therefore delivers the BGP message carried by the TCP connection to the bgpd daemon.

bgpd first validates the received BGP UPDATE according to the BGP session and message parameters. This includes validating the configured address family (AFI/SAFI) and the AS information according to the eBGP session. The routes received from the peer are then placed in the BGP Adj-RIB-In. Configured BGP ingress policies are applied to the received routes, and the BGP decision process evaluates the eligible paths. The best path is selected and installed in the BGP Loc-RIB.

Spine-11 can now advertise the selected route to Leaf-102. Before sending the route, bgpd applies the configured BGP egress policy for the Leaf-102 peer. The resulting route information is placed in the peer-specific Adj-RIB-Out.

Because Spine-11 and Leaf-102 have an eBGP session, Spine-11 advertises itself as the BGP next hop and adds its AS number to the beginning of the AS_PATH attribute. bgpd then constructs a new BGP UPDATE message for Leaf-102 and sends it over the established BGP session.

Leaf-102 receives the BGP UPDATE and processes it through the same sequence: validation, Adj-RIB-In, BGP policy, and the BGP decision process.

Figure 6-13: BGP Update Message Processing - BGP Message Pipeline.

 

Local Route Processing

After bgpd selects and installs the best route to destination 10.0.10.0/24 in the BGP Loc-RIB, the route is passed to zebra. Zebra installs the route into the Linux kernel routing table and, through its FPM interface, sends the route information to fpmsyncd in an FPM message. Fpmsyncd receives the message and publishes the route information to APPL_DB.

fpmsyncd creates the ROUTE_TABLE:10.0.10.0/24 entry with the nexthop_group field set to 5. The corresponding NEXT_HOP_GROUP_ TABLE:5 entry identifies the next hop through the nexthop field and the outgoing interface through the ifname field. In this case, nexthop is set to the IPv6 link-local address fe80::ecb:81ff:fe19:a, while ifname is set to Ethernet0.

In the SWSS container, the RouteOrch component of orchagent subscribes to APPL_DB and detects the new route entry. RouteOrch processes the route and programs the corresponding route and next-hop objects into ASIC_DB. The syncd daemon in the SyncD container subscribes to ASIC_DB, reads the updated objects, and uses the SAI API to program the corresponding forwarding objects through the vendor SDK. The vendor SDK then programs the forwarding information into the switch ASIC.

Figure 6-14: BGP Update Message Processing - Redis Database Pipeline.

Data Plane Operation

When a packet destined for 10.0.10.101 arrives at Spine-11 from Leaf-102, the ASIC performs a longest-prefix match on the destination IPv4 address. The address 10.0.10.101 matches the 10.0.10.0/24 route. Before the packet can be forwarded, the ASIC must determine the next hop, the egress interface, and the source and destination MAC addresses required to construct the new Ethernet header.

Let's now look at how SONiC represents this forwarding information in ASIC_DB. The following examples show the simplified SAI objects stored in ASIC_DB that are used to program the forwarding behavior. These objects represent the forwarding information in a vendor-independent SAI format, they are not the actual internal hardware tables of the switch ASIC. The vendor SDK translates the SAI objects into the vendor-specific representation used by the hardware.

The route entry programmed for the 10.0.10.0/24 prefix contains a reference to the next-hop object. In the corresponding ASIC_DB ROUTE_ENTRY, the SAI_ROUTE_ENTRY_ATTR_ NEXT_HOP_ID attribute identifies the next-hop object associated with this route. 

ROUTE_ENTRY

  dest: 10.0.10.0/24

  SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID:

      oid:0x4000000000a4a 

The corresponding next-hop object specifies the IPv6 link-local address fe80::ecb:81ff:fe19:a and references the router interface through SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID.

 

NEXT_HOP

  oid: 0x4000000000a4a

  SAI_NEXT_HOP_ATTR_IP:

      fe80::ecb:81ff:fe19:a

  SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID:

      oid:0x6000000000a48

  SAI_NEXT_HOP_ATTR_TYPE:

      SAI_NEXT_HOP_TYPE_IP

 The router interface identifies the egress port through ATTR_PORT_ID and provides the source MAC address used when the packet is transmitted. 

ROUTER_INTERFACE

  oid: 0x6000000000a48

  ATTR_PORT_ID:

      oid:0x1000000000002

  ATTR_SRC_MAC_ADDRESS:

      0C:22:34:B6:00:0A

  ATTR_VIRTUAL_ROUTER_ID:

      oid:0x300000000003a

 The port identified by ATTR_PORT_ID corresponds to Ethernet0, identifying the physical egress interface through which the packet is transmitted. 

PORT

  oid: 0x1000000000002

  SAI_HOSTIF_ATTR_NAME:

      Ethernet0

  SAI_HOSTIF_ATTR_OPER_STATUS:

      true

 The next-hop IPv6 address is resolved to a destination MAC address using the neighbor information. The neighbor entry associates fe80::ecb:81ff:fe19:a with the destination MAC address 0C:CB:81:19:00:0A.

 NEIGHBOR_ENTRY

  ip:  fe80::ecb:81ff:fe19:a

  rif: oid:0x6000000000a48

  ATTR_DST_MAC_ADDRESS:

      0C:CB:81:19:00:0A

 With this information, the ASIC has everything it needs to forward the packet: the destination prefix, the next-hop information, the egress interface, and the source and destination MAC addresses.

Figure 6-15 depicts the relationships between the ASIC_DB entries involved in packet forwarding in original format.

Figure 6-15: Object Relationship in ASIC_DB.

Before closing this chapter, let’s verify that the data plane works. The trace from Host-2 (10.0.20.102, VLAN 20), connected to Leaf-102, to Host-1 (10.0.10.101/24, VLAN 10), connected to Leaf-101, works as expected: the IPv6-only transport network correctly transmits IPv4 packets without tunneling.

Figure 6-16: Data Plane Test - Pinging Host-2 from Host-1.

SONiC Deep Dive: BGP Peer Configuration - BGP Unnumbered

 

After enabling IPv6 on the Ethernet0 interface, we can configure BGP peering. Instead of statically defining a peer IP address and AS number, the BGP neighbor is defined at the interface level. This tells BGP to expect a neighbor and initialize the BGP peering over Ethernet0. Because our intent is to transport IPv4 traffic over the IPv6 network, we activate both the IPv4 and IPv6 address families. We also configure the peer as an external BGP (eBGP) neighbor by using a command remote-as external. This tells BGP that the peer must use an AS number different from the local AS number.

Because the network uses IPv6 as the transport for IPv4 traffic, IPv4 Network Layer Reachability Information (NLRI) exchanged through BGP use IPv6 next-hop addresses. BGP therefore needs the Extended Next Hop Encoding capability to support IPv4 routes with IPv6 next hops. In this configuration, the capability is explicitly enabled with the capability extended-nexthop command.

In FRRouting, the v6only option controls which IP version is used for interface-based BGP Unnumbered peering. Without v6only, FRR uses the interface's IPv6 link-local address for peering only when no suitable IPv4 address is configured on the interface. With v6only, FRR skips searching for an IPv4 address on the interface and establishes the BGP Unnumbered peering using the interface's IPv6 link-local address.

Example below shows the complete BGP configuration on Leraf-101.

router bgp 65101

 router-id 1.1.1.101

 log-neighbor-changes

 timers 60 180

 !

 neighbor interface Ethernet0

  v6only

  remote-as external

  capability extended-nexthop

  !

  address-family ipv4 unicast

   activate

  !

  address-family ipv6 unicast

   activate

Example 6-2: BGP Configuration Script on Leaf-101.

 

Note: Traditionally, the remote AS number is configured explicitly, which provides control over which AS is allowed to establish a BGP session. This is particularly important at network edges, where BGP sessions commonly exist between different administrative organizations. In a datacenter network, AS numbers are used only for BGP loop prevention.

Figure 6-8 shows the BGP configuration on Leaf-101. We use a multi-ASN
design in which the Spine switches share the same ASN, while each Leaf
switch has a unique ASN. The BGP neighbor is configured as an IPv6-only
unnumbered neighbor over Ethernet0.

BGP Configuration in CONFIG_DB

The BGP configuration is stored in the Redis CONFIG_DB. SONiC separates the configuration into three tables according to its scope: global BGP parameters, neighbor-specific parameters, and address-family-specific parameters.

 

BGP_GLOBALS in CONFIG_DB

 

Global BGP parameters that apply to the BGP instance are stored in the BGP_GLOBALS table. On Leaf-101, these parameters are stored in the BGP_GLOBALS|default entry. BGP configuration in SONiC is organized per VRF. The default component in BGP_GLOBALS|default identifies the default VRF.

 

The configuration hierarchy can be represented as follows:

BGP Globals:

 BGP Instance

  Local ASN: 65101

  Router ID: 1.1.1.101

  Hold time: 180 seconds

  Keepalive interval: 60 seconds

  Other global BGP parameters

 

These values identify the BGP instance running on Leaf-101.

 

BGP_NEIGHBOR

Parameters that apply to a specific BGP neighbor are stored in the BGP_NEIGHBOR table. For the BGP Unnumbered peer connected through Ethernet0, the configuration is stored in:

BGP_NEIGHBOR|default|Ethernet0.

The default component identifies the VRF, while Ethernet0 identifies the BGP neighbor interface.

The configuration hierarchy is:

 

BGP Neighbor:

 Ethernet0

  Administrative status: true

  Peer type: external

  Extended next-hop capability: true

  IPv6-only transport: true

The important parameters for BGP Unnumbered are v6only and extended_nexthop capability. The v6only setting allows the BGP session to use the IPv6 link-local address of Ethernet0 without requiring a numbered IPv4 or IPv6 address. The extended next-hop capability allows IPv4 routes to use IPv6 next hops.

BGP_NEIGHBOR_AF

Address-family-specific parameters are stored separately in the BGP_NEIGHBOR_AF table. SONiC creates a separate entry for each address family enabled for a neighbor.

On Leaf-101, Ethernet0 has two address families enabled:

BGP_NEIGHBOR_AF|default|Ethernet0|ipv4_unicast

BGP_NEIGHBOR_AF|default|Ethernet0|ipv6_unicast

 

BGP Neighbor Address Families:

  Ethernet0 / IPv4 Unicast

     Administrative status: true

  Ethernet0 / IPv6 Unicast

    Administrative status: true

 

Activating both address families is required because IPv6 provides the transport for the BGP session, while IPv4 allows IPv4 routes to be exchanged over that IPv6-based BGP session.

The configuration in CONFIG_DB describes what BGP should be configured to do. It does not contain the complete operational state of the BGP session, such as whether the neighbor is currently in the Established state or how many routes have been received. This operational state is maintained by FRR after the configuration has been applied.

Applying BGP Configuration to FRR

The frrcfgd daemon running in the BGP container monitors BGP-related configuration changes in CONFIG_DB. When BGP configuration is written to CONFIG_DB, Redis notifies frrcfgd about the changed entries. Frrcfgd reads the updated configuration and translates it into FRR configuration commands, which are applied to the running FRR routing stack.

FRR (FRRouting) is a routing software suite that consists of several routing daemons. The bgpd daemon is responsible for BGP, while other daemons, such as zebra, provide common routing functions and maintain the system routing table. Therefore, frrcfgd does not directly configure a SONiC database consumer bgpd. Instead, it provides the BGP configuration to the running FRR software, where bgpd processes the BGP-specific configuration.

After the configuration is applied, bgpd creates the BGP instance and initializes the required neighbor and routing structures. It then starts the peer initialization process and attempts to establish the BGP session with the configured neighbor.

Figure 6-8: BGP Configuration Pipeline.

 

BGP Neighbor Initialization Pipeline

 

The following packet capture, taken on Spine-11’s Ethernet0 interface, shows the beginning of the BGP neighbor initialization, including the TCP three-way handshake and the first BGP OPEN message.

TCP three-way handshake

The TCP three-way handshake consists of a SYN (Flags [S]), a SYN-ACK (Flags [S.]), and an ACK (Flags [.]) packet. Immediately after the connection is established, Spine-11 sends the BGP OPEN message in a TCP packet with the PSH and ACK flags (Flags [P.]). The PSH flag instructs the receiving TCP stack to deliver the BGP payload immediately to the bgpd process.

BGP Capabilities

The BGP OPEN message contains several capabilities. The Multiprotocol Extensions capability advertises support for both IPv4 Unicast (AFI 1, SAFI 1) and IPv6 Unicast (AFI 2, SAFI 1). This allows the BGP session, although transported over IPv6, to negotiate the exchange of both IPv4 and IPv6 unicast routes.

The Extended Next Hop Encoding capability is also advertised. This capability is important for our IPv4-over-IPv6 routing design because it allows IPv4 routes to use IPv6 next-hop addresses. The capture shows Capability Code 5, although tcpdump does not decode the capability name and therefore reports no decoder for Capability 5.

The remaining capabilities include Route Refresh, Enhanced Route Refresh, 32-bit AS Numbers, Multipath, Graceful Restart, and Long-Lived Graceful Restart. These capabilities are not specific to BGP Unnumbered and are omitted from the tcpdump output shown here.

 

admin@Spine-11:~$ sudo tcpdump -i Ethernet0 -nn -s 0 -vv 'tcp port 179'

 

tcpdump: listening on Ethernet0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

 

10:41:28.303725 IP6 (class 0xc0, flowlabel 0x421d8, hlim 1, next-header TCP (6) payload length: 40) fe80::e22:34ff:feb6:a.47296 > fe80::ecb:81ff:fe19:a.179: Flags [S], cksum 0x0234 (correct), seq 3251084010, win 63280, options [mss 9040,sackOK,TS val 174604957 ecr 0,nop,wscale 9], length 0

 

10:41:28.308179 IP6 (class 0xc0, flowlabel 0x41bc9, hlim 255, next-header TCP (6) payload length: 40) fe80::ecb:81ff:fe19:a.179 > fe80::e22:34ff:feb6:a.47296: Flags [S.], cksum 0xf542 (correct), seq 3128045848, ack 3251084011, win 63196, options [mss 9040,sackOK,TS val 1700510797 ecr 174604957,nop,wscale 9], length 0

 

10:41:28.308237 IP6 (class 0xc0, flowlabel 0x421d8, hlim 1, next-header TCP (6) payload length: 32) fe80::e22:34ff:feb6:a.47296 > fe80::ecb:81ff:fe19:a.179: Flags [.], cksum 0x380a (correct), seq 1, ack 1, win 124, options [nop,nop,TS val 174604961 ecr 1700510797], length 0

 

10:41:28.309421 IP6 (class 0xc0, flowlabel 0x421d8, hlim 1, next-header TCP (6) payload length: 161) fe80::e22:34ff:feb6:a.47296 > fe80::ecb:81ff:fe19:a.179: Flags [P.], cksum 0x7f3a (correct), seq 1:130, ack 1, win 124, options [nop,nop,TS val 174604962 ecr 1700510797], length 129: BGP

        Open Message (1), length: 129

          Version 4, my AS 65011, Holdtime 180s, ID 1.1.1.11

          Optional parameters, length: 100

            Option Capabilities Advertisement (2), length: 6

              Multiprotocol Extensions (1), length: 4

                AFI IPv4 (1), SAFI Unicast (1)

                0x0000:  0001 0001

            Option Capabilities Advertisement (2), length: 8

              Extended Next Hop Encoding (5), length: 6

                no decoder for Capability 5

                0x0000:  0001 0001 0002

            Option Capabilities Advertisement (2), length: 6

              Multiprotocol Extensions (1), length: 4

                AFI IPv6 (2), SAFI Unicast (1)

                0x0000:  0002 0001

            <snipped>           

Example 6-3: BGP Neighbor Session Initialization.

Verification

 

Figure 6-9 shows the basic BGP peering verification commands. The first command, show bgp ipv4 unicast, shows that instead of a peer IP address, the peer is identified by the connected interface name. The second command, show bgp ipv4 unicast neighbors interface Ethernet 0, verifies that the BGP session is established using IPv6 link-local addresses.

Note that both commands are executed for the IPv4 address family, even though the BGP session itself uses IPv6 as the transport. The same BGP session can therefore carry IPv4 and IPv6 address families. The output is the same for both address families, except for the timestamp, confirming that both address families are active on the BGP session.

Figure 6-9: BGP Peering Verification.

BGP Configuration Pipeline from sonic-cli to bgpd

To better understand how BGP configuration is passed from SONiC to FRRouting, we can examine the Python source code of the frrcfgd daemon. The following command locates the source file used by the running frrcfgd process and searches it for terms related to Redis, CONFIG_DB, VTY, bgpd, FRR, and sockets:

root@Leaf-101:/# grep -nEi 'redis|config_db|vty|bgpd|frr|FRR|socket' \

$(python3 -c "import frrcfgd.frrcfgd; print(frrcfgd.frrcfgd.__file__)")

Example 6-4: Locating the frrcfgd source code and searching for relevant terms..

The Python expression returns the exact path of the frrcfgd source file installed on the switch. Using this path as input to grep allows us to examine how frrcfgd accesses CONFIG_DB and communicates with FRRouting.

 

The first example shows that frrcfgd associates specific CONFIG_DB tables with the FRR daemons that process their configuration. Several BGP-related tables are assigned to bgpd:

87:  'DEVICE_METADATA': ['bgpd'],

88:  'BGP_GLOBALS': ['bgpd'],

89:  'BGP_GLOBALS_AF': ['bgpd'],

96:  'BGP_PEER_GROUP': ['bgpd'],

97:  'BGP_NEIGHBOR': ['bgpd'],

98:  'BGP_PEER_GROUP_AF': ['bgpd'],

99:  'BGP_NEIGHBOR_AF': ['bgpd'],

Example 6-5: CONFIG_DB tables Associated with frrcfgd.

This shows that frrcfgd knows which FRR daemon is responsible for each configuration table. For example, the global BGP configuration in BGP_GLOBALS and the neighbor configuration in BGP_NEIGHBOR are handled by bgpd.

The next example shows that frrcfgd directly accesses these tables through its CONFIG_DB connector:

2719: self.config_db = ExtConfigDBConnector(...)

2724: db_entry = self.config_db.get_entry('DEVICE_METADATA', 'localhost')

2742: glb_table = self.config_db.get_table('BGP_GLOBALS')

2750: global_af_table = self.config_db.get_table('BGP_GLOBALS_AF')

2759: nbr_table = self.config_db.get_table('BGP_NEIGHBOR')

2760: pg_table = self.config_db.get_table('BGP_PEER_GROUP')

Example 6-6: CONFIG_DB Connector.

The configuration is not only read when frrcfgd starts. The daemon also subscribes to configuration-table changes:

2993: table_list = self.config_db.get_table(table)

3009: self.config_db.subscribe(table, hdlr)

Example 6-7: CONFIG_DB Subscriber.

 

This allows frrcfgd to react when the BGP configuration in CONFIG_DB changes. A corresponding handler processes the change and prepares the configuration that must be applied to FRRouting.

The next part of the source code shows how frrcfgd translates the SONiC configuration into FRRouting commands. For example, the code constructs a command that enters BGP configuration mode and activates an address family:

3488: command = "vtysh -c 'configure terminal' \

-c 'router bgp {} vrf {}' \

-c 'address-family ipv4 unicast' -c '{}'"

Example 6-8: Command Translation.

Therefore, frrcfgd acts as the SONiC-to-FRR configuration translator. When a BGP configuration is created or changed through sonic-cli, the configuration is stored in CONFIG_DB. frrcfgd detects the change, reads the relevant CONFIG_DB entry, and generates the corresponding FRRouting configuration commands.

The generated commands are then passed to the FRR VTY interface. The source code shows that frrcfgd uses UNIX domain sockets to communicate with the FRR daemons:

205: sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

209: serv_addr = '/run/frr/%s.vty' % daemon

Example 6-9.1: Unix Domain Socket.

For bgpd, this results in communication through its VTY socket:

/run/frr/bgpd.vty

Example 6-9.2: VTY Socket for bgpd.

The complete configuration pipeline can therefore be summarized as:

Sonic-cli  CONFIG_DB  frrcfgd  FRR VTY interface  bgpd

In the BGP Unnumbered configuration used in this chapter, the same pipeline is used to configure the BGP process, the interface-based neighbor, and the required address families. After bgpd receives the configuration, it establishes and manages the BGP session, including establishing the TCP connection using the IPv6 link-local address associated with the unnumbered interface.