After
enabling IPv6 on the Ethernet0 interface, we can configure BGP peering. Instead of statically defining a peer IP address and AS number, the
BGP neighbor is defined at the interface level. This tells BGP to expect a neighbor
and initialize the BGP peering over Ethernet0. Because our intent is to transport IPv4 traffic
over the IPv6 network, we activate both the IPv4 and IPv6 address families. We
also configure the peer as an external BGP (eBGP) neighbor by using a command remote-as external. This tells BGP that the peer must use an AS
number different from the local AS number.
Because the network uses IPv6 as the
transport for IPv4 traffic, IPv4 Network Layer Reachability Information (NLRI) exchanged
through BGP use IPv6 next-hop addresses. BGP therefore needs the Extended Next
Hop Encoding capability to support IPv4 routes with IPv6 next hops. In this
configuration, the capability is explicitly enabled with the capability extended-nexthop
command.
In FRRouting, the v6only option controls which IP version is used
for interface-based BGP Unnumbered peering. Without v6only, FRR uses the interface's IPv6 link-local
address for peering only when no suitable IPv4 address is configured on the
interface. With v6only, FRR skips searching for an IPv4 address on the interface and
establishes the BGP Unnumbered peering using the interface's IPv6 link-local
address.
Example below
shows the complete BGP configuration on Leraf-101.
router bgp 65101
router-id 1.1.1.101
log-neighbor-changes
timers 60 180
!
neighbor interface Ethernet0
v6only
remote-as external
capability extended-nexthop
!
address-family ipv4 unicast
activate
!
address-family ipv6 unicast
activate
Example 6-2: BGP Configuration Script on
Leaf-101.
Note: Traditionally, the remote AS
number is configured explicitly, which provides control over which AS is
allowed to establish a BGP session. This is particularly important at network
edges, where BGP sessions commonly exist between different administrative
organizations. In a datacenter network, AS numbers are used only for BGP loop
prevention.
Figure 6-8 shows the BGP
configuration on Leaf-101. We use a multi-ASN
design in which the Spine switches share the same ASN, while each Leaf
switch has a unique ASN. The BGP neighbor is configured as an IPv6-only
unnumbered neighbor over Ethernet0.
BGP Configuration in
CONFIG_DB
The BGP configuration is stored in
the Redis CONFIG_DB. SONiC separates the configuration into three tables
according to its scope: global BGP parameters, neighbor-specific parameters,
and address-family-specific parameters.
BGP_GLOBALS in CONFIG_DB
Global BGP parameters that apply to
the BGP instance are stored in the BGP_GLOBALS table. On Leaf-101, these
parameters are stored in the BGP_GLOBALS|default entry. BGP configuration in SONiC
is organized per VRF. The default component in BGP_GLOBALS|default identifies the default VRF.
The
configuration hierarchy can be represented as follows:
BGP Globals:
BGP Instance
Local ASN: 65101
Router ID: 1.1.1.101
Hold time: 180 seconds
Keepalive interval: 60 seconds
Other global BGP parameters
These values identify the BGP
instance running on Leaf-101.
BGP_NEIGHBOR
Parameters that apply to a specific
BGP neighbor are stored in the BGP_NEIGHBOR table. For the BGP Unnumbered peer
connected through Ethernet0, the configuration is stored in:
BGP_NEIGHBOR|default|Ethernet0.
The default
component identifies the VRF, while Ethernet0 identifies the BGP neighbor
interface.
The
configuration hierarchy is:
BGP Neighbor:
Ethernet0
Administrative
status: true
Peer type:
external
Extended
next-hop capability: true
IPv6-only
transport: true
The important parameters for BGP
Unnumbered are v6only and extended_nexthop capability. The v6only setting allows the BGP session to use the IPv6
link-local address of Ethernet0 without requiring a numbered IPv4 or IPv6
address. The extended next-hop capability allows IPv4 routes to use IPv6 next
hops.
BGP_NEIGHBOR_AF
Address-family-specific parameters
are stored separately in the BGP_NEIGHBOR_AF table. SONiC creates a separate
entry for each address family enabled for a neighbor.
On Leaf-101, Ethernet0 has two
address families enabled:
BGP_NEIGHBOR_AF|default|Ethernet0|ipv4_unicast
BGP_NEIGHBOR_AF|default|Ethernet0|ipv6_unicast
BGP Neighbor Address Families:
Ethernet0 / IPv4 Unicast
Administrative
status: true
Ethernet0 / IPv6 Unicast
Administrative status: true
Activating
both address families is required because IPv6 provides the transport for the
BGP session, while IPv4 allows IPv4 routes to be exchanged over that IPv6-based
BGP session.
The
configuration in CONFIG_DB describes what BGP should be configured to do. It
does not contain the complete operational state of the BGP session, such as
whether the neighbor is currently in the Established state or how many routes
have been received. This operational state is maintained by FRR after the
configuration has been applied.
Applying BGP Configuration
to FRR
The frrcfgd daemon
running in the BGP container monitors BGP-related configuration changes in CONFIG_DB.
When BGP configuration is written to CONFIG_DB, Redis notifies frrcfgd about the changed entries. Frrcfgd reads the updated configuration and
translates it into FRR configuration commands, which are applied to the running
FRR routing stack.
FRR (FRRouting) is a routing software suite that consists of several
routing daemons. The bgpd daemon is responsible for BGP, while other daemons, such as zebra, provide common routing
functions and maintain the system routing table. Therefore, frrcfgd does not directly
configure a SONiC database consumer bgpd. Instead, it provides the BGP configuration to the running FRR
software, where bgpd
processes the BGP-specific configuration.
After the configuration is applied, bgpd creates the BGP instance and initializes
the required neighbor and routing structures. It then starts the peer
initialization process and attempts to establish the BGP session with the
configured neighbor.
Figure 6-8: BGP
Configuration Pipeline.
BGP Neighbor Initialization
Pipeline
The following packet capture, taken
on Spine-11’s Ethernet0 interface, shows the beginning of the BGP neighbor
initialization, including the TCP three-way handshake and the first BGP OPEN
message.
TCP three-way handshake
The TCP three-way handshake consists
of a SYN (Flags [S]), a SYN-ACK (Flags [S.]), and an ACK (Flags [.]) packet.
Immediately after the connection is established, Spine-11 sends the BGP OPEN
message in a TCP packet with the PSH and ACK flags (Flags [P.]). The PSH flag
instructs the receiving TCP stack to deliver the BGP payload immediately to the
bgpd process.
BGP Capabilities
The BGP OPEN message contains
several capabilities. The Multiprotocol Extensions capability advertises
support for both IPv4 Unicast (AFI 1, SAFI 1) and IPv6 Unicast (AFI 2, SAFI 1).
This allows the BGP session, although transported over IPv6, to negotiate the
exchange of both IPv4 and IPv6 unicast routes.
The Extended
Next Hop Encoding capability is also advertised. This capability is important
for our IPv4-over-IPv6 routing design because it allows IPv4 routes to use IPv6
next-hop addresses. The capture shows Capability Code 5, although tcpdump does
not decode the capability name and therefore reports no decoder for Capability
5.
The remaining
capabilities include Route Refresh, Enhanced Route Refresh, 32-bit AS Numbers,
Multipath, Graceful Restart, and Long-Lived Graceful Restart. These
capabilities are not specific to BGP Unnumbered and are omitted from the
tcpdump output shown here.
admin@Spine-11:~$ sudo tcpdump -i Ethernet0 -nn -s 0
-vv 'tcp port 179'
tcpdump: listening on Ethernet0, link-type EN10MB
(Ethernet), snapshot length 262144 bytes
10:41:28.303725 IP6 (class 0xc0, flowlabel 0x421d8,
hlim 1, next-header TCP (6) payload length: 40) fe80::e22:34ff:feb6:a.47296 >
fe80::ecb:81ff:fe19:a.179: Flags [S], cksum 0x0234 (correct), seq
3251084010, win 63280, options [mss 9040,sackOK,TS val 174604957 ecr
0,nop,wscale 9], length 0
10:41:28.308179 IP6 (class 0xc0, flowlabel 0x41bc9,
hlim 255, next-header TCP (6) payload length: 40) fe80::ecb:81ff:fe19:a.179 >
fe80::e22:34ff:feb6:a.47296: Flags [S.], cksum 0xf542 (correct), seq
3128045848, ack 3251084011, win 63196, options [mss 9040,sackOK,TS val
1700510797 ecr 174604957,nop,wscale 9], length 0
10:41:28.308237 IP6 (class 0xc0, flowlabel 0x421d8,
hlim 1, next-header TCP (6) payload length: 32) fe80::e22:34ff:feb6:a.47296 >
fe80::ecb:81ff:fe19:a.179: Flags [.], cksum 0x380a (correct), seq 1, ack
1, win 124, options [nop,nop,TS val 174604961 ecr 1700510797], length 0
10:41:28.309421 IP6 (class 0xc0, flowlabel 0x421d8,
hlim 1, next-header TCP (6) payload length: 161) fe80::e22:34ff:feb6:a.47296 >
fe80::ecb:81ff:fe19:a.179: Flags [P.], cksum 0x7f3a (correct), seq
1:130, ack 1, win 124, options [nop,nop,TS val 174604962 ecr 1700510797],
length 129: BGP
Open
Message (1), length: 129
Version
4, my AS 65011, Holdtime 180s, ID 1.1.1.11
Optional parameters, length: 100
Option Capabilities Advertisement (2), length: 6
Multiprotocol Extensions (1),
length: 4
AFI IPv4 (1), SAFI Unicast (1)
0x0000: 0001 0001
Option Capabilities Advertisement (2), length: 8
Extended Next Hop Encoding (5),
length: 6
no decoder for Capability 5
0x0000: 0001 0001 0002
Option Capabilities Advertisement (2), length: 6
Multiprotocol Extensions (1),
length: 4
AFI IPv6 (2), SAFI Unicast (1)
0x0000: 0002 0001
<snipped>
Example 6-3:
BGP Neighbor Session Initialization.
Verification
Figure 6-9 shows the basic BGP peering
verification commands. The first command, show bgp ipv4
unicast, shows that instead of a peer IP address, the
peer is identified by the connected interface name. The second command, show
bgp ipv4 unicast neighbors interface Ethernet 0,
verifies that the BGP session is established using IPv6 link-local addresses.
Note that both commands are executed for the
IPv4 address family, even though the BGP session itself uses IPv6 as the
transport. The same BGP session can therefore carry IPv4 and IPv6 address
families. The output is the same for both address families, except for the
timestamp, confirming that both address families are active on the BGP session.
Figure 6-9: BGP
Peering Verification.
BGP Configuration Pipeline from sonic-cli to bgpd
To better understand how BGP configuration is passed from SONiC to
FRRouting, we can examine the Python source code of the frrcfgd daemon.
The following command locates the source file used by the running frrcfgd process and searches it for terms related to Redis, CONFIG_DB, VTY,
bgpd, FRR, and sockets:
root@Leaf-101:/#
grep -nEi 'redis|config_db|vty|bgpd|frr|FRR|socket' \
$(python3 -c
"import frrcfgd.frrcfgd; print(frrcfgd.frrcfgd.__file__)")
Example 6-4:
Locating the frrcfgd source code and searching for relevant terms..
The Python expression returns the exact path
of the frrcfgd source file installed on
the switch. Using this path as input to grep allows us to examine how frrcfgd
accesses CONFIG_DB and communicates with FRRouting.
The first example shows that frrcfgd associates specific CONFIG_DB tables with the FRR daemons that
process their configuration. Several BGP-related tables are assigned to bgpd:
87: 'DEVICE_METADATA': ['bgpd'],
88: 'BGP_GLOBALS': ['bgpd'],
89: 'BGP_GLOBALS_AF': ['bgpd'],
96: 'BGP_PEER_GROUP': ['bgpd'],
97: 'BGP_NEIGHBOR': ['bgpd'],
98: 'BGP_PEER_GROUP_AF': ['bgpd'],
99: 'BGP_NEIGHBOR_AF': ['bgpd'],
Example 6-5: CONFIG_DB
tables Associated with frrcfgd.
This shows that frrcfgd knows which FRR daemon is responsible for each configuration table.
For example, the global BGP configuration in BGP_GLOBALS and the neighbor
configuration in BGP_NEIGHBOR are handled by bgpd.
The next example shows that frrcfgd directly accesses these tables through its CONFIG_DB connector:
2719: self.config_db = ExtConfigDBConnector(...)
2724: db_entry =
self.config_db.get_entry('DEVICE_METADATA', 'localhost')
2742: glb_table =
self.config_db.get_table('BGP_GLOBALS')
2750: global_af_table =
self.config_db.get_table('BGP_GLOBALS_AF')
2759: nbr_table =
self.config_db.get_table('BGP_NEIGHBOR')
2760: pg_table =
self.config_db.get_table('BGP_PEER_GROUP')
Example 6-6:
CONFIG_DB Connector.
The configuration is not only read when frrcfgd starts. The daemon also subscribes to configuration-table changes:
2993: table_list
= self.config_db.get_table(table)
3009:
self.config_db.subscribe(table, hdlr)
Example 6-7:
CONFIG_DB Subscriber.
This allows frrcfgd to react when the BGP configuration in CONFIG_DB changes. A
corresponding handler processes the change and prepares the configuration that
must be applied to FRRouting.
The next part of the source code shows how frrcfgd translates the SONiC configuration into FRRouting commands. For
example, the code constructs a command that enters BGP configuration mode and
activates an address family:
3488: command =
"vtysh -c 'configure terminal' \
-c 'router bgp {}
vrf {}' \
-c
'address-family ipv4 unicast' -c '{}'"
Example 6-8:
Command Translation.
Therefore, frrcfgd acts as the SONiC-to-FRR configuration translator. When a BGP
configuration is created or changed through sonic-cli, the configuration is stored in CONFIG_DB. frrcfgd detects the change, reads the relevant CONFIG_DB entry, and
generates the corresponding FRRouting configuration commands.
The generated commands are then passed to the
FRR VTY interface. The source code shows that frrcfgd uses UNIX domain sockets
to communicate with the FRR daemons:
205: sock =
socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
209: serv_addr = '/run/frr/%s.vty'
% daemon
Example 6-9.1: Unix Domain Socket.
For bgpd,
this results in communication through its VTY socket:
/run/frr/bgpd.vty
Example 6-9.2: VTY Socket for bgpd.
The complete configuration pipeline can
therefore be summarized as:
Sonic-cli
In the BGP Unnumbered configuration used in
this chapter, the same pipeline is used to configure the BGP process, the
interface-based neighbor, and the required address families. After bgpd receives the configuration, it establishes and manages the BGP
session, including establishing the TCP connection using the IPv6 link-local
address associated with the unnumbered interface.
