Tuesday, 23 June 2026

Chapter 2: Installing SONiC NOS

 

ONIE-Based SONiC Installation

Many switch vendors have added SONiC NOS support to at least part of their switch portfolio. Depending on the vendor and switch model, customers may be able to order a switch with a vendor-customized SONiC version that is supported at the same level as the vendor's own network operating system. Some vendors also allow customers to run the community-based SONiC distribution.

The support model for Community SONiC depends on the vendor. Some hardware vendors provide full support, while others provide no support at all. Compared with vendor-specific SONiC distributions, Community SONiC provides greater flexibility because it can be customized, rebuilt, and adapted to customer requirements. However, running a Community SONiC deployment without vendor support or in-house expertise is generally not a recommended operating model.

Community SONiC is typically installed by using ONIE (Open Network Install Environment) [1], a small open-source installation environment that provides a standardized method for installing network operating systems on supported switches. Figure 2-1 illustrates a conceptual ONIE-based Community SONiC installation process.

If the switch is delivered with a vendor-specific SONiC distribution already installed, it may boot directly into that operating system without requiring a separate ONIE installation workflow. For Community SONiC deployments, the SONiC installation image is downloaded from a trusted source such as a vendor repository or the Community SONiC project. Depending on the installation environment, the image may be stored on a USB flash drive, an HTTP server, or a TFTP server. In this example, the image is stored on an HTTP server.

During the DHCP process, the switch receives network parameters and may also receive information that assists installer discovery, such as an installer URL or TFTP-related parameters. In the HTTP-based path shown in Figure 2-1, ONIE downloads the installer image from the HTTP server and launches it. The installer then performs the SONiC installation, after which the switch reboots into the newly installed SONiC operating system.


Figure 2-1: Conceptual ONIE Installation Workflow – ZTP.

Next, let us examine the automatic SONiC installation process in more detail. Example 2-1 shows a simple dnsmasq-based DHCP configuration that is used by the ONIE installation workflow. The service is bound to the interface facing the target switches, and the DHCP pool assigns addresses from 192.168.1.50 to 192.168.1.150 with a 12-hour lease time. The same configuration also provides the default gateway and DNS server address for the management subnet.

For this example, the most important entry is DHCP option 114. It provides ONIE with an explicit installer URL that points to the SONiC ONIE installer image on the HTTP server. The example uses the platform-derived filename onie-installer-x86_64-kvm_x86_64-r0, which matches the platform identifier shown later in the ONIE discovery output. The same filename is also used in DHCP option 67 as the TFTP bootfile name. The configuration also includes DHCP option 66, which identifies the TFTP server. In this example, both the HTTP and TFTP services are located on the same server.

When DHCP option 114 is present, ONIE typically attempts to use the provided installer URL directly. If a valid installer image is not available through that URL, ONIE can continue with its normal discovery mechanisms, including TFTP-based discovery.

  GNU nano 5.4                    /etc/dnsmasq.conf *                                    
# Bind explicitly to the interface facing your target switches (e.g., eth0 for Management)
interface=eth0
bind-interfaces
# DHCP IP Range: Pool starts at .50 and ends at .150 with a 12-hour lease
dhcp-range=192.168.1.50,192.168.1.150,255.255.255.0,12h
# Gateway and DNS Options
dhcp-option=option:router,192.168.1.1
dhcp-option=option:dns-server,1.1.1.1
# ONIE Provisioning URL (Matches Option 114 "default-url" requested by ONIE)
dhcp-option=114,"http://192.168.1.10/onie-installer-x86_64-kvm_x86_64-r0"
# Alternative TFTP discovery information for ONIE (Options 66 and 67)
dhcp-option=option:tftp-server,192.168.1.10
dhcp-option=option:bootfile-name,"onie-installer-x86_64-kvm_x86_64-r0"
# Logging for deep troubleshooting
log-dhcp
^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^_ Go To Line

Example 2-1: DHCP Server Example Configuration.

When the SONiC switch is powered on, it starts the platform bootloader. In this example, the bootloader is GRUB (Grand Unified Bootloader). The GRUB menu contains the installed network operating system entries and the ONIE entry. Selecting the *ONIE menu entry does not start the SONiC installer directly. It opens the ONIE menu, where the operator selects the ONIE boot mode used for installation, rescue, uninstall, update, or embed operations.

                             GNU GRUB  version 2.02                           
+----------------------------------------------------------------------------+
|VendorX-default-nos-v1.0                                                    |
|VendorX-hardened-SONiC-v1.0                                                 |
| *ONIE                                                                      |
|                                                                            |
| <empty lines snipped>                                                      |
|                                                                            |
|                                                                            |
+----------------------------------------------------------------------------+
      Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected OS, `e' to edit the commands
      before booting or `c' for a command-line.
   The highlighted entry will be executed automatically in 2s.

Example 2-2: ONIE Installer Phase 1 – Select “*ONIE”

After the ONIE entry has been selected, the ONIE menu is displayed. The automatic SONiC installation process is started by selecting the “*ONIE: Install OS” option, as shown in Example 2-3. This mode starts ONIE with the discovery service enabled, allowing ONIE to search for a network operating system installer automatically. Before starting the process, it is good practice to verify that the device has enough storage space for the SONiC installation. If the existing network operating system must be removed, this can be done by selecting the ONIE: Uninstall OS option.


                           GNU GRUB  version 2.02                              
 +----------------------------------------------------------------------------+
 |*ONIE: Install OS                                                           |
 | ONIE: Rescue                                                               |
 | ONIE: Uninstall OS                                                         |
 | ONIE: Update ONIE                                                          |
 | ONIE: Embed ONIE                                                           |
 |                                                                            |
 | <empty lines snipped>                                                      |
 |                                                                            |
 |                                                                            |
 +----------------------------------------------------------------------------+
      Use the ^ and v keys to select which entry is highlighted.
      <Rest of the text removed for brevity>

Example 2-3: ONIE Installer Phase 2 – Select “ONIE: Install OS”.

Example 2-4 shows the ONIE-side view of this process after the process starts. The purpose of Example 2-4 is not to show a complete successful installation, but to show what ONIE does after installation mode has been selected. ONIE initializes the platform, discovers available network interfaces, attempts DHCP on interfaces with link, and then begins service discovery.

In this output, ONIE receives an IPv4 address on eth0, but it does not retrieve a valid installer image. Instead, it continues into TFTP-based service discovery and tries several candidate installer filenames. After the discovery attempt fails, ONIE waits for 20 seconds and starts the discovery cycle again. This behavior is important to understand because ONIE discovery is a continuous loop that repeats until a valid installer is found and executed.

Note: Some messages shown in the example originate from the virtual lab platform used to generate the output and are not part of a normal hardware installation.

# Step 1: ONIE enters OS install mode and prints platform information.
ONIE: OS Install Mode ...
Platform  : x86_64-kvm_x86_64-r0
Version   : master-201811170418
Build Date: 2018-11-17T04:18+00:00
# Step 2: ONIE mounts kernel filesystems and initializes the platform environment.
Info: Mounting kernel filesystems... done.
ERROR: Getting ONIE boot device timeout
Info: BIOS mode: legacy
Running demonstration platform init pre_arch routines...
Running demonstration platform init post_arch routines...
grub-editenv: error: cannot open `/mnt/onie-boot/grub/grubenv.new': No such file or directory.
Info: Making NOS install boot mode persistent.
Installing for i386-pc platform.
grub-install: error: failed to get canonical path of `rootfs'.
ERROR: grub-install failed on:
network_driver: Running demonstration pre_init routines...
network_driver: Running ASIC/SDK init routines...
network_driver: Running demonstration post_init routines...
# Step 3: ONIE initializes network interfaces and learns their MAC addresses.
Info: Using eth0 MAC address: 0c:9a:32:79:00:00
Info: Using eth1 MAC address: 0c:9a:32:79:00:01
Info: Using eth2 MAC address: 0c:9a:32:79:00:02
Info: Using eth3 MAC address: 0c:9a:32:79:00:03
Info: Using eth4 MAC address: 0c:9a:32:79:00:04
Info: Using eth5 MAC address: 0c:9a:32:79:00:05
Info: Using eth6 MAC address: 0c:9a:32:79:00:06
Info: Using eth7 MAC address: 0c:9a:32:79:00:07
Info: Using eth8 MAC address: 0c:9a:32:79:00:08
Info: Using eth9 MAC address: 0c:9a:32:79:00:09
# Step 4: eth0 has link, so ONIE starts DHCPv4 and receives an IPv4 address.
Info: eth0:  Checking link... up.
Info: Trying DHCPv4 on interface: eth0
ONIE: Using DHCPv4 addr: eth0: 192.168.1.148 / 255.255.255.0
# Step 5: ONIE checks the remaining interfaces. Interfaces without DHCP either use link-local addressing or are skipped if the link is down.
Info: eth1:  Checking link... up.
Info: Trying DHCPv4 on interface: eth1
Warning: Unable to configure interface using DHCPv4: eth1
ONIE: Using link-local IPv4 addr: eth1: 169.254.186.223/16
Info: eth2:  Checking link... up.
Info: Trying DHCPv4 on interface: eth2
<Additional interface checks omitted for brevity.>
# Step 6: ONIE starts basic services and starts its discovery logic for locating an installer image.
Starting: klogd... done.
Starting: dropbear ssh daemon... done.
Starting: telnetd... done.
discover: installer mode detected.  Running installer.
Starting: discover... done.
Please press Enter to activate this console. Info: eth0:  Checking link... up.
Info: Trying DHCPv4 on interface: eth0
ONIE: Using DHCPv4 addr: eth0: 192.168.1.148 / 255.255.255.0
Info: eth1:  Checking link... up.
Info: Trying DHCPv4 on interface: eth1
Warning: Unable to configure interface using DHCPv4: eth1
ONIE: Using link-local IPv4 addr: eth1: 169.254.152.102/16
Info: eth2:  Checking link... up.
<Info related to eth2-8 is removed for brevity>
Info: Trying DHCPv4 on interface: eth9
Warning: Unable to configure interface using DHCPv4: eth9
ONIE: Using link-local IPv4 addr: eth9: 169.254.28.73/16
# Step 7: ONIE starts service discovery and tries TFTP-based candidate installer paths. The platform-specific names are derived from the platform identifier printed earlier, such as x86_64-kvm_x86_64-r0. In this lab example, no installer image is available on either the HTTP or TFTP server, so ONIE eventually repeats the discovery cycle.
ONIE: Starting ONIE Service Discovery
Info: Attempting tftp://onie-server/0c-9a-32-79-00-00/onie-installer-x86_64-kvm_x86_64-r0 ...
Info: Attempting tftp://onie-server/onie-installer-x86_64-kvm_x86_64-r0 ...
Info: Attempting tftp://onie-server/onie-installer-x86_64-kvm_x86_64-r0.bin ...
Info: Attempting tftp://onie-server/onie-installer-x86_64-kvm_x86_64 ...
Info: Attempting tftp://onie-server/onie-installer-x86_64-kvm_x86_64.bin ...
Info: Attempting tftp://onie-server/onie-installer-kvm_x86_64 ...
Info: Attempting tftp://onie-server/onie-installer-kvm_x86_64.bin ...
Info: Attempting tftp://onie-server/onie-installer-x86_64-qemu ...
Info: Attempting tftp://onie-server/onie-installer-x86_64-qemu.bin ...
Info: Attempting tftp://onie-server/onie-installer-x86_64 ...
Info: Attempting tftp://onie-server/onie-installer-x86_64.bin ...
Info: Attempting tftp://onie-server/onie-installer ...
Info: Attempting tftp://onie-server/onie-installer.bin ...
Info: Sleeping for 20 seconds
4..3..2..1..
# Step 8: No installer was found, so ONIE starts a new discovery cycle.
Info: eth0:  Checking link... up.

Example 2-4: ONIE Installer Phase 3 – Discovery Loop.

Platform information is important because ONIE uses it during automatic discovery to build installer filenames that match the target hardware. In this example, the platform identifier x86_64-kvm_x86_64-r0 appears in candidate names such as onie-installer-x86_64-kvm_x86_64-r0. ONIE follows a waterfall-style search. It first tries the most specific names, including MAC- and platform-based paths, and then falls back to more generic names such as onie-installer and onie-installer.bin. This helps ONIE find a suitable platform-specific installer while still allowing generic fallback options.

Example 2-5 shows the corresponding DHCP server-side events from dnsmasq. When reading the output, focus on four things: the ONIE vendor and user class information, the DHCP Discover/Offer/Request/Acknowledgement exchange, the address assigned to the ONIE client, and the TFTP-related bootfile and server information returned by the DHCP server. The detailed relationship between this output and the ONIE-side output in Example 2-4 is discussed after the example.

# Step A: dnsmasq is started in foreground mode with the configuration from Example 2-1.
root@sonic:~# sudo dnsmasq -d -C /etc/dnsmasq.conf
dnsmasq: started, version 2.85 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC loop-detect inotify dumpfile
dnsmasq-dhcp: DHCP, IP range 192.168.1.50 -- 192.168.1.150, lease time 12h
dnsmasq-dhcp: DHCP, sockets bound exclusively to interface eth0
dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver 192.168.1.10#53
dnsmasq: read /etc/hosts - 2 addresses
# Step B: ONIE sends DHCP messages from eth0 and identifies itself with ONIE vendor and user class information.
dnsmasq-dhcp: 1021583124 available DHCP range: 192.168.1.50 -- 192.168.1.150
dnsmasq-dhcp: 1021583124 vendor class: onie_vendor:x86_64-kvm_x86_64-r0
dnsmasq-dhcp: 1021583124 user class: onie_dhcp_user_class
# Step C: DHCP Discover and Offer correspond to ONIE trying DHCPv4 on eth0 in Example 2-4.
dnsmasq-dhcp: 1021583124 DHCPDISCOVER(eth0) 0c:9a:32:79:00:00
dnsmasq-dhcp: 1021583124 tags: eth0
dnsmasq-dhcp: 1021583124 DHCPOFFER(eth0) 192.168.1.148 0c:9a:32:79:00:00
dnsmasq-dhcp: 1021583124 requested options: 1:netmask, 3:router, 6:dns-server, 7:log-server,
dnsmasq-dhcp: 1021583124 requested options: 12:hostname, 15:domain-name, 28:broadcast,
dnsmasq-dhcp: 1021583124 requested options: 42:ntp-server, 119:domain-search
dnsmasq-dhcp: 1021583124 bootfile name: onie-installer-x86_64-kvm_x86_64-r0
dnsmasq-dhcp: 1021583124 server name: 192.168.1.10
dnsmasq-dhcp: 1021583124 next server: 192.168.1.10
dnsmasq-dhcp: 1021583124 sent size:  1 option: 53 message-type  2
dnsmasq-dhcp: 1021583124 sent size:  4 option: 54 server-identifier  192.168.1.10
dnsmasq-dhcp: 1021583124 sent size:  4 option: 51 lease-time  12h
dnsmasq-dhcp: 1021583124 sent size:  4 option: 58 T1  6h
dnsmasq-dhcp: 1021583124 sent size:  4 option: 59 T2  10h30m
dnsmasq-dhcp: 1021583124 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp: 1021583124 sent size:  4 option: 28 broadcast  192.168.1.255
dnsmasq-dhcp: 1021583124 sent size:  4 option:  6 dns-server  1.1.1.1
dnsmasq-dhcp: 1021583124 sent size:  4 option:  3 router  192.168.1.1
dnsmasq-dhcp: 1021583124 available DHCP range: 192.168.1.50 -- 192.168.1.150
dnsmasq-dhcp: 1021583124 vendor class: onie_vendor:x86_64-kvm_x86_64-r0
dnsmasq-dhcp: 1021583124 user class: onie_dhcp_user_class
# Step D: DHCP Request and ACK complete the address assignment for 192.168.1.148.
dnsmasq-dhcp: 1021583124 DHCPREQUEST(eth0) 192.168.1.148 0c:9a:32:79:00:00
dnsmasq-dhcp: 1021583124 tags: eth0
dnsmasq-dhcp: 1021583124 DHCPACK(eth0) 192.168.1.148 0c:9a:32:79:00:00
dnsmasq-dhcp: 1021583124 requested options: 1:netmask, 3:router, 6:dns-server, 7:log-server,
dnsmasq-dhcp: 1021583124 requested options: 12:hostname, 15:domain-name, 28:broadcast,
dnsmasq-dhcp: 1021583124 requested options: 42:ntp-server, 119:domain-search
dnsmasq-dhcp: 1021583124 bootfile name: onie-installer-x86_64-kvm_x86_64-r0
dnsmasq-dhcp: 1021583124 server name: 192.168.1.10
dnsmasq-dhcp: 1021583124 next server: 192.168.1.10
dnsmasq-dhcp: 1021583124 sent size:  1 option: 53 message-type  5
dnsmasq-dhcp: 1021583124 sent size:  4 option: 54 server-identifier  192.168.1.10
dnsmasq-dhcp: 1021583124 sent size:  4 option: 51 lease-time  12h
dnsmasq-dhcp: 1021583124 sent size:  4 option: 58 T1  6h
dnsmasq-dhcp: 1021583124 sent size:  4 option: 59 T2  10h30m
dnsmasq-dhcp: 1021583124 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp: 1021583124 sent size:  4 option: 28 broadcast  192.168.1.255
dnsmasq-dhcp: 1021583124 sent size:  4 option:  6 dns-server  1.1.1.1
dnsmasq-dhcp: 1021583124 sent size:  4 option:  3 router  192.168.1.1

Example 2-5: DHCP Events.

Example 2-4 and Example 2-5 show the same installation attempt from two different viewpoints. Example 2-4 is the ONIE console output from the switch, while Example 2-5 is the DHCP server output from dnsmasq. When ONIE prints the platform value x86_64-kvm_x86_64-r0, the same value appears in the DHCP log as the ONIE vendor class. This allows the DHCP server to recognize the requesting device as an ONIE client and, in more advanced deployments, to return platform-specific installer information.

The eth0 MAC address also matches between the two examples. In Example 2-4, ONIE reports eth0 with the MAC address 0c:9a:32:79:00:00. In Example 2-5, the DHCP server receives DHCPDISCOVER, DHCPOFFER, DHCPREQUEST, and DHCPACK messages for the same MAC address. The address offered and acknowledged by the DHCP server is 192.168.1.148, which is the same address that ONIE later shows as assigned to eth0.

 

Manual Installation

In a manual installation workflow, the SONiC ONIE installer image is first downloaded from a trusted source and placed on storage reachable by the switch, such as an HTTP server or local media. The switch is then booted into ONIE: Rescue mode. Rescue mode provides an interactive ONIE shell and does not rely on the automatic discovery loop used by ONIE Install Mode. From the ONIE shell, the operator manually configures network connectivity and starts the installation by running the following commands.

The first command stops any active ONIE discovery processes. The management interface is then configured manually, a default route is added, and the SONiC installer is downloaded and executed by using the onie-nos-install command.

ONIE:/# onie-stop
ONIE:/# ifconfig eth0 192.168.1.148 netmask 255.255.255.0
ONIE:/# ip route add default via 192.168.1.1
ONIE:/# onie-nos-install http://192.168.1.10/onie-installer-x86_64-kvm_x86_64-r0 

Example 2-6: Manual ONIE Installation Commands.

 


No comments:

Post a Comment