ONIE and Cumulus Linux on a switch
When you think of switches for datacenters, you probably think of products by Juniper, Cisco, HP, or Arista. Although high-end switches use their own operating systems for configuration, the hardware takes care of directing packets. The ASICs (Application-Specific Integrated Circuits) that handle the switching tasks often do not differ to any great extent; manufacturers create unique value through the configurations they can map. The control software determines what these maps look like – and in some cases, the control software is Linux.
Linux runs on practically any CPU architecture, and it has even found its way onto control units in switches. Ordinary Linux, however, does not provide the drivers and tools necessary to help administrators manage the switch hardware. The Cumulus Linux  project is an effort to provide a robust and versatile version of Linux tailored to run on network switches. The Cumulus distribution runs on devices by Dell, Edgecore, and a couple other manufacturers. Cumulus is not free but is, instead, provided on a subscription basis (see the box "Licensed to Switch").
Licensed to Switch
Cumulus Networks does not provide its distribution free of charge but offers a license in the form of a subscription. Prices start at $700. See the Cumulus Networks website  for a complete price list; prices are driven by the switch performance and the duration of use. The price includes the right to use the software as well as updates and support. Although the software does not automatically disable, the license does not allow administrators to use it in production environments after the expiration date.
For the operating system to work on the switch, it must be compatible with the Open Network Install Environment (ONIE) . ONIE is an open bootloader environment – a mini Linux/BusyBox system that runs on the bare hardware and supports the installation of a network OS through remote provisioning. According to the project website, ONIE "… allows end-users and channel partners to install the target network OS as part of data center provisioning, in the fashion that servers are provisioned."
The ONIE team refers to it as the "Linux Kernel with BusyBox." ONIE discovers the actual operating system on the network or on a plugged-in USB stick, triggers the installation, and provides repair tools if something goes wrong during the installation or upgrade.
The boot sequence for a bare metal ONIE-capable device is as follows:
- After powering on, the minimal bootloader on the device launches (typically a version of U-Boot).
- The bootloader initiates the Linux kernel of the ONIE installer.
- ONIE searches for the operating system and installs it.
The ONIE installer  looks for the operating system in several locations. If ONIE finds a USB storage device and a file with the right name in its root directory (the naming rules are disclosed in the ONIE documentation ), the installation starts. If the admin connects a PC with a web server directly using a network cable, ONIE uses neighbor discovery to check the server's IP address for files that comply with the naming rules. The image simply needs to reside in the web server's document root.
The typical case on a network is to use DHCP (Figure 1). You need to set up DHCP option 114 (
default-url) for the host with a direct link to the firmware. From then on, everything else works without any intervention. This approach means you can take the switch out the box, install it, connect the management interface, and power on.
For more complex DHCP setups, ONIE adds a vendor ID to the DHCP requests so that the administrator can distinguish between the DHCP responses. If something goes wrong during the installation, ONIE offers a command line in the style of BusyBox. Administrators can use it to check out what has happened or restart the installation.
Cumulus Linux is a Debian-based distribution with a couple of extras that controls the switch hardware. Former staff at VMware and Cisco founded the company behind Cumulus back in 2010 with the aim of developing a "Network Linux." Figure 2 shows how the typical Linux components, the Cumulus extensions, and the switch hardware collaborate.
The switch we used in our lab, a 4600-54T by Edgecore, has 52 ports and a management interface.
You can view these ports using the
ifconfig -a command; not as
eth, but as
switchd daemon shown in Figure 2 intercepts all the standard Linux calls and converts them for the switch hardware. The switch hardware, in turn, is configured like a Linux computer with 53 interfaces.
Where the administrator of a Cisco switch would say
no shutdown, you need to say
ip link up. You can enable switching between the switch ports with
brctl, the standard bridge configuration tool.
For the configuration to survive a reboot, it needs to reside in
/etc/network/interfaces. To switch ports 1 to 5 together, you need to add the lines from Listing 1 to the
Simple Switch Configuration
01 auto br0 02 iface br0 03 bridge-ports swp1 swp2 swp3 swp4 swp5 04 bridge-stp on
Because the list would be pretty long for 52 ports, you can also do the following specify a series of ports:
bridge-ports glob swp1-10
Readers who have worked with bridge and switch systems will see that Cumulus extends the syntax and options for standard Linux bridges. Of course, it is also possible to assign VLANs to ports and to bundle ports to create redundant uplinks.
In addition to the
brctl bridge configuration command, Cumulus supports Open V-Switch virtual switching technology. This support for V-Switch gives you a hardware-accelerated option for configuring overlay networks with Virtual Extensible LAN (VXLAN). If you want the device to operate in layer 3, you can assign the bridge or SWP interfaces IP addresses. The Quagga Suite handles dynamic routing protocols.
Tcpdump is one of the standard tools for troubleshooting, and one that is typically missing on switches. In most switch environments, you first need to configure a mirror port and connect a Tcpdump-capable device to it. In contrast, Cumulus lets you view traffic via
tcpdump -n -i br0.
Large Pcap files still need a mirror report for analysis because the flash memory on the switch is just not big enough and too slow. But, if you have a full Linux system like Cumulus running on your switch, you can try out other tricks. For example, Collectd  will run on the switch and support monitoring with a matching server.
The ONIE Installer makes it easy to support unattended provisioning for switches. Clients for Puppet and Chef are available in the repository, and Ansible is also supported. If you have ready-to-use Cookbooks or Playbooks that modify
/etc/network/interfaces, you can use automation software to configure the switch.
Buy this article as PDF