Lead Image © Spectral, Fotolia.com

Lead Image © Spectral, Fotolia.com

Creating a redundant array of inexpensive links

RAID for the Network

Article from ADMIN 27/2015
The Fault Tolerant Router daemon uses multipath routing among multiple Internet connections to keep you connected, even when some connections go down.

Stable Internet connections (uplinks) are mission critical in many enterprises. Unfortunately, they often break down. If you want to connect two uplinks redundantly using two or more providers, you will typically experiment with the Border Gateway Protocol (BGP). This solution can be a fairly expensive, though, because providers charge dearly for enterprise-level connections. With a few restrictions, you can achieve redundancy far less expensively by opting for Linux and the Fault Tolerant (FT) Router [1].

A Linux host typically sends its packets with the help of a routing table. All packets that do not belong to a specific route follow the default route, which usually leads to the Internet. If this link fails, all the users on the inside are cut off from the Internet. The reasons for failure can be many, including bulldozers digging up cables, Layer 2 or 3 software failures, or routers that fail one hop downstream on the provider's network.

To avoid hard disk failures, administrators have relied on RAID for a long time; in the simplest case, this means simply doubling the number of disks in a mirroring RAID [2]. This isn't quite as easy for access lines. In the classic setup for this scenario, at least two Internet providers safeguard the network; that is, your own connection has two uplinks.

The administrator needs to inform the rest of the world using BGP (on internal networks, this can also be an internal routing protocol such as OSPF, or Open Shortest Path First). If one link fails, the protocols notice this and stop sending packets over the dead link.

The protocols detect failures automatically. If the Internet Protocol (IP) fails even though the link is working perfectly at the lowest level, the routing protocol notices this through active monitoring. Although it can take a while, at least the changeover happens without intervention (i.e., without forcing the administrator out of bed in the middle of the night).

Although the "I" in RAID stands for "Inexpensive," a redundant network connection is not something you can have for $19.99 a month. Instead, you are probably looking at a three-figure amount. However, administrators with less cash at their disposal would probably relish the prospect of turning their Redundant Array of Links (RAL) into a RAIL.

Fortunately, Linux comes with almost all the components that you need to implement such a RAIL. However, the network administrator does need to enable it kernel-side. Linux fails to detect link failures automatically; instead, it would continue sending data packets into a black hole at the end of the failed link, and the user would receive non-reproducible load errors. Before you integrate FT Router into your setup, it is a good idea to familiarize yourself with the underlying software.

Wild Routing with iproute2

The ip command in the iproute2 package can configure everything the Linux kernel offers in terms of network technology. Many functions can only be configured with this tool (e.g., routes in additional routing tables); in other cases, it replaces standard tools such as ifconfig, route, or vconfig, which creates VLANs.

For example, ip can create multiple-path routes. To do so, you need to assign one of your new routes the table <X> parameter, where <X> can be a number or name that points to a specific routing table in the /etc/iproute2/rt_tables file. The command

ip route add table 5 via

assigns the route for network to routing table number 5, for example.

Policy Routing

If multiple routing tables exist, the administrator needs to tell the kernel when to use which routing table with the ip rule command, which is used to create rules for the source and target IP addresses – network blocks are also permitted. You can define which network interface to use and a value for the type of service (TOS) field in the IP protocol.

To keep the rules as flexible as possible, you can also use the fwmark argument. This hexadecimal value lets you track the firewall marker set by iptables; iptables references this along with the mangle attribute to route by ports or protocols (Listing 1). For more detailed information, search the web for "Policy Routing" [3].

Listing 1

Iptables Rules for Policy Routing

01 iptables -t mangle -A PREROUTING -i <source_interface> \
  -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
02 iptables -t mangle -A PREROUTING -i <source_interface> \
  -j CONNMARK -p tcp --dport 80 -m conntrack --ctstate NEW --set-mark 0x10

To send all the packets from subnet to routing table 8 with the default route, you would enter the following two commands:

ip route add table 8 default via
ip rule add from lookup 8

Administrators need to be aware of one small thing when using multiple routing tables: You do not have the routing entries the kernel creates automatically on creating a new IP address. You need to add these manually if you want to use them (e.g., in Policy Routing). Otherwise, Linux sends all response packets via the wrong route.

If you only want to forward outgoing HTTP traffic over a different route, you can create the two iptables rules from Listing 1. The first will restore the markers on the outgoing packets, and the second assigns packets that have a target of port 80 a hexadecimal marker (--set-mark 0x10). You will want the routing rule to watch out for this in future:

ip rule add fwmark 0x10 lookup 8

Using this approach, you can redirect packets, but the kernel does not distribute them across different routes.

Multipath Routing

Using the ip route command, administrators can set up multiple default routes – even with different weighting. This assumes that the kernel was built with multipath routing support up front.

Linux typically balances data packets at the packet or connection level. Distributing the packets individually will achieve maximum utilization of both links, but it does entail a couple of risks in practical applications. For example, a counterpart at the other end of the link organizes the packets again in a similar way; otherwise, only the upstream is load balanced.

If the links use different speeds, it's possible that the sender transmits the packets in the right order, but they arrive in the wrong order. Because TCP connections only compensate for differences up to the buffer size, the speed settles to that of the slowest link. Additionally, address translation is impossible on this route.

Alternatively, admins can set up each new connection on a different link. Although the individual streams are, at most, as fast as the quickest single link, this still leads to improved data throughput for users.

To set up to default routes with load balancing, you would do this:

ip route add default nexthop via dev eth1 \
  nexthop via dev eth2

If you want to work at the packet level, you need to add the equalize keyword after the add command. To achieve weighting, type weight <number> behind the individual hops. If a route now fails, every second connection will hit a timeout.

At this point, you have all the building blocks in place to send data across multiple links, monitor with a couple of simple commands, and remove one link from the cluster in case of failure. So, it's time for FT Router to enter the game.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Routing with Quagga

    Cisco and Juniper have implemented routing protocols to help your router find the optimum path. On Linux, you can use software like Quagga, with its Zebra daemon, to help automate this process.

  • Open source multipoint VPN with VyOS
    The VyOS Linux distribution puts network routing, firewall, and VPN functionality together and presents a fully working dynamic multipoint VPN router as an alternative or addition to a Cisco DMVPN mesh.
  • Border Gateway Protocol
    We look at the Border Gateway Protocol, how it routes packets through the Internet, its weaknesses, and some hardening strategies.
  • Tested: Barracuda firewall X201
    With a number of new firewalls, Barracuda seeks to expand its portfolio to include small and medium-sized companies. We take a closer look at the Barracuda firewall X201.
  • Flexible software routing with open source FRR
    The FRR open routing stack can be integrated into many networks because it supports a large number of routing protocols, though its strong dependence on the underlying kernel means it requires some manual configuration.
comments powered by Disqus
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs

Support Our Work

ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.

Learn More”>


		<div class=