Scalable network infrastructure in Layer 3 with BGP

Growth Spurt

What Next?

The clinch for the admin is how to design the network so that it will still scale horizontally in 10 or more years but without becoming unmaintainable. In your mind's eye, you almost automatically picture not being able to see the wood for the trees. Building giant computer networks is a solved problem; the necessary technology has existed for decades. After all, the Internet is just a huge contiguous network that is divided into a number of physical segments. What could be so obvious as applying the available technology to the local network?

Layer 3 Basics

The differences between Layer 2 networks and Layer 3 networks – in terms of planning and implementation – are huge. First, you need to ditch a central assumption that is essential for Layer 2 networks: that each host is a member of the same network segment. The direct connection between two hosts is handled in the Layer 2 world with the help of the Address Resolution Protocol (ARP). A network design based on the Layer 3, however, no longer relies on all servers residing on the same physical network segment. Routing is a core function of Layer 3.

The Layer 3 setup is based in part on technical work by global corporations such as Google and Facebook, who implemented suitable concepts years ago using the Internet Protocol (IP) as the common fabric in the data center, thus engendering the name "IP fabric." In a direct comparison with Layer 2 networks, the IP fabric principle means that additional components are required at various points of the setup. Why are they needed? A look under the hood helps to understand the details of IP fabrics; the need for additional tools in the setup is then almost automatic.


A Layer 3-based network also uses Layer 2, but with different basic assumptions. A host within the scope of a network can basically talk to different targets: On the one hand, it can reach directly the servers that reside in the same network segment. Two servers on the same switch use ARP to find each other and then exchange the desired communication. If a host server wants to communicate outside its own network segment, it happens exclusively on Layer 3 of the OSI model: IPv4 is the most common example.

The communication source knows its own IP address as well as the address of another server on the same network, which forwards its packets to the target computer (gateway). IP fabrics, which work on the basis of OSI Layer 3, take advantage of exactly this property: The basic assumption is simply that every other host is only reachable via a gateway. Traffic for other servers in such a setup is thus always via a gateway.

For this principle to work, each server needs to know how to reach the other servers in the setup, which is where BGP comes into play. Using the BGP protocol, each host that is part of the IP fabric announces the routes over which it can be reached. The switches also play an important role: They form the BGP counterpart to the individual servers and distribute the routes they have learned across the entire setup. Any host in the setup can thus communicate with any other host via OSI Layer 3 using the switches as gateways. Packets that are directed to servers outside of the setup, such as Internet traffic, are routed via external gateways on routes learned from BGP.

Three components are thus necessary for an IP fabric: (1) an external gateway to communicate with the outside world, (2) a smart switch, and (3) a similarly smart node that uses BGP to announce its routing information. Because BGP is a standardized protocol, such a setup can be created in many ways with various components. The following example is just one implementation option – concrete hardware-based examples will follow later.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • OS10 and Dell's open networking offensive
    Dell's OS10 is a Linux-based operating system for network hardware that is designed to free network admins from the stranglehold of established manufacturers. We look at what it is, how the system works, and what it can do for you.
  • Software-defined networking with Windows Server 2016
    Windows Server 2016 takes a big step toward software-defined networking, with the Network Controller server role handling the centralized management, monitoring, and configuration of network devices and virtual networks. This service can also be controlled with PowerShell and is particularly interesting for Hyper-V infrastructures.
  • Useful tools for automating network devices
    Armed with the right tools, you can manage your network infrastructure both automatically and effectively in a DevOps environment.
  • Spanning Tree Protocol
    Ethernet is so popular because it simply works and is inexpensive. However, the administration side looks a bit more complicated: For the network to run smoothly, the admin might need to make important decisions about the Spanning Tree protocol.
  • Network overlay with VXLAN
    VXLAN addresses the need for overlay networks within virtualized data centers accommodating multiple tenants.
comments powered by Disqus
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs

Support Our Work

ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.

Learn More”>


		<div class=