Lead Image © Lucy Baldwin, 123RF.com

Lead Image © Lucy Baldwin, 123RF.com

Comparing startup times of Linux distros in the cloud

Cloud Startup

Article from ADMIN 65/2021
A cloud speed test pits Linux distributions against one another.

Launching instances on Amazon Elastic Compute Cloud (EC2) can be a lengthy affair for those used to instant gratification, taking entire minutes to complete. Historically, Windows instances have brought up the rear by taking considerably longer to initialize than Linux-based instances because of sysprep and the reboot that follows. The open source world is somewhat more streamlined, but there is still considerable variance between Linux distributions, with one to two minutes being a reasonable expectation of first availability.

One Ping Only, Please

Trying to connect multiple times as an instance boots is inelegant. Fortunately, you can use shell-fu to script your way out of this. The BSD version of ping [1], notably on macOS, includes a convenient "one ping only" option (-o) that I would like to think honors Sean Connery's famous quote in Hunt for Red October . The option terminates ping once the first reply is received. Like Connery's character, Captain Marko Ramius, you can use this option to ask for "one ping only, please" (Listing 1).

Listing 1

One Ping Only

$ ping -o; sleep 2; ssh ubuntu@
PING ( 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
64 bytes from icmp_seq=3 ttl=48
time=40.492 ms
[ output truncated ]
Welcome to Ubuntu 20.04.1 LTS
(GNU/Linux 5.4.0-56-aws x86_64)

Whether you launch the instance with the Amazon Web Services (AWS) Console [2] or the AWS command-line interface (CLI) [3], you can easily find the moment it becomes accessible; however, that is only part of the picture. An instance is only useful as the service it exists to provide can be reached by users. Completing service bootstrap is distinct from the point in time when the kernel network stack becomes active. The SSH daemon is present in nearly all official distribution images, and it provides a convenient testing standard for service availability.

Perhaps less steeped in movie lore, but nonetheless equally effective, is this GNU-compatible one-liner, waiting in a loop for the SSH service to start up (Listing 2).

Listing 2

Waiting for SSH

$ until ssh ubuntu@; do sleep 1; done
ssh: connect to host port 22: Connection refused
ssh: connect to host port 22: Connection refused
[ output truncated ]
Welcome to Ubuntu 20.04.1 LTS
(GNU/Linux 5.4.0-56-aws x86_64)

Booting Benchmarks

These two tests provide good insight into the initialization speed of stock operating system images, with the notable exception of Microsoft Windows, where SSH is absent (see the "Hibernate Support" box). FreeBSD Security Officer Emeritus and author of Tarsnap [4] Colin Percival has done all the hard work and released a new tool to time these events. The ec2-boot-bench tool [5] takes four measurements: (1) the duration of the RunInstances API call, (2) the time to transition from a pending to a running state, (3) how much longer the network stack took to initialize (RST TCP response), and (4) actual service availability (SYN+ACK TCP response). The ec2-boot-bench tool uses the SSH service's availability for the fourth metric, just as the shell script in Listing 2, but you can easily change the port with a recompile and use Colin's code to benchmark the specific service of your own Amazon Machine Image (AMI).

Hibernate Support

Newly started instances may take longer to be ready for reasons extending beyond boot time. On the first run, many distributions run one-time instance setup code (e.g., AWS Linux and FreeBSD will install package updates if any are available). Moreover, services can have a much longer startup time than SSH. Fortunately, EC2 now supports Hibernate [8]. The ability to save the RAM state of a running system was developed for mobile computing, but it is increasingly becoming useful for quickly bringing cloud instances online with significant state. Hibernate works by dumping to Elastic Block Storage (EBS) the whole content of RAM and restoring it as needed, which is a much faster process than recreating all that state – and much cheaper to operate than keeping the instance running.

The first two metrics do not vary with distribution, and Colin himself has published [6] data indicating an average of 1.5 and 6.9 seconds, respectively; however, these numbers may vary with AWS region, availability zone, or instance type and are part of the delay experienced by the user. My own measurements in the us-east-1 region are consistent with Colin's findings and average around 1.7 and 8.5 seconds, respectively (Figure 1). I used T3.micro instances for my tests instead of C5.xlarge, accounting for some of the variance between our results.

Figure 1: A few runs with ec2-boot-bench. A shell loop is recommended.

The tool itself is is still a rough instrument, with no distribution packages, man pages, or GitHub docs – and no way to check API errors. (I will be submitting a couple of patches for these before this article goes to press.) However, it does its job remarkably well already. Instances are benchmark-terminated, but if you Ctrl+C the tool, you might have stragglers left running: Remember to clean up after tests or your AWS bill will suffer.

Ubuntu's Lot

Ubuntu on 64-bit instances appears to have made consistent progress over the years, cutting startup time by nearly a third, from 14.6 to 9.9 seconds (32%), as shown in Table 1 for 10 sample averages after a warm-up run. The latest Ubuntu Server LTS release, Focal Fossa (20.04), can be expected to be ready for use in 15 seconds once the EC2 virtual machine has been initialized, compared with 18 seconds with Trusty Tahr (14.04). The more than doubling of the SSH startup time between Trusty and Xenial may show how the transition to systemd [7] affected Ubuntu's overall optimization, or it may just be an artifact of startup ordering because the resulting total is not significantly affected.

Table 1

Ubuntu Startup Improvement

Distribution AMI Tested OS Boot (s) Service Start (s) Total (s)
Trusty (14.04) ami-05dc324761386f3a9 14.581 3.170 17.75
Xenial (16.04) ami-0133407e358cc1af0 9.300 7.001 16.301
Bionic (18.04) ami-0186d369d234b536f 12.946 5.608 18.554
Focal (20.04) ami-000b3a073fc20e415 9.934 4.938 14.872

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus