Config Management with Bash

07/12/2011 01:33 pm

The guys at Netflix, who built the ChaosMonkey app I talked about last time, say that, in the move to cloud, you have to be prepared to unlearn everything you know about hosting.

Cloud hosting differs in one important way. You can trace many cloud architectures, applications, and best practices back to this difference. Every time you plan an architecture, it comes back to the same thing.

Cloud instances are transient.

One of the tools to work around instances disappearing is configuration management. Anyone who has run more than a handful of servers will have used config management, whether home-spun or through a tool like Chef or Puppet.

So configuration management isn't really a cloud-specific topic, but it does make things a hell of a lot easier for you.

Configuration management manages the O/S and software stack running on top of each instance. Large data-centers have used this kind of thing for years to update hundreds of servers at once because it saves time. For anyone working with just a few servers, it doesn't make sense to spend time and money on something that you can do in a few minutes manually.

But cloud instances are transient. They will probably die, and when they do, who's going to configure them? Not you, if you have any sense. The config management kicks in, installs all of the required packages, and uses your preferred configuration.

You can use tools such as Chef and Puppet, but to start with, let's do the poor man's configuration using a bash script.

The following script - ok pseudo script - installs Apache, grabs your code from S3, and starts the server up.

apt-get -y install apache2
cd /tmp
wget -O your-code.tgz &nbsp;<a href=""></a>
mkdir tmp-extract && cd tmp-extract
tar xzf your-project.tgz
mv * /var/www/vhosts/
service apache2 restart

Once this is done, the instance is up and running with your app.

To inject this into an EC2 instance when it boots up, thereby creating an cloud instance that actually does something, you can make use Ubuntu's CloudInit script.

If you're on AWS, you can punch in the user data on boot up (Figure 1).

CloudInit makes the instance do stuff when it boots up, which is a really good way to get around bundling everything into images all the time.

To use this, just shove the script from above in the “runcmd:" section - e.g.

- apt-get -y install apache2
- cd /tmp
- wget -O your-code.tgz
- mkdir tmp-extract && cd tmp-extract
- tar xzf your-project.tgz
- mv * /var/www/vhosts/
- service apache2 restart

Sticking with the poor man's config management system, you can still make things a little nicer by pushing the script to S3 and then downloading it from there:

- wget -O
- chmod +x && ./

This way you can version the script in git, try it out on a vanilla instance (and isn't cloud all about trying stuff out cheaply). When you're happy, push it up to S3 and all your new instances will start up with the new version.

Soon I'll cover chef and puppet for config management, but these tools rely on your ability to get the app running 100% from a script. There can be no human help in the cloud.

Do you have a home-spun config system like this? Many do. Or are you using a different, off-the-shelf system? I'd like to hear how you've solved this problem.

Until next time...

Related content

  • A Real-World Look at Scaling to the Amazon Cloud
    The Amazon cloud environment adapts easily to custom solutions. We'll show you how one company built their solution in the cloud.
  • Securing Your Data In The Cloud: an insiders perspective

    As the increasing use of cloud computing and other technologies is changing the world of data management, keeping your data private and secure is an ongoing concern for everyone. Memset, a cloud computing Infrastructure as a Service (IaaS) provider gives an insider’s perspective on what you should be doing to keep your data safe.

  • Exploring Ubuntu cloud tools
    Cloud computing, promising manageable, quickly deployed, virtual machines in large networks, is so appealing in a world where everyone is trying to cut costs. Ubuntu Enterprise Cloud Services (UEC) is Canonical's entry into the cloud market.
  • Delivering Cloud Services: Five Essentials for Success

    You can meet the challenges of delivering cloud services successfully and profitably, if you have the right identity management capabilities. This paper outlines the five infrastructure essentials for successfully delivering cloud services to a rapidly growing customer base. You'll learn how these essentials can benefit both you, the cloud services provider, and your customer, the cloud services consumer – to your ultimate economic benefit. Plus, we'll outline why traditional identity management infrastructures fall short, and how a new approach is the key to building the essentials into your business.

  • Getting started with the OpenStack cloud computing framework
    OpenStack brings common virtualization technologies such as KVM, Xen, Hyper-V, and QEMU into the cloud.

Dan Frost is Technical Director of, cloud hosting consultants and web developers based in London and Brighton, UK

Dan has been building cloud hosting, writing, and talking about the cloud since before it was trendy. Since he spun up his first AWS instance, he's been trying out new services and finding ways of getting more out of hardware without actually owning any of it.

Mon Tue Wed Thu Fri Sat Sun
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31