Scalable mail storage with Dovecot and Amazon S3

Storage Space

Object Stores to the Rescue

The situation changed dramatically, however, with the advent of object stores. These solutions were given their name because they handle all data in the same way internally – as binary objects. The trick here is that binary objects can be split up and put back together as desired, as long as this happens in the same order.

This trick lets object stores offer real horizontal scaling, because the object store itself "only" has to make sure that the binary objects are split correctly and distributed neatly across the existing hard drives. If more hard drives are added to the installation, the object store automatically uses them and thus expands scalability limits to theoretical dimensions.

The existing cloud computing solutions have brought a whole wave of different stores into the limelight. Red Hat acquired Ceph [1]-[3] and introduced its own Storage Server [4] as a solution for storing objects. OpenStack entered the game with Swift, which is also an object store in the classical sense. Moreover, you have those who provide object stores as a service for users, such as Amazon S3 or Dropbox.

Setting up scalable storage systems with all the services is certainly possible, and it would be great for email platform administrators if such a storage solution could be harmonized with the email architecture referred to previously. After all, there is no obstacle to treating an email message like a binary object. That is what Timo Sirainen, the author of the Dovecot secure IMAP mail server [5], probably thought and drew his own conclusions: The enterprise version of Dovecot offers an Amazon S3 plugin that perfectly exploits the benefits of the object store.

Dovecot with S3

Sirainen has offered the Dovecot S3 plugin for some time. Importantly, the plugin only runs with the enterprise version of the Dovecot mail server (Figure 1). The license for the enterprise-grade Dovecot Pro edition costs around EUR5,000 per year for 10,000 mailboxes. Admittedly, that is not exactly cheap, but the number has little meaning.

Figure 1: The license for the S3 plugin with Dovecot is not inexpensive; however, the costs can usually be offset by cheap hardware.

Dovecot Pro might cost more than the free version, but using it with an object store for backup will, in many cases, mean significant hardware discounts in the enterprise, because it removes the need for SAN storage, so off-the-shelf hardware will do. Companies should certainly take such considerations into account if they are considering using Dovecot with the S3 plugin.

How, specifically, does the S3 plugin work for Dovecot? Sirainen explains this in detail in the documentation for the plugin. Generally, anyone who wants to use the Dovecot S3 back end needs access to an object store as per the Amazon S3 standard. Login credentials in the form of two values are usually attached to such accounts: The access key acts as a kind of username, and the secret key is the password. Anyone who creates an account with Amazon receives both pieces of information automatically.

To storing email with Dovecot, you also need to create your own bucket in S3. At first, you might be a little uneasy and feel insecure because not all users get their own buckets – but this is an illusion. Not all users have their own filesystem with a regular mail server, after all; the responsibility for enforcing access rights lies with Dovecot as the mail server in both cases.

Dovecot Configuration

The next step involves the configuration of Dovecot itself: Anyone who already uses the program for IMAP or IMAPS will be familiar with the paragraph-like structure of the configuration files. Registering an additional paragraph that takes over the plugin configuration is all it takes for the Amazon S3 plugin. The example in Listing 1 is taken directly from the Dovecot documentation (Figure 2).

Listing 1

A Dovecot Plugin

plugin {
# Use 100 GB cache for mails in /var/lib/dovecot/cache. The cache directory is \
  the same for all users.
obox_fs = fscache 100G:/var/lib/dovecot/cache:s3:https://Accesskey:Secret@\
          Bucket-Name.s3.amazonaws.com/
}
Figure 2: The Dovecot website provides a PDF file that describes how to set up the enterprise repository and the S3 plugin.

In this case, Dovecot uses a local cache with 100GB of storage space to process local access to frequently used objects as quickly as possible. Clearly, it is not very difficult to dock Dovecot onto S3, as long as the required Dovecot license is available for the plugin.

Those who would prefer to store their data in Microsoft's Azure cloud can do just that – a plugin for Azure is available too. Dropbox support is available on top of that; thus, Dropbox can also be used as back-end storage for Dovecot.

This very fact, however, leads to a discussion that is much more legal than technical: Do companies actually want to use Dovecot to store their email on Amazon, Microsoft, or Dropbox? Skepticism is perfectly understandable in light of the Snowden revelations.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Getting Ready for the New Ceph Object Store

    The Ceph object store remains a project in transition: The developers announced a new GUI, a new storage back end, and CephFS stability in the just released Ceph v10.2.x, Jewel.

  • Ceph object store innovations
    The Ceph object store remains a project in transition: The developers announced a new GUI, a new storage back end, and CephFS stability in the just released Ceph c10.2.x, Jewel.
  • Ceph and OpenStack Join Forces

    When building cloud environments, you need more than just a scalable infrastructure; you also need a high-performance storage component. We look at Ceph, a distributed object store and filesystem that pairs well in the cloud with OpenStack.

  • Comparing Ceph and GlusterFS
    Many shared storage solutions are currently vying for users’ favor; however, Ceph and GlusterFS generate the most press. We compare the two competitors and reveal the strengths and weaknesses of each solution.
  • Troubleshooting and maintenance in Ceph
    We look into some everyday questions that administrators with Ceph clusters tend to ask: What do I do if a fire breaks out or I run out of space in the cluster?
comments powered by Disqus