Lead Image © Vlad Kochelaevskiy, Fotolia.com

Lead Image © Vlad Kochelaevskiy, Fotolia.com

Scalable mail storage with Dovecot and Amazon S3

Storage Space

Article from ADMIN 26/2015
By
Admins can solve typical mail server scalability problems with Dovecot’s Amazon S3 plugin and the Ceph distributed object store.

Mail server administrators have a hard life. On one hand, few other services produce as many complaints about improper use as mail operations – incorrect blacklisting is an example. On the other hand, people are beginning to realize that email is a product from another age that is now struggling to deal with today's fast-paced IT. Users' demands for receiving and sending email have grown steadily in recent years.

Several years ago, it was quite common to allow email attachments only up to a particular, very low size, but this practice is frowned upon today: Users now expect to be able to send their 20MB PowerPoint presentations by email. Moreover, the number of email users is still growing steadily, and the amount of data stored by all of a provider's users is growing proportionally.

This problem might not be specific to email, because other Internet services are also experiencing the effect of the steady growth of the data volumes that need to be handled. However, working around this problem for email setups is relatively complex, primarily because of the architecture inherent in the "email" principle.

Architecture

Normal setups usually follow the same old pattern: On one side, SMTP daemons ensure that customers can send email and that incoming messages reach the recipients. On the other side, the services that deliver email to customers carry out the majority of the work. Some providers still exclusively rely on POP3 or only offer IMAP against payment; however, on the whole, the IMAP protocol has asserted itself.

This situation means additional worries for email service providers: Customers quite rightly expect to find their email on the provider's servers at any time. These customers rarely even download local copies of their email and simply rely on their providers. The providers then have to figure out how to cope with the burden and memory requirements of such a setup. Even if each user only has a small mailbox, the need for storage is already substantial if you have a large numbers of users.

Sooner or later, you will need to adjust your infrastructure to accommodate the increased number of users, which means growing your infrastructure. From experience, load issues with services can be dealt with relatively simply by using multiple mail exchanger (MX) entries (on the SMTP side) and load balancers for IMAP or POP3 daemons. The storage problem is trickier.

The Storage Problem

When email administrators wanted to add storage space to their servers in the past, they mainly used central storage systems. SAN models from the major manufacturers are the classic candidates. However, those things are expensive and scale vertically at best. Remember that vertical scalability means repowering the existing hardware, which is the opposite of horizontal scalability, for which new machines are simply added to existing machines. The load is then distributed across the devices that are part of the installation.

Sharding

Traditional storage systems can be expanded by installing new hard drives, but the end of the road is always reached eventually, and additional storage space can only be managed by purchasing multiple storage systems. The process of storing data records across multiple machines (known as sharding) has been the standard approach for many years, with users being assigned a variety of storage systems.

This approach brings a raft of disadvantages: First, SAN storage is significantly more expensive than regular, off-the-shelf hardware; second, multiple storage islands make centralized administration more difficult. Admins previously just had to grin and bear this because of the lack of alternatives; the storage market seems to have been in a deep sleep when it came to horizontal scaling.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Getting Ready for the New Ceph Object Store

    The Ceph object store remains a project in transition: The developers announced a new GUI, a new storage back end, and CephFS stability in the just released Ceph v10.2.x, Jewel.

  • Ceph object store innovations
    The Ceph object store remains a project in transition: The developers announced a new GUI, a new storage back end, and CephFS stability in the just released Ceph c10.2.x, Jewel.
  • Ceph and OpenStack Join Forces

    When building cloud environments, you need more than just a scalable infrastructure; you also need a high-performance storage component. We look at Ceph, a distributed object store and filesystem that pairs well in the cloud with OpenStack.

  • Comparing Ceph and GlusterFS
    Many shared storage solutions are currently vying for users’ favor; however, Ceph and GlusterFS generate the most press. We compare the two competitors and reveal the strengths and weaknesses of each solution.
  • Troubleshooting and maintenance in Ceph
    We look into some everyday questions that administrators with Ceph clusters tend to ask: What do I do if a fire breaks out or I run out of space in the cluster?
comments powered by Disqus