An admin faced with the task of building a new mail server setup will commonly feel slightly disoriented and even discouraged. However, today, a stable and reliable mail server can be operated on any popular Linux distribution – provided you know what you are doing. In this article, I list the basic considerations that play a role in securing mail servers or that relate to the IT environment.

Concerns

People around the world have been sending and receiving digital messages for more than 40 years now. And the success of email drew the interest of crooks and thieves. Suddenly, unwelcome advertising messages, fraud attempts, spoofed sender addresses, and many other dirty tricks began to flourish.

Since then, associations such as the Internet Engineering Task Force (IETF), which defines the standards for the Internet, have found themselves trapped in a kind of vicious circle. Time and time again they see themselves forced to extend the standards for email and add patches to iron out design flaws in the protocol with new technology. Simply abolishing the technology and replacing it with something completely new is not an option. A global changeover of this kind would be almost impossible to organize. Therefore, network users have to live with the disadvantages and various negative effects that come with email.

Some smaller companies outsource the problem to service providers such as Microsoft or Google, who now have huge amounts of experience in dealing with email. Other companies do not want to relinquish control of their data. For better or worse, they have no other option than to operate their own mail servers. Many an admin only realizes in the middle of the process that an ad hoc approach does not work; instead, it requires meticulous planning and perfect technical implementation. This approach applies all the more if the issue of email security is to be taken seriously.

Conditions

A number of factors need to be considered, starting with the mail server itself. The aim is to secure the service so that attackers cannot hijack and exploit it with just a few simple actions. If you don't want to end up on inspect lists or deny lists in the blink of an eye, you need working validation for your own domain. The way email is transported between the client and server and between the mail servers that forward email messages to each other also needs to be secure. Spam plays a role, as well, because cyber criminals love to distribute their malware by email, so you have to make sure you have working spam and malware filters.

As anyone involved in the operation of mail servers will be aware, it is difficult to find another task for which so many roads lead to Rome. It's true that you are unlikely to find Sendmail in the wild these days – thank goodness, as many a long-suffering admin would probably add. However, even without the oldtimer, the mail server market is confusing, even if you just focus on Linux and ignore the countless Microsoft Exchange setups. Therefore, you need to define a few requirements, particularly with regard to the software to be used.

Postfix is now widely used and is cutting edge in terms of both features and security. The service can be operated on any recent enterprise distribution – I use Ubuntu 22.04, but you can apply all the advice to other distributions, too. ClamAV is used in conjunction with SpamAssassin as a tool for filtering mail. Where action relating to name servers is required, I will not be providing specific commands or instructions and would instead advise you to contact the domain name system (DNS) administrators who are usually exclusively responsible for this task.

Beginnings

I start with a freshly installed Ubuntu 22.04 on which to install a mail server. Before you tackle the mail server itself, the first step is to harden the system. The usual rules apply: Restrict access to the system to as few people as possible. If your company already has a central user management system, it is highly recommended that you connect the mail server to it, so you can enable and disable accounts centrally.

A mail server without direct access to the Internet cannot usually be operated without massive changes to the firewall. However, if the server does need to be accessible from outside your environment, you will at least want to restrict external access to ports that are not relevant for mail. In particular, you will want to configure the firewall or the host itself to deny access to port 22 (i.e., the SSH port). Simple Mail Transfer Protocol (SMTP) ports 25 and 587 should be open. If you also intend to operate with Internet Message Access Protocol (IMAP) on the system, you need to allow access to ports 143 and 993. The same basic rules apply to security measures on mail servers as for all other servers. If you use Ubuntu, for example, make sure you configure AppArmor correctly instead of disabling it across the board. The same applies to SELinux on Red Hat Enterprise Linux (RHEL).

Next on the agenda are the security settings of the mail server itself. Operating the mail server without user administration – as with the classic mail protocol – is virtually out of the question. Instead, you will want the ability to send email conditional on successful authentication on the client side. The problem has long since been solved in the Linux world but is still a long way from being used everywhere. More specifically, this requires Simple Authentication and Security Layer (SASL), a bridge construct that acts as an intermediary between the mail server on the one hand and a variety of possible authentication back ends on the other.

As a rule, companies today manage their customer data in some form of directory service – usually by Active Directory or with the Lightweight Directory Access Protocol (LDAP). An SASL plugin for LDAP connects the mail server to the user directory. SASL's LDAP back end can also be linked to Active Directory through its LDAP compatibility interface. The result of all this hard work is that users can only send email if they have an active and valid account in the user directory. Incidentally, an alternative approach to this procedure if you use a pluggable authentication module (PAM) to connect your system to LDAP lets you simply connect SASL to PAM and indirectly inherit the LDAP users.