HA and Backup

To give customers high availability and failure safety, a proxy is deployed upstream of each search cluster. It forwards the traffic to the appropriate server, regardless of whether changes to orders need to be made. At the same time, the proxy reads the information that the ES instances send to ZooKeeper. Based on this knowledge, the proxy decides whether it can forward traffic to other instances, or whether it needs to completely block requests so that a stressed cluster's state of health does not deteriorate.

In the backup section, ZooKeeper looks after the leader election. For backups, admins use the backup and restore API by Elasticsearch that does not save instances, just indexes and cluster settings. Found therefore only preserves the content of clusters, not the servers and instances.

Because the API does not support periodic snapshots, Found implemented its own scheduler. To make it as reliable as the ES cluster for which it is responsible, a scheduler runs on each server with ES instances. This approach removes the single point of failure for Found's HA customers. The schedulers are not supposed to run on independent servers because this introduces additional sources of error. Because the backups occur on a per cluster basis and not per instance, the backup scheduler must be coordinated. One scheduler alone triggers snapshots by choosing a leader for each cluster via ZooKeeper.

Found also needs a connection that is as latency free as possible, because many systems rely on ZooKeeper. Thus, one ZooKeeper cluster is used per region. Having the client and server in the same region increases the reliability of the network. However, admins need to consider errors that occur during maintenance work on the cluster.

Found's experience also shows that it is extremely important to consider in advance what information a client should keep in its local cache, and which actions it can carry out if the connection to ZooKeeper breaks down.

Size Matters

Although Found uses ZooKeeper very intensively, the company also makes sure not to exceed certain limits. Just because A and B use ZooKeeper, the admins do not exclusively send messages via ZooKeeper. Instead the value and urgency of the information must be high enough to justify the cost of a transmission, which consists of the size and update frequency. Admins must consider the following:

• Application logs: As annoying as missing logs might be when debugging, they are still usually the first thing an admin sacrifices when a system reaches its limits. Admins should choose a solution with fewer consistency requirements.
• Binary files are simply too big. They would necessitate optimizing ZooKeeper to a point at which exception problems would probably arise that nobody has ever tested. That's why Found stores binaries in Amazon S3 and only manages the URLs with ZooKeeper.
• Metrics: This may work at the beginning, but in the longer term, scaling problems will result. Sending metrics via ZooKeeper simply would be too expensive because the admin would have to provide a buffer for the intended and available capacity. This applies to metrics in general – with the exception of two critical metrics for application logic: current disk usage and memory usage of each node. Proxies use the latter to set the indexes if a customer exceeds disk capacity. The former will be important when it comes to updating customer contracts.

First Aid

ZooKeeper has become a fairly large open source project with numerous developers who implement advanced features with a strong focus on quality. Of course, some effort is required to become familiar with ZooKeeper, but that should not deter admins. The effort is worthwhile, especially for managers of distributed systems.

A few manuals can help you get started: The "Getting Started Guide" [9] shows how to set up a single ZooKeeper server, how to connect with it via a shell, and how to implement a few basic operations. The more detailed "Programmer's Guide" [10] brings together several important tips that admins should note before installing ZooKeeper. The "Administrator's Guide" [11] describes the options that are relevant to a production cluster.

Some admins doubt that using one system for deployments and another for upgrades actually provides any benefits. If you do this, however, make sure that each of these systems is as small and independent as possible. For Found, ZooKeeper is an important step toward achieving this goal.