Avoiding chaos in clusters with fencing

Fenced In

Article from ADMIN 09/2012

By Martin Loschwitz

Sometimes you have to be cruel to be kind: To avoid letting broken nodes in a Pacemaker cluster cause damage, you really need to let cluster nodes kill each other. A couple of details are very important.

Administrators frequently come across the word Fencing in the context of high-availability clusters, but here, I use it to describe a method that allows one node in a cluster to throw another node out of the cluster – using brute force if necessary. This sounds archaic, but it is very much part of real life in a production environment. Cluster nodes blast each other out of the cluster to avoid simultaneous write access to sensitive data leading to inconsistency.

Is the Cluster Doing What it Should?

Fencing mechanisms are important in the context of clusters whenever you need to ensure data integrity on the one hand and service availability on the other. Fencing always serves to guarantee the functional state of the entire cluster by removing individual resources or nodes. The cluster management software always assumes that it is responsible for interpreting all of the actions that occur in the cluster. In the case of Pacemaker, it assumes it has issued all the commands running in the cluster. To ensure that this happens, communication with cluster nodes must work without a hitch. Additionally, you must ensure that the services configured in the cluster really work as their maker intended.

Fencing has to intervene in the cluster if one of these two points is not fulfilled. How should the cluster manager handle nodes that suddenly disappear from the cluster? In the worst case, Pacemaker itself might crash on the missing node, with resources still running that are beyond Pacemaker's control. The Pacemaker instance on the node that is still running can only assert its rights by fencing the other machine. In practical terms, this means rebooting the machine.

The same thing applies to resources that cannot be stopped gracefully. For example, if Pacemaker tells MySQL to terminate on a node, it expects MySQL to do precisely that. If the stop action fails, Pacemaker

...

Use Express-Checkout link below to read the full article (PDF).