The well-known saying that you should "never switch horses midstream" is especially true in the ever-changing world of IT. The phrase was popularized by Abraham Lincoln when campaigning in 1864, saying, "it is not best to swap horses while crossing the river." The message must have worked, because he won the presidential election for a second term. However, in IT things aren't so elegant – though the sentiment still applies. You don't deal with horses much in support, but you do deal with vendors, managers, and other support personnel. These "horses" shouldn't be switched midstream during a project. Allow me to explain further.

Years ago, I worked for a large telecommunications company, and we used a Unix-based mail program that worked very well. Although it was built on older technology, it was serviceable and stable. A few months after I joined the company, we made a huge shift away from those Unix mail servers and migrated to Novell's GroupWise. The process proceeded smoothly, with a few minor bumps along the way. Seriously, this was a "huge lift," as they say in IT shops, and it went better than expected. Hats off to the consultants, training, and phased planning and migration.

A year or so later, we shifted again to Microsoft's Outlook mail client and then to Microsoft Exchange as our mail server solution. Again, the project went well for such a large endeavor.

However, some projects don't go as well as those I've mentioned. If you've been a part of one, you can sympathize. Let me preface this story with the statement that leaders often make decisions on the basis of emotion or hearsay. Neither bodes well for the support staff subjected to the madness that follows.

I was on a project a few years back that used a commercially supported Linux distribution that had some limitations in some of the third-party packages. The required packages didn't exist in this distro's supported repositories. One option was to compile those packages from source, which is a huge undertaking because of the sheer number of required packages and the high frequency of security and regular updates. These factors made it unwieldy to consider compiling from source.

A second option was to switch to a similar distribution supported by the cloud vendor. This option was never considered by the leaders. The third option was to change to a very different – but also commercially supported – Linux distribution, which was the option chosen against my advice and judgment. The distribution itself was not such a poor choice (although I would likely never have considered it), but the team's lack of experience with the distro was a big issue.

The project moved forward with a lot of angst, grumbling, and irritation – mostly from me, but from some other team members, too. As the project rocketed toward cutover day, the problems grew from a hum to a scream, meaning that we were all scared to make the change. Finally, the night came to migrate the databases and systems. The process went well, with a few minor bumps along the way.

Several days later, disaster struck when the general population of users began to hit the site and issue large queries to the backend database. It was a 36-hour ordeal that won't soon leave my memory, involving some rebuilds, new database replicas, and reversals during the troubleshooting marathon. In the end, it was a lesson learned. There were a few firings – but not enough, in my opinion. Saying, "I told you so," was not in the realm of possibilities when some higher-ups were looking for scapegoats. It wasn't the time to point fingers of guilt at the guilty. It was time to just stay silent and move forward.

The takeaway from these stories is that planning, experience, and good judgment are always what you should rely on when approaching any large-scale changes, rather than emotion, whim, or anger. I know this makes intuitive sense, but if everyone knew it, all my experiences would have been positive, with no story to tell. Returning to my original premise, my best advice is: Don't ride any horse too long, but never, ever change horses midstream, because it could have devastating consequences for you, while the horse and your fellow riders walk away without issue.

Ken Hess * Senior ADMIN Editor