I had an interesting discussion with my colleagues at work a few days ago that led to the question, "How do you document failure?" The discussion led us to the conclusion that failures aren't widely documented. They should be. Think about it. When you search for a solution to a problem, you want an answer that worked for someone else. You probably don't care about what didn't work or the trial and error process of solving the problem – you only want to know about the successful solution. Whoever said, "Those who don't know history are destined to repeat it," was a genius. That person understood, intuitively perhaps, that documenting your failures is a wise choice.

Applying this to system administration, or any technology job, is easy. Quite possibly, you also tend to document only what works; so when you search your own knowledge bases, you find successful outcomes. However, when you're faced with a "new" problem, how do you go about resolving it? If you're like everyone else since humans first walked upright, you fail until you succeed. How do you know what not to try next time? You don't know because most of us don't document our failures.

I've had many experiences of consulting a senior-level coworker about a particular problem only to hear, "We tried that, and it didn't work." And I didn't just hear it once. That sentence should have been on a loop that someone played for me every time I walked into their cubicle. How do I know you really tried it? How do I know you're not guessing that it won't work? I'm from a time and place that makes me require proof rather than just take someone's word for something, so because of who I am, I will try all the same things that failed before eventually coming up with a common solution. Why? Because no one documented their failures.

If I could see a list of things that failed, I would be convinced that they didn't work, especially if some explanation was also present.

For example, back in the old days of the 1990s, when sys admins always had to recompile the Linux kernel to remove bloat and to enable features that weren't available in the default kernel, some things just didn't work. Certain video drivers, Ethernet drivers, and other hardware peripherals didn't work even with a recompile. Also, not that many hardware devices were Linux compatible. Unless you were a programmer, you couldn't get them to work, either. It would have been nice to see documentation around all the things that didn't work. It would be sort of like a hardware compatibility list in reverse.

A hardware incompatibility list or a software incompatibility list could have helped a lot. I had to deal with some quirky hardware, and it seems like each model or submodel was so different from every other one that it was almost impossible to ever create a "golden" image to make once and install everywhere. I didn't have the luxury of adhering to the hardware compatibility list. I had to work with what I was given.

The Linux kernel has evolved and so have devices and their drivers, but many other issues could do with good failure documentation. (Insert your own list here.) The solution is to document your failures in your knowledge base or wiki. Someone eventually will need that information and will thank you for taking the time to document it.

You've heard that you shouldn't try to reinvent the wheel, and it's true, you shouldn't. Instead, you should try to improve the wheel, but you'll need something to go on first. I don't want to reinvent the wheel. I just want a wheel that works, but when it doesn't, I need some help. Perhaps I can summarize it with my version of an old rhyme we all learned as children: If at first you don't succeed, document your failures so that I don't have to try, try again.

Ken Hess * ADMIN Senior Editor