#13: Document Everything

The importance of good, solid documentation simply cannot be overstated. Documentation provides audit trails to work that has been completed. It provides guides for future system administrators so that they can take over systems that existed before they arrived. It can provide the system administrator and his management with accomplishment records. (These can be very handy at personnel review time.) Good documentation can also help with problem solving.

1. The first audience is the author himself. Documentation aimed at the software author makes it much easier for the author to go back and debug problems in his own code that may occur years later. When you look back at your own two- or three-year-old work (system or network design or programs, for example), it can be almost impossible to remember why you did something a particular way, and not another way, despite having excellent reasons at the time. Write comments in your code. Write manuals, even if they are just a couple of pages long.

2. The second audience is sometime in the future. The people who maintain the systems and the applications today aren’t going to be around forever. And you can’t assume that any experienced personnel will be around to mentor newcomers. You must prepare for a massive exodus of your experienced people. A popular sign on many system administrators’ desks reads: “Remember, 100 years from now, all new people!”

3. The third audience is management. Keeping good notes and documenting your work helps demonstrate to management that you have been diligent and shows what measures have been taken to keep the systems running and productive. If a system does not meet the requirements set forth in the documentation, it should be easy to figure out why and to determine what must be done to make the system compliant. Even if your management does not understand the value of the services you provide to the organization, they will usually understand the value when it is presented as a thick stack of documentation.

Make sure that documentation is stored on paper too. If the system is down when you need the manuals, you’re not going to be able to get to them. Keep them in binders so that they can be easily changed and updated. After the documentation is written, don’t forget to review it and update it on a regular basis. Bad documentation is worse than none at all. In a crisis, especially if the most knowledgeable people are not around, documentation is likely to be followed verbatim. If it’s wrong, things can go from bad to worse. A common question about preparing documentation is at what technical level it should be written. One school of thought says that documentation should be written so that anyone in the organization, from a janitor to the CEO, can follow it and, if required, bring the systems up after a disaster. After all, the system administrative staff may not be available at the time, and someone has to do it. While that is a noble goal, it is an impractical one. The state of the critical systems changes all the time. The more detail that is included in that documentation, the quicker it becomes out-of-date. In order to write documentation so that an untrained smart person could bring systems up or do other critical work on them in time of crisis, it would have to be so detailed as to be unmanageable. If the task at hand is to edit a file to change one variable to another, for an appropriately trained system administrator, the documentation would only need to say, “Change ABC to XYZ in /directory/file.txt.” For an untrained smart person, the documentation would need to say:

1. Change directory to /directory by typing “cd /directory”.

2. Open the vi text editor by typing “vi file.txt”.

3. Move the cursor down to line 300 by typing “300G”.

and so on (in whatever operating environment is appropriate). Every time the line number in the file changed, the documentation would have to be changed. Many error conditions would need to be handled for the inexperienced person as well; for example, if the terminal type or line count is set wrong, the vi editor may have trouble dealing with it. In that event, what should the inexperienced person do? The right answer, therefore, is to write documentation so that an experienced system administrator could use it. Don’t expect the administrator to have a great deal of experience in your shop, just general experience. In a Windows environment, target a Microsoft Certified Engineer (MSCE). In Unix environments, target experienced system administrators. If something happens to the current system administrative staff, you can reasonably assume that they will be replaced by experienced people. Documentation is a lot like a fine piece of literature. Everybody wants to say that they have read Moby Dick, but nobody wants to actually sit down and read Moby Dick.

#12: Employ Service Level Agreements

Before disaster strikes, many organizations put written agreements in place with their user community to define the levels of service that will be provided. Some agreements include penalties for failing to meet, and rewards for greatly exceeding, agreed-to service levels. Service-level agreements might deal with the following areas: Availability levels. What percentage of the time are the systems actually up? Or, how many outages are acceptable during a given service period? And how long can each outage be? Hours of service. During what hours are the systems actually critical? On what days of the week? What about major holidays? What about lesser holidays? What about weekend holidays? Locations. Are there multiple locations? Can all locations expect the same levels of service? What about locations that do not have on-site staff? Priorities. What if more than one system is down at the same time? Which group of users gets priority? Escalation policy. What if agreements cannot be met? Who gets called? After how much time has elapsed? What are the ramifications? Limitations. Some components of service delivery will be outside local control. Do those count against availability guarantees anyway? These agreements are usually the result of considerable negotiations between (and often significant pain to) both customer and service provider. When designing SLAs, beware of ones that commit you to deliver nines of availability. You are betting the success of your job (if not the job itself) on whether or not the system will go down. Nines are a sensible approach only if you know exactly how many times the system will go down over a given period of time. Unfortunately, though, managers, especially non-technical managers, really like to use the nines as an approach to SLAs because they are simple, straightforward, and easily measurable. They are also easy to explain to other non-technical people. If your performance is being judged on your ability to fulfill SLAs, avoid nines-based SLAs. A much more sensible way to design an SLA is to discuss specific types of failures and outages, and how long the system will be down as a result of them. When estimating downtimes, be conservative; everything takes longer than you think it will.

#11: Plan Ahead

Planning is a vital component to any significant project. When you are dealing with critical systems that have dozens or hundreds of users, their requirements must be taken into account any time a system is changed. Crisis situations, such as disasters, require significant planning so that everyone who is responsible for the recovery knows exactly where to go and what to do. In addition, no matter how good your automated protection tools are, there will occasionally be events that are outside of their ability to cope. Multiple simultaneous failures will stretch most software and organizations to their limits. By having plans in place that are the result of calm and rational thinking well in advance of the calamity, a smooth recovery can be facilitated. Without a good plan, you’ll find yourself prioritizing and coordinating in real time and leaving yourself open to myriad variations on the theme of “fix it now!” Any kind of documented recovery plans should be approved by management and key personnel and may be part of a service level agreement. Keep these plans offline, in binders, and in multiple locations so that the acting, senior person on-site can execute them when required. Make sure they are kept current. If they contain confidential or personal information, be very careful with the distribution of the plans, and limit access only to those who need it. Planning also includes coordination. If you are planning to bring down some critical systems for scheduled maintenance, be sure you coordinate that downtime with the users of the systems. They may have deadlines of their own, and your scheduled downtime could interfere with that downtime. In most shops, a wise approach is to give the same level of scrutiny to scheduled downtime as to changes; a committee should bless scheduled downtime periods.