The Costs of Downtime

The only way to convince the people who control the purse strings that there is value in protecting uptime is to approach the problem from a dollars-andcents perspective. In this section, we provide some ammunition that should help make the case to even the most stubborn manager.

Direct Costs of Downtime

The most obvious cost of downtime is probably not the most expensive one: lost user productivity. The actual cost of that downtime is dependent upon what work your users perform on the affected systems. If your users are developers, then perhaps the cost seems to be nothing more than the time and carrying cost for the idled developers. Of course, for a large development organization, those costs can be quite significant. A developer may be paid $800 to $1,500 a day, though that figure may vary significantly, depending upon innumerable factors. It is quite reasonable to assume that idling a group of 50 developers for a week could cost $400,000 or more. But even the $400,000 is nothing more than the direct cost of the developers’ idle time. Not taken into account is the overtime required to make up for the lost time to ensure that delivery deadlines do not slip. If your developers are consultants, or other hourly employees, then at time-and-a-half, the overtime costs could exceed an additional $600,000. These figures do not take into account factors such as fatigue from working all that overtime or the impact on morale. On the other hand, rather than pay the overtime, you might elect to slip your project deadlines a week to make up for the outage. (Does that ever actually happen?) The costs of slipping deadlines are not as obvious or as easily stated as employee carrying costs, but they are just as real. For production users, costs will obviously vary depending on the line of work performed on the affected servers. For equities traders in large trading firms on Wall Street, the number often quoted for downtime during the trading day is $2 million per trader per 20-minute outage.

One assumption in that $2 million number is that all trades made on the trading floor are profitable. Consider that the outage could have prevented a trader from making a moneylosing deal, thus saving the firm money. Wall Street firms are oddly loath to discuss this possibility. Of course, the traders’ salaries and carrying costs are lost, and while many traders do make a lot of money for both themselves and their firms, very few of them can consistently bring in $2 million dollars every 20 minutes. A trading firm’s loss is not as easily quantifiable as the loss to the development organization, because the trading firm’s losses are also composed of opportunity losses.3 Other industries’ direct costs of downtime, as reported by various services, are listed in Table 3.1. Over time, of course, these values will change; the actual values are less important than the orders of magnitude of the values, and how much the costs can vary by industry. You don’t need to precisely quantify a loss to know that a significant loss has occurred.

Indirect Costs of Downtime

As expensive and unpleasant as the direct costs of downtime are, the indirect costs can be significantly higher and have much greater long-term impact on the enterprise. Imagine that you wish to order an $80 sweater from one of those catalog clothing stores via telephone (or the Internet). When you call, the operator politely tells you that their systems are down, so could you please call back in a couple of hours? But you want your sweater now.4 So you call a different catalog company and order your sweater there instead. The direct loss to Company 1 is clear; it is the price of the sweater that you didn’t buy from them. But there are other, more indirect costs. You took your business to its competitor; this helps the competitor, so not only did Company 1 lose $80, but Company 2 made an extra $80. Assuming that Company 2 serviced your order satisfactorily, which company will you call next time you want a sweater? Most likely, Company 2. More losses to Company 1. And when your friends ask you where you got that nice new sweater, and you tell them Company 2, the losses to Company 1 increase. They increase further if you tell the whole story to your friends, convincing them to take their business to Company 2. Multiply that by the number of customers who may call during the system outage, and you can see that the impact is significant, though perhaps impossible to quantify precisely. The following are some examples of indirect costs: Customer satisfaction. As in the preceding example, if a customer visits a web site intending to purchase something, or to take part in the activities that the web site delivers, and he cannot, he will surely be frustrated.

Depending on his past experience, he may decide to go away and do other things, returning to the site later. If he is a first-time visitor to the site, he may never return. If he has experienced downtime on other occasions, he may or may not ever return. If he is a happy customer, he will likely return later. But there are few better ways to turn happy customers into unhappy ones than repeating bad experiences. Remember the rule of thumb: It costs 10 times as much to bring in a new customer as it does to retain an old one. Bad publicity and press. Although the news media’s feeding frenzy has diminished from 1999 and 2000, when large companies’ web sites experience significant downtime, it still makes the news, though generally not the front page. Keeping your systems running for years at a time is not newsworthy (perhaps it should be), but one outage can make it onto the evening news and injure a company’s reputation. When you see availability stories on the evening news, beware: The networks tend to skip over the technical details in favor of sensationalism and attempts to scare their viewers. As far as we know, no TV networks employ specialized availability consultants. (Your authors can recommend a couple, though, if any networks are interested.) Stock price. A common follow-on to the bad publicity and press from downtime is that Wall Street may perceive a corporate weakness and sell the company’s stock. This can have a wide-reaching effect throughout the company, personally affecting everyone all the way up to the CEO. Legal liability. Stockholders routinely bring lawsuits against companies whose stock has taken precipitous drops. Many of these are dismissed as frivolous, but others are not. The likelihood of corporate officers being demonstrably liable will increase when there is the perception that those officers of the company did not take appropriate measures to maintain computer system reliability. Corporations can be held liable if their officers do not deliver on their obligations of fiduciary responsibility. An extended and expensive outage can certainly throw into question whether or not the officers have done so. There are also many laws that have been on the articles for years that require corporations to take adequate steps to protect their critical resources; failure to do so could create a morass of legal issues. Employee morale. Some bad press, a stock drop, and a few liability cases can easily create serious morale problems around a company, as employees start to wonder what will go wrong next. Some talented employees may leave, and others may not work as hard if they believe things are going bad at their company. External reputation. If word gets out that key people are unhappy and are leaving the enterprise, that can further hurt its reputation, putting the enterprise into a vicious cycle where bad news makes the reputation worse, and the bad reputation causes things to get worse inside the company.

The Value of Availability

Downtime costs money. It can cost big. It can cost entire companies. Unfortunately, it can also cost big money to protect systems against downtime. There is no free lunch. Later in this article, we discuss in detail the technologies that can improve system availability and allow your critical systems to achieve high availability. In this section, we are going to justify the use of these technologies from a business and financial perspective.