Networks are naturally susceptible to failures because they contain many components and are affected by the configuration of every component. Where, exactly, is your network? In the switch? The drop cables? Bounded by all of the network interface cards in your systems? Any of those physical components can break, resulting in network outages or, more maddeningly, intermittent network failures. Networks are also affected by configuration problems. Incorrect routing information, duplicate hostnames or IP addresses, and machines that misinterpret broadcast addresses can lead to misdirected packets. You’ll also have to deal with redundancy in network connections, as you may have several routers connecting networks at multiple points. When that redundancy is broken, or its configuration is misrepresented, the network appears to be down. When a network that you trust and love is connected to an untrusted or unmanaged network, you run the risk of being subject to a denial-of-service attack or a network penetration attempt from one of those networks. These types of attacks happen within well-run networks as well. Security mogul Bill Cheswick asks the attendees at his talks if they leave their wallets out in the open in their offices. Nary a hand goes up. Then he asks how many leave unprotected network access points like twisted-pair wall jacks in open offices, and you see the tentative hands raised. Access to the network is valuable and has to be protected while still allowing user activity to proceed without onerous overhead.

Finally, networks use a variety of core services or basic information services that we lump into the network fabric. Naming systems like NIS or DNS, security and authentication servers, or host configuration servers for hosts requiring DHCP to boot and join a network will bring down a network if they are not functioning or are giving out wrong answers.

File and Print Server Failures

When file and print servers fail, clients will hang or experience timeouts. A timeout can mean that a print job or a file request fails. The timeout can also lead to wrong answers or data corruption. For example, using Network File Systems (NFS) soft mounts, a write operation that times out will not be repeated. This can lead to holes in data files that will only be detected when the file is read.

Database System Failures

Like any complex application, database systems contain many moving parts. These moving parts are not found in fans or disk drives, however: They are the interrelated subapplications that make up any large enterprise application. The heart of a database system is the server process, or database engine, the main and primary database component that does the reading and writing to the disk, manages the placement of data, and responds to queries with (we hope) the correct answers. If this process stops working, all users accessing the database stop working. The database engine may be assisted by reader-writer or block manager processes that handle disk I/O operations for the engine, allowing it to execute database requests while other processes coordinate I/O and manage the disk block cache. Between the users and the database server sits the listener process. The listener takes the incoming queries from the users and turns them into a form that the database server can process. Then, when the server returns its answer, the listener sends the answer back to the user who requested it. The users, at their client workstations, run their end-user application, which is almost always one level removed from the actual SQL (structured query language) engine.

The end-user application translates the user’s request into SQL, which is then sent across the network to the listener. Well-written end-user applications also shield the user from the dreary complexities of the nearly perfect grammar that SQL requires, and from ordinary problems with the database, such as server crashes and other widespread downtime. Obviously, the failure of any of these processes in the chain will cause the database to be unavailable to its users. Possible failures can include the following:

Application crashes. The application stops running completely, leaving an error message (we hope) that will enable the administrators to determine the nature of the problem.

Application hangs. A more insidious problem with databases or other systems that have significant interaction with the operating system is when a component process, such as a listener, reader-writer process manager, or the database kernel, hangs waiting for a system resource to free or gets trapped in a deadlock with another process. Some very longrunning database operations (such as a scan and update of every record) may appear to make the system hang when they are really just consuming all available cycles.

Resource shortfalls. The most common resource shortfall to strike most database environments is inadequate disk space. If the space allocated to the database fills up, the database engine may crash, hang, or simply fail to accept new entries. None of these is particularly useful in a production environment. If the database itself doesn’t fill, the logs can overflow. There are logs that are written into the database disk space itself, and others that may be written into regular filesystem space. When data cannot be written to either type of log, the database will not perform as desired; it could hang, crash, stop processing incoming requests, or act in other antisocial ways.

Database index corruption. A database server may manage terabytes of data. To find this data quickly on their disks, database servers (and filesystems, for that matter) use a confusing array of pointers and links. Should these pointers become corrupted, the wrong data can be retrieved, or worse, the attempt to retrieve data from an illegal space can result in the application or the system crashing completely. Data corruption problems are fairly unusual because most good RDBMSs have consistency checkers, which scan the database for corruption on startup.

Buggy software. Almost by definition, software has bugs. (There is an old saw in computing that says all programs have at least one bug in them and can be shortened by at least one line. By extension, that means that all programs can be cut down to one line in length, and that line will have a bug in it.) Software is written by humans, and most of us, from time to time, make mistaks. Bugs can impact the system in various ways, from a simple misspelling in a log entry to a fatal bug that crashes the server and/or system. When trying to solve a problem, always consider the possibility that it was caused by a bug. Don’t just assume that all problems were caused by bugs, but at the same time, don’t strike bug from the list of possible causes for almost any problem. And these bugs can occur at any point in the subapplication chain, server processes, listener processes, client SQL engines, user applications, or even with the user’s keyed input.