All Technological Systems Can Fail

Most modern technological systems, from transistor radios to airliners, have been engineered and produced to be remarkably reliable. Failure is rare enough to be surprising. Yet the larger and more complex a system is, the more ways there are in which it can go wrong—and the more widespread the possible effects of failure. A system or device may fail for different reasons: because some part fails, because some part is not well matched to some other, or because the design of the system is not adequate for all the conditions under which it is used. One hedge against failure is overdesign—that is, for example, making something stronger or bigger than is likely to be necessary. Another hedge is redundancy—that is, building in one backup system or more to take over in case the primary one fails.

If failure of a system would have very costly consequences, the system may be designed so that its most likely way of failing would do the least harm. Examples of such "fail-safe" designs are bombs that cannot explode when the fuse malfunctions; automobile windows that shatter into blunt, connected chunks rather than into sharp, flying fragments; and a legal system in which uncertainty leads to acquittal rather than conviction. Other means of reducing the likelihood of failure include improving the design by collecting more data, accommodating more variables, building more realistic working models, running computer simulations of the design longer, imposing tighter quality control, and building in controls to sense and correct problems as they develop.

All of the means of preventing or minimizing failure are likely to increase cost. But no matter what precautions are taken or resources invested, risk of technological failure can never be reduced to zero. Analysis of risk, therefore, involves estimating a probability of occurrence for every undesirable outcome that can be foreseen—and also estimating a measure of the harm that would be done if it did occur. The expected importance of each risk is then estimated by combining its probability and its measure of harm. The relative risk of different designs can then be compared in terms of the combined probable harm resulting from each.