The Logic of Failure: Recognizing and Avoiding Error in Complex Situations

Chapter 5: Time Sequences

... we seem to rely on only a few mechanisms of prognostication to gain insight into the future. The primary such mechanism is extrapolation from the moment. In other words, those aspects of the present that anger, worry, or delight us the most will play a key role in our predictions of the future .... Two factors come together in extrapolations from the moment: first, limited focus on a notable feature of the present and, second, extension of the perceived trend in a more or less linear and "monotone" fashion (that is, without allowing for any change in direction) ... If we can identify the typical difficulties people have in dealing with time and in recognizing temporal patterns, we can suggest ways to overcome these difficulties and to improve temporal intuition.

One example which Dorner discusses at length is the rate of spread of AIDS, and the many ways in which statistical data has been misinterpreted. His analysis is relevant to many "developing crises," viz:

• After initial recognition of a phenomena, it may appear that the phenomena is rapidly increasing, as a backlog of instances of the phenomena are suddenly recognized.
• After a period of time, the rate of increase in a phenomena tapers off as other factors act to limit further growth.
• It is very risky to chose a mathematical model for a phenomena based on limited data, and then to use that model to extrapolate very far into the future.
• "we cannot interpret numbers solely on the basis of their size. To understand what they mean, we have to take into account the process that produced them, and that is not always easy."

In another study, Dorner describes a simulation in which test subjects were asked to control the temperature in a room using an un-calibrated thermostat and a thermometer. The simulation included physical factors such as thermal lag. Again, the test subjects had difficulty in dealing with the situation, in some cases developing what psychologists call "superstitious behavior". [Learned superstitious behavior is seen in chickens, for example, which are kept in a cage and fed on a "random reinforcement schedule." Evidently the chickens perceive a cause-effect relationship in what they are doing at the instant of random reinforcement.] Dorner describes this phenomena as follows: "people tend ... to over-generalize from local experience, to ritualize ..."

Other simulations are described which fall into the general category of "predator and prey ecosystems," in which experimental subjects are asked to control the population of predator and prey within certain target limits. Again, people generally fared poorly, although presenting the experimental subjects with relevant graphs of the various parameters as a function of time had some beneficial effect on their efficiency.

Chapter 6: Planning

Chapter 7: So Now What Do We Do?