notes-cog-ai-whyComputersCrash

One issue in AI is, why do computer programs 'crash' when humans do not?

Computers crash because we want them to crash. Periodically people come up with schemes for having computers carry on even after errors, but the received wisdom of program design is that if the program computes something that the programmer thought would be impossible, then it is best that the program should immediately stop and await further instructions, because this is proof positive that the program is doing something nonsensical and therefore all subsequent results should be regarded as potentially meaningless.

But why isn't this the same case with humans? It's because of the sorts of tasks to which we put computers. We tend to use computers in one of two ways. (1) as a simple tool for manipulating, storing, and communicating data that should function without its users understanding its internal workings (for example, if you send an email, you want the end user to conceptualize this as writing a letter and then sending it; you don't want the end user to have to think about the nuances of TCP/IP routing protocols; similarly if the user is using a word processor you want them to think of writing on a piece of paper with a pencil; you don't want them to have to think of list insertion algorithms). (2) as an oracle that runs a predefined computation and gives an answer to a question. For example, you might have the computer calculate how much load a bridge can bear. In both of (1) and (2), hidden incorrect behavior is worse than useless. If an email program, you don't want the computer to act like a drunken mail carrier who delivers the letter that you addressed to your lover, to your boss, and vice versa. If a word processor, you don't want the computer to silently overwrite your saved files with nonsense. If a bridge's maximal load is being computed, you don't want the computer to confidently spout out some incorrectly high answer.

Contrast with humans, who evolved from animals. An animal's job is to stay alive and reproduce. Most of the time, the animal has to recognize food, go over to it, and eat it, while avoiding eating poison; to find a nice place to sleep; and to hide and run away from predators. Every now and then the animal has to mate, but it only has to get that complex sequence correct a few times in it's life. If the animal does some non-sensical things sometimes, for instance if it wanders around in circles a few times, if it sometimes chews up some food only to spit it out , or if it thinks something might be a predator which is not and runs away, it's usually not fatal. Whereas, it would be fatal if the animal, realizing that it is not sure if it was going in a circle or not, simply lay down where it was and starved to death while wait for further instructions from its creator. The rule for animals is "the show must go on"; in the face of uncertainty, make a guess and carry on; and if you realize you may be doing something nonsensical, whatever, just try not to hurt yourself and be sure to interrupt your nonsense to run away from a predator if you see one. Even if the animal blunders into a sharp object and causes a minor injury, it cannot simply lie down and stop moving until the wound heals; it must continue to go about its business and accept the risk that it will repeat the mistake and make the wound worse. (animals do sometimes temporarily freeze in place, but this is not like a computer crashing; the freeze is not permanent, and given enough time and the lack of any scary stimulus, the animal will spontaineously begin to move again and go about its business).

To summarize, computers are used as tools and as question-answering devices by humans. We would rather our tool not do anything than silently mess things up, and we would prefer our question-answering device tell us that it doesn't know the answer rather than confidently giving us the wrong answer. In both cases, a human operator can then decide what to do in the context of the larger task. Whereas for animals, there is no outside 'operator' to do anything if the animal were to stop behaving and freeze in place. The animal must carry on behaving no matter what, accepting the risk that it will behave irrationally and make things worse for itself.