Notes on the Turing Test by Darrin Chandler
Naive Judges
Loebner's contest used ordinary people as judges. At first blush this seems reasonable, but at this time it is not. The sophistication of an "ordinary" person to judge whether an entity is a person is not at a sufficient level. When moviegoers first saw King Kong they were frightened by the clumsy, stop-action goliath. Who would be scared by this film today? The difference is in the line the viewer draws between what can be faked and what must be reality. At the time the special effects were cutting edge, and the viewers experience told them that what they were seeing must be real. An effective Turing Test should use judges easily capable of telling a simple chatterbox such as Eliza from a real person.
Scope
Loebner's early contests (at least) limited the scope of conversation to a few topics, in order that the existing technology could compete at all. While this lowers the bar, it basically invalidates the whole concept of the Turing Test. Real people are ignorant on many topics, but can hold a conversation anyway. An artificial entity needs to do likewise.
Validity
Is the Turing Test a worthwhile litmus test of intelligence? I believe it is. If an artificial entity can fool a savvy judge into thinking it human then it's got a lot going for it. Is passing the Turing Test a requirement of intelligence? No.
Arguments against the Turing Test from the AI community seem to stem mostly from the feeling that it devalues work currently being done. There are many reasons for that feeling. A researcher working on video pattern matching may feel that the work is important but the Turing Test doesn't measure it. I'd say that's true, but it doesn't invalidate the test itself. It's just not testing video pattern matching. The big problem is that the AI community rejects the test altogether, for any meaningful purpose. Looks like they're trying to avoid accountability to any absolute standard. Not surprising when you consider their lackluster progress. Here's a simple text-based test ready to use right now, and nobody can pass it. What's all that grant money got us? Nothing! Oops. Did I say that out loud?
One argument states that if all possible conversations (of a limited length) were mapped out, and a program simply traced the resultant tree and gave the indicated resposes, then it would pass the Turing Test yet not be intelligent. A behavior-indepenant test is needed. There are major flaws with this argument. First, people use mapped out response patterns all the time to great effect. It's an invaluable tool for a thinking being. Second, if a subsequent conversation parallelled the previous, it would be a dead give-away to a discerning judge. Third, deferring all notions of testing until intelligence can be strictly defined is an old, tired trick that has little value to anyone interested in anything except protecting grant money.
Alan Turing's own arguments for the test are valid, and intuitive. The refutations come off as whining, self-serving denials of reality.
Extending the Turing Test
There have been many proposals to extend the test to cover virtual reality, video, speech, and other technologies. Exciting stuff, but largely irrelevant for now. It also defers the need to pass the existing Turing Test, and in doing so causes more harm than good to the state of the art. The original text-based test is more than strong enough to disqualify all contenders at present. There's no need to extend the test except as a showcase for bells and whistles. When we have artificial entities capable of passing the turing test then we'll surely raise the bar. Until then let's get to work on the fundamentals.
Failing the Turing Test
...does not mean failure. There are a great many areas where AI technologies can be applied that the Turing Test does not measure. Progress in these areas can be useful, exciting, and are desperately needed. The existence of areas not measured by the Turing Test does not, however, invalidate the test itself.
Conclusions
The Turing Test provides a valuable litmus test for determining intelligent behavior. Dodging of this test by the AI community at large is a flat refusal to be held to the standard of producing any measurable progress. Acceptance of this test for what it is would be a huge first step in advancing the state of the art of AI.
References
Alan Turing's original paper
The Loebner Prize
Stanford Encyclopedia of Philosophy: The Turing Test
Hosting by Intrepid