By automatically extracting information from the Web, we can scale up the resulting knowledge bases to much greater sizes than current collections of manually gathered and user-contributed knowledge.
In a demonstration match here at the I.B.M. laboratory against two researchers recently, Watson appeared to be both aggressive and competent, but also made the occasional puzzling blunder.
For example, given the statement, “Bordered by Syria and Israel, this small country is only 135 miles long and 35 miles wide,” Watson beat its human competitors by quickly answering, “What is Lebanon?”
Moments later, however, the program stumbled when it decided it had high confidence that a “sheet” was a fruit.
The goal is to have Watson successfully compete against human Jeopardy champions. Qualification matches will apparently begin later this year.
Fascinating. Jeopardy uses data from a broad range of literature, popular culture, history, politics, and so forth. These are areas of human interest. People can often effortlessly rise to the challenge of answering questions, even if they involve puns, temporal relations, rhymes or other hurdles. This is a much tougher problem for computers.
It’s a game in which according to IBM the best players provide correct answers more than 85% of the time. Educated guessing can be useful, but on the other hand, successful players also need to know, when they simply do not know the answer and should not risk giving a wrong one. It is also not enough to be able to answer a question; the question needs to be answered before other players manage to do so.
It should be very interesting to observe progress of this project. If successful, this could push the envelope in natural language processing, search and reasoning as well as influence applications in areas of business and life.
Social relationship do not last very long here, unless perhaps, if the Virtual Receptionist maintains a memory of the people it has encountered and utilizes the information during subsequent interactions. I also cannot tell, in how far emotions and/or gestures play a part here. There are however several very interesting bits in that demo. The system identifies the number of participants in the current interaction as well as their poses. It carries on conversations and performs administrative tasks. It also appears able to address a person that it identified as not being part of the current conversation.
It is educational to observe the behavior of the system, but it is also very useful to observe how people’s behavior as they interact with the system.
It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either “X is A and Y is B” or “X is B and Y is A.”
[...]
In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as “I am the woman, don’t listen to him!” to her answers, but it will avail nothing as the man can make similar remarks.
We now ask the question, “What will happen when a machine takes the part of A in this game?” Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman?
He effectively changed the question from whether computers can think to whether they can emulate human reasoning and conversational behavior in such a way as to fool an observing person. Turing explored nine different objections that may be raised against the idea of a machine being capable of that emulation.
I found it intriguing that Turing seemed to find the argument involving extrasensory perception and telepathy in particular most compelling:
I assume that the reader is familiar with the idea of extrasensory perception, and the meaning of the four items of it, viz., telepathy, clairvoyance, precognition and psychokinesis. These disturbing phenomena seem to deny all our usual scientific ideas. How we should like to discredit them! Unfortunately the statistical evidence, at least for telepathy, is overwhelming. It is very difficult to rearrange one’s ideas so as to fit these new facts in. Once one has accepted them it does not seem a very big step to believe in ghosts and bogies. The idea that our bodies move simply according to the known laws of physics, together with some others not yet discovered but somewhat similar, would be one of the first to go.
This argument is to my mind quite a strong one. One can say in reply that many scientific theories seem to remain workable in practice, in spite of clashing with ESP; that in fact one can get along very nicely if one forgets about it. This is rather cold comfort, and one fears that thinking is just the kind of phenomenon where ESP may be especially relevant.
I did not know there was overwhelming statistical evidence for the existence of telepathy, not even in Turing’s time. Given the paper’s date, Turing may have referred to experiments that J.B. Rhine conducted with Zener cards in the 1930s. Supposedly, they had some statistically significant test results. Is that what Turing based his statement on?
[Rhine] claimed in his first book (Extra-Sensory Perception, 1934) that he’d done over 90,000 trials and could justifiably conclude that ESP is “an actual and demonstrable occurrence.” However, there were attempts to duplicate these trials at Princeton, Johns Hopkins, Colgate, Southern Methodist, and Brown, all without success.
Shortly afterward, Samuel George Soal, a critic of Rhine’s conducted seemingly conclusive experiments, too:
Soal was able to report a significant displacement effect in his data for two of his earlier participants. Carington and Soal co-authored a paper on the effect, published in Nature in 1940. Soal thereupon sought to confirm these observations with new studies with these participants: Basil Shackleton (a celebrated London portrait photographer, later to become associated with the “Grape Cure” for cancer) and Gloria Stewart.
These studies (conducted in collaboration with K. Goldney and F. Bateman) were widely reviewed as among the most challenging proofs of precognition and telepathy. Not only were the significances of the studies – in terms of the correspondence between ESP guesses and random targets – extraordinary, the procedures appeared to allow no alternative hypothesis. There was also the testimony of 21 prominent observers who, individually, monitored Soal’s work with Shackleton, that they were satisfied with the conditions, and could conceive of no means by which the results could be obtained by normal means other than ESP.
His findings and methods were however also quickly brought into doubt:
Arguments against Soal’s data have, however, been raised ever since their publication. These required some startling propositions themselves, including the notion of “unconscious whispering” – from the ‘agent;’ to the ‘percipient’, always unheard by the intervening experimenter, and unmitigated by distance – and the total inapplicability of probability theory to science (as offered by George Spencer-Brown). Outright fraud was also advanced (firstly by George R. Price) and prominently canvassed in the American journal Science.
Ultimately it appears that Soal was not able to replicate his experiments in later years. His findings were eventually rejected as fraudulent and impermissible.
What study did Turing refer to, when he spoke of statistically overwhelming evidence for telepathy in 1950?