Back in the days when ICQ was the default messaging platform for us teenagers I can remember becoming rather familiar with all manner of chatbots that’d grace my presence. Most of the time they were programmed to get you to go to a website, sometimes legitimate although almost always some kind of scam, but every so often you’d get one that just seemed to be an experiment to see how real they could make one. It wouldn’t take long to figure out if there was a real person on the end or not though as their variety of responses were limited and they would often answer questions with more questions, a telltale sign of an expert system. Since those heydays my contact with chatbots has been limited mostly to those examples that have done well in Turing Test competitions around the world. Even those however have proved to be less than stellar, showing that this field still has a long way to go.
However news has been making the rounds that a plucky little chatbot named Eugene Goostman has passed the Turing Test for the first time. Now the definition of the Turing Test itself is somewhat nebulous, being only that a human judge isn’t able to tell the difference between computer generated responses from that of a human, and if you take that literally it’s already been passed several times over. Many, including myself, take it to mean a little more in that a chatbot would have to be able to fool the majority of people into thinking it was human before it could be accepted as having passed the test. In that regard Eugene here hasn’t really passed the test at all, although I do admit it’s creator’s strategy was a good attempt at poking holes in the test’s vague definition.
You see Eugene isn’t your typical generic chatbot, instead he’s actually been programmed to be a 13 year old Ukrainian boy to whom English is a second language. It’s clever because you can then limit the problem space of what he can answer significantly as you wouldn’t expect a 13 year old to know a lot of things and when you’re questioning a non-native speaker the verbal tools you have available to you are again limited. At the same time however this is simply an artificial way of making the chatbot seem more human than it actually is. Indeed this is probably the biggest criticism that has been levelled at Eugene since its rise to fame as it appeared to dodge more responses than it could give answers to, a telltale sign that you’re speaking to an AI.
So as you can probably tell by the tone of my writing I don’t think that Eugene qualifies as having passed the Turing Test as the criteria used (33% of the judges were fooled) weren’t sufficient as otherwise several other bots would have claimed that title previously. I wholly admit this is due in part to the nebulous nature of how Turing first posited the test, whereby the interpretation of “passed” varies wildly between individuals, but my sentiment does seem to echo with the wider AI community. I think the ideas behind generating the Eugene chatbot are interesting as it shows how the problem space can be narrowed down but if the chance of it fooling someone is less than random then, in my mind, that does not qualify as a pass.
I don’t expect that the Turing Test will be past to a majority of the AI community’s satisfaction for some time to come as it requires duplicating so many human functions that we just haven’t been able to translate into code yet. For me the easiest way to tell a bot from a human is to teach it something esoteric and have it repeat its own interpretation back to me, something which no chatbot has been able to do to date. Indeed just that simple example, being able to teach it something and have it interpret it based on its own knowledge base, entails leaps forward in AI that just don’t exist in a general form yet. I’m not saying that it will never happen, far from it, but the system that first truly passes the Turing Test is yet to come and is likely many years away from reality.
