This paper provides a survey of philosophical discussion of the "the Turing Test". In particular, it provides a very careful and thorough discussion of the famous 1950 paper that was published in Mind.

The Turing Test is one of the most disputed topics in artificial intelligence, philosophy of mind, and cognitive science. This paper is a review of the past 50 years of the Turing Test. Philosophical debates, practical developments and repercussions in related disciplines are all covered. We discuss Turing's ideas in detail and present the important comments that have been made on them. Within this context, behaviorism, consciousness, the 'other minds' problem, and similar topics in philosophy (...) of mind are discussed. We also cover the sociological and psychological aspects of the Turing Test. Finally, we look at the current situation and analyze programs that have been developed with the aim of passing the Turing Test. We conclude that the Turing Test has been, and will continue to be, an influential and controversial topic. -/- (This paper was reprinted in: The Turing Test: The Elusive Standard of Artificial Intelligence, J. H. Moor, ed., Kluwer Academic, pp. 23-78, 2003.). (shrink)

The Turing test has been studied and run as a controlled experiment and found to be underspecified and poorly designed. On the other hand, it has been defended and still attracts interest as a test for true artificial intelligence (AI). Scientists and philosophers regret the test’s current status, acknowledging that the situation is at odds with the intellectual standards of Turing’s works. This article refers to this as the Turing Test Dilemma, following the (...) observation that the test has been under discussion for over seventy years and still is widely seen as either too bad or too good to be a valuable experiment for AI. An argument that solves the dilemma is presented, which relies on reconstructing the Turing test as a thought experiment in the modern scientific tradition. It is argued that Turing’s exposition of the imitation game satisfies Mach’s characterization of the basic method of thought experiments and that Turing’s uses of his test satisfy Popper’s conception of the critical and heuristic uses of thought experiments and Kuhn’s association of thought experiments to conceptual change. It is emphasized how Turing methodically varied the imitation game design to address specific challenges posed to him by other thinkers and how his test illustrates a property of the phenomenon of intelligence and suggests a hypothesis on machine learning. This reconstruction of the Turing test provides a rapprochement to the conflicting views on its value in the literature. (shrink)

Turing''s test has been much misunderstood. Recently unpublished material by Turing casts fresh light on his thinking and dispels a number of philosophical myths concerning the Turing test. Properly understood, the Turing test withstands objections that are popularly believed to be fatal.

The Turing Test, originally proposed as a simple operational definition of intelligence, has now been with us for exactly half a century. It is safe to say that no other single article in computer science, and few other articles in science in general, have generated so much discussion. The present article chronicles the comments and controversy surrounding Turing's classic article from its publication to the present. The changing perception of the Turing Test over the last (...) fifty years has paralleled the changing attitudes in the scientific community towards artificial intelligence: from the unbridled optimism of 1960's to the current realization of the immense difficulties that still lie ahead. I conclude with the prediction that the Turing Test will remain important, not only as a landmark in the history of the development of intelligent machines, but also with real relevance to future generations of people living in a world in which the cognitive capacities of machines will be vastly greater than they are now. (shrink)

Stuart M. Shieber’s name is well known to computational linguists for his research and to computer scientists more generally for his debate on the Loebner Turing Test competition, which appeared a decade earlier in Communications of the ACM. 1 With this collection, I expect it to become equally well known to philosophers.

I have discussed the frame problem and the Turing test at length, but I have not attempted to spell out what I think the implications of the frame problem ...

The Turing Test is claimed by many to be a way to test for the presence, in computers, of such ``deep'' phenomena as thought and consciousness. Unfortunately, attempts to build computational systems able to pass TT have devolved into shallow symbol manipulation designed to, by hook or by crook, trick. The human creators of such systems know all too well that they have merely tried to fool those people who interact with their systems into believing that these (...) systems really have minds. And the problem is fundamental: the structure of the TT is such as to cultivate tricksters. A better test is one that insists on a certain restrictive epistemic relation between an artificial agent A, its output o, and the human architect H of A – a relation which, roughly speaking, obtains when H cannot account for how A produced o. We call this test the ``Lovelace Test'' in honor of Lady Lovelace, who believed that only when computers originate things should they be believed to have minds. (shrink)

In 1950, Alan Turing proposed his eponymous test based on indistinguishability of verbal behavior as a replacement for the question "Can machines think?" Since then, two mutually contradictory but well-founded attitudes towards the Turing Test have arisen in the philosophical literature. On the one hand is the attitude that has become philosophical conventional wisdom, viz., that the Turing Test is hopelessly flawed as a sufficient condition for intelligence, while on the other hand is the (...) overwhelming sense that were a machine to pass a real live full-fledged Turing Test, it would be a sign of nothing but our orneriness to deny it the attribution of intelligence. The arguments against the sufficiency of the Turing Test for determining intelligence rely on showing that some extra conditions are logically necessary for intelligence beyond the behavioral properties exhibited by an agent under a Turing Test. Therefore, it cannot follow logically from passing a Turing Test that the agent is intelligent. I argue that these extra conditions can be revealed by the Turing Test, so long as we allow a very slight weakening of the criterion from one of logical proof to one of statistical proof under weak realizability assumptions. The argument depends on the notion of interactive proof developed in theoretical computer science, along with some simple physical facts that constrain the information capacity of agents. Crucially, the weakening is so slight as to make no conceivable difference from a practical standpoint. Thus, the Gordian knot between the two opposing views of the sufficiency of the Turing Test can be cut. (shrink)

I advocate a theory of syntactic semantics as a way of understanding how computers can think (and how the Chinese-Room-Argument objection to the Turing Test can be overcome): (1) Semantics, considered as the study of relations between symbols and meanings, can be turned into syntax – a study of relations among symbols (including meanings) – and hence syntax (i.e., symbol manipulation) can suffice for the semantical enterprise (contra Searle). (2) Semantics, considered as the process of understanding one domain (...) (by modeling it) in terms of another, can be viewed recursively: The base case of semantic understanding –understanding a domain in terms of itself – is syntactic understanding. (3) An internal (or narrow ), first-person point of view makes an external (or wide ), third-person point of view otiose for purposes of understanding cognition. (shrink)

In this paper we discuss the views on the Turing test of four influential thinkers who belong to the tradition of analytic philosophy: Ludwig Wittgenstein, Noam Chomsky, Hilary Putnam and John Searle. Based on various beliefs about philosophical and/or linguistic matters, they arrive at different assessments of both the significance and suitability of the imitation game for the development of cognitive science and AI models. Nevertheless, they share a rejection of the idea that one can treat Turing (...) test as a test for “machine thinking.” This seems to stem from a concern for the proper use of language —one that is a fundamental methodological commitment of analytic philosophy. (shrink)

In the classical Turing test, participants are challenged to tell whether they are interacting with another human being or with a machine. The way the interaction takes place is not direct, but a distant conversation through computer screen messages. Basic forms of interaction are face-to-face and embodied, context-dependent and based on the detection of reciprocal sensorimotor contingencies. Our idea is that interaction detection requires the integration of proprioceptive and interoceptive patterns with sensorimotor patterns, within quite short time lapses, (...) so that they appear as mutually contingent, as reciprocal. In other words, the experience of interaction takes place when sensorimotor patterns are contingent upon one’s own movements, and vice versa. I react to your movement, you react to mine. When I notice both components, I come to experience an interaction. Therefore, we designed a “minimal” Turing test to investigate how much information is required to detect these reciprocal sensorimotor contingencies. Using a new version of the perceptual crossing paradigm, we tested whether participants resorted to interaction detection to tell apart human from machine agents in repeated encounters with these agents. In two studies, we presented participants with movements of a human agent, either online or offline, and movements of a computerized oscillatory agent in three different blocks. In each block, either auditory or audiovisual feedback was provided along each trial. Analysis of participants’ explicit responses and of the implicit information subsumed in the dynamics of their series will reveal evidence that participants use the reciprocal sensorimotor contingencies within short time windows. For a machine to pass this minimal Turing test, it should be able to generate this sort of reciprocal contingencies. (shrink)

The Turing Test is routinely understood as a behaviourist test for machine intelligence. Diane Proudfoot has argued for an alternative interpretation. According to Proudfoot, Turing’s claim that intelligence is what he calls ‘an emotional concept’ indicates that he conceived of intelligence in response-dependence terms. As she puts it: ‘Turing’s criterion for “thinking” is…: x is intelligent if in the actual world, in an unrestricted computer-imitates-human game, x appears intelligent to an average interrogator’. The role of (...) the famous test is thus to provide the conditions in which to examine the average interrogator’s responses. I shall argue that Proudfoot’s analysis falls short. The philosophical literature contains two main models of response-dependence, what I shall call the transparency model and the reference-fixing model. Proudfoot resists the thought that Turing might have endorsed one of these models to the exclusion of the other. But the details of her own analysis indicate that she is, in fact, committed to the claim that Turing’s account of intelligence is grounded in a transparency model, rather than a reference-fixing one. By contrast, I shall argue that while Turing did indeed conceive of intelligence in response-dependence terms, his account is grounded in a reference-fixing model, rather than a transparency one. This is fortunate, because, as an account of intelligence, the transparency model is arguably problematic in a way that the reference-fixing model isn’t. (shrink)

The paper examines the nature of the behavioral evidence underlying attributions of intelligence in the case of human beings, and how this might be extended to other kinds of cognitive system, in the spirit of the original Turing Test. I consider Harnad's Total Turing Test, which involves successful performance of both linguistic and robotic behavior, and which is often thought to incorporate the very same range of empirical data that is available in the human case. However, (...) I argue that the TTT is still too weak, because it only tests the capabilities of particular tokens within a preexisting context of intelligent behavior. What is needed is a test of the cognitive type, as manifested through a number of exemplary tokens, in order to confirm that the cognitive type is able to produce the context of intelligent behavior presupposed by tests such as the TT and TTT. (shrink)

This paper discusses some difficulties in understanding the Turing test. It emphasizes the importance of distinguishing between conceptual and empirical perspectives and highlights the former as introducing more serious problems for the TT. Some objections against the Turingian framework stemming from the later Wittgenstein’s philosophy are exposed. The following serious problems are examined: 1) It considers a unique and exclusive criterion for thinking which amounts to their identification; 2) it misidentifies the relationship of speaking to thinking as that (...) of a criterion; 3) it neglects the “natural” course of the development in semantics. However, these considerations suggest only that it is problematic to label a successful chatbot as a “thinking entity” without further qualifications, but not necessarily and once and for all incorrect. Philosophy has only little to say about the technical possibility of creating such an effective program. (shrink)

Alan Turing, einer der Gründerväter der modernen Informatik, diskutierte in seinem 1950 veröffentlichten Artikel »Computing Machinery and Intelligence« die Frage, ob Maschinen denken können. Dies wirft jedoch die schwierige Frage auf, was Denken denn sei. Um diese Frage zu umgehen, schlägt Turing vor, stattdessen eine Frage zu stellen, die sich durch bloße Beobachtung klären lässt, nämlich ob eine Maschine ein bestimmtes Spiel erfolgreich spielen könne.

This paper argues against the moral Turing test as a framework for evaluating the moral performance of autonomous systems. Though the term has been carefully introduced, considered, and cautioned about in previous discussions :251–261, 2000; Allen and Wallach 2009), it has lingered on as a touchstone for developing computational approaches to moral reasoning :98–109, 2015). While these efforts have not led to the detailed development of an MTT, they nonetheless retain the idea to discuss what kinds of action (...) and reasoning should be demanded of autonomous systems. We explore the flawed basis of an MTT in imitation, even one based on scenarios of morally accountable actions. MTT-based evaluations are vulnerable to deception, inadequate reasoning, and inferior moral performance vis a vis a system’s capabilities. We propose verification—which demands the design of transparent, accountable processes of reasoning that reliably prefigure the performance of autonomous systems—serves as a superior framework for both designer and system alike. As autonomous social robots in particular take on an increasing range of critical roles within society, we conclude that verification offers an essential, albeit challenging, moral measure of their design and performance. (shrink)

The main factor of intelligence is defined as the ability tocomprehend, formalising this ability with the help of new constructsbased on descriptional complexity. The result is a comprehension test,or C- test, which is exclusively defined in computational terms. Due toits absolute and non-anthropomorphic character, it is equally applicableto both humans and non-humans. Moreover, it correlates with classicalpsychometric tests, thus establishing the first firm connection betweeninformation theoretical notions and traditional IQ tests. The TuringTest is compared with the C- (...) class='Hi'>test and the combination of the two isquestioned. In consequence, the idea of using the Turing Test as apractical test of intelligence should be surpassed, and substituted bycomputational and factorial tests of different cognitive abilities, amuch more useful approach for artificial intelligence progress and formany other intriguing questions that present themselves beyond theTuring Test. (shrink)

The Turing Test is best regarded as a model to test for intelligence, where an entity’s intelligence is inferred from its ability to be attributed with ‘human-likeness’ during a text-based conversation. The problem with this model, however, is that it does not care if or how well an entity produces a meaningful conversation, as long as its interactions are humanlike enough. As a consequence, the TT attracts projects that concentrate on how best to fool the judges. In (...) light of this, I propose a new version of the TT: the Questioning Turing Test. Here, the entity has to produce an enquiry rather than a conversation; and it is parametrised along two further dimensions in addition to ‘human-likeness’: ‘correctness’, evaluating if the entity accomplishes the enquiry; and ‘strategicness’, evaluating how well the entity accomplishes the enquiry, in terms of the number of questions asked. (shrink)

In 2014, widespread reports in the popular media that a chatbot named Eugene Goostman had passed the Turing test became further grist for those who argue that the diversionary tactics of chatbots like Goostman and others, such as those who participate in the Loebner competition, are enabled by the open-ended dialog of the Turing test. Some claim a new kind of test of machine intelligence is needed, and one community has advanced the Winograd schema competition (...) to address this gap. We argue to the contrary that implicit in the Turing test is the cooperative challenge of using language to build a practical working understanding, necessitating a human interrogator to monitor and direct the conversation. We give examples which show that, because ambiguity in language is ubiquitous, open-ended conversation is not a flaw but rather the core challenge of the Turing test. We outline a statistical notion of practical working understanding that permits a reasonable amount of ambiguity, but nevertheless requires that ambiguity be resolved sufficiently for the agents to make progress. (shrink)

Brain damage can cause massive changes in consciousness levels. From a clinical and ethical point of view it is desirable to assess the level of residual consciousness in unresponsive patients. However, no direct measure of consciousness exists, so we run into the philosophical problem of other minds. Neurologists often make implicit use of a Turing test-like procedure in an attempt to gain access to damaged minds, by monitoring and interpreting neurobehavioral responses. New brain imaging techniques are now being (...) developed that permit communication with unresponsive patients, using their brain signals as carriers of messages relating to their mental states. (shrink)

This paper aims to show that dominant conceptions of intelligence used in artificial intelligence (AI) are biased by normative assumptions that originate from the Global North, making it questionable if AI can be uncritically applied elsewhere without risking serious harm to vulnerable people. After the introduction in Sect. 1 we shortly present the history of IQ testing in Sect. 2, focusing on its multiple discriminatory biases. To determine how these biases came into existence, we define intelligence ontologically and underline its (...) constructed and culturally variable character. Turning to AI, specifically the Turing Test (TT), in Sect. 3, we critically examine its underlying intelligence conceptions. The test has been of central influence in AI research and remains an important point of orientation. We argue that both the test itself and how it is used in practice risk promoting a limited conception of intelligence which solely originated in the Global North. Hence, this conception should be critically assessed in relation to the different global contexts in which AI technologies are and will be used. In Sect. 4, we highlight how unequal power relations in AI research are a real threat, rather than just philosophical sophistry while considering the history of IQ testing and the TT’s practical biases. In the last section, we examine the limits of our account and identify fields for further investigation. Tracing colonial continuities in AI intelligence research, this paper points to a more diverse and historically aware approach to the design, development, and use of AI. (shrink)

In the last 20 years the Turing test has been left further behind by new developments in artificial intelligence. At the same time, however, these developments have revived some key elements of the Turing test: imitation and adversarialness. On the one hand, many generative models, such as generative adversarial networks, build imitators under an adversarial setting that strongly resembles the Turing test. The term “Turing learning” has been used for this kind of setting. (...) On the other hand, AI benchmarks are suffering an adversarial situation too, with a ‘challenge-solve-and-replace’ evaluation dynamics whenever human performance is ‘imitated’. The particular AI community rushes to replace the old benchmark by a more challenging benchmark, one for which human performance would still be beyond AI. These two phenomena related to the Turing test are sufficiently distinctive, important and general for a detailed analysis. This is the main goal of this paper. After recognising the abyss that appears beyond superhuman performance, we build on Turing learning to identify two different evaluation schemas: Turing testing and adversarial testing. We revisit some of the key questions surrounding the Turing test, such as ‘understanding’, commonsense reasoning and extracting meaning from the world, and explore how the new testing paradigms should work to unmask the limitations of current and future AI. Finally, we discuss how behavioural similarity metrics could be used to create taxonomies for artificial and natural intelligence. Both testing schemas should complete a transition in which humans should give way to machines—not only as references to be imitated but also as judges—when pursuing and measuring machine intelligence. (shrink)

Mahadevan proposes that we are at the cusp of imagination science, one of whose primary concerns will be the design of imagination machines. Programs have been written that are capable of generating jokes, producing line-drawings that have been exhibited at such galleries as the Tate, composing music in several styles reminiscent of such greats as Vivaldi and Mozart, proving geometry theorems, and inducing quantitative laws from empirical data. In recent years, Dartmouth has been hosting Turing Tests in creativity in (...) three categories: short stories, sonnets, and dance music DJ sets. In this post, I will provide a brief and non-exhaustive survey of some plausible responses to these imagination machines and the related prospects for our understanding of the imagination. (shrink)

Turing’s much debated test has turned 70 and is still fairly controversial. His 1950 paper is seen as a complex and multilayered text, and key questions about it remain largely unanswered. Why did Turing select learning from experience as the best approach to achieve machine intelligence? Why did he spend several years working with chess playing as a task to illustrate and test for machine intelligence only to trade it out for conversational question-answering in 1950? Why (...) did Turing refer to gender imitation in a test for machine intelligence? In this article, I shall address these questions by unveiling social, historical and epistemological roots of the so-called Turing test. I will draw attention to a historical fact that has been only scarcely observed in the secondary literature thus far, namely that Turing’s 1950 test emerged out of a controversy over the cognitive capabilities of digital computers, most notably out of debates with physicist and computer pioneer Douglas Hartree, chemist and philosopher Michael Polanyi, and neurosurgeon Geoffrey Jefferson. Seen in its historical context, Turing’s 1950 paper can be understood as essentially a reply to a series of challenges posed to him by these thinkers arguing against his view that machines can think. Turing did propose gender learning and imitation as one of his various imitation tests for machine intelligence, and I argue here that this was done in response to Jefferson's suggestion that gendered behavior is causally related to the physiology of sex hormones. (shrink)

This article describes research to build an embodied conversational agent as an interface to a question-and-answer system about a National Science Foundation program. We call this ECA the LifeLike Avatar, and it can interact with its users in spoken natural language to answer general as well as specific questions about specific topics. In an idealized case, the LifeLike Avatar could conceivably provide a user with a level of interaction such that he or she would not be certain as to whether (...) he or she is talking to the actual person via video teleconference. This could be considered a extended version of the seminal Turing test. Although passing such a test is still far off, our work moves the science in that direction. The Uncanny Valley notwithstanding, applications of such lifelike interfaces could include those where specific instructors/caregivers could be represented as stand-ins for the actual person in situations where personal representation is important. Possible areas that come to mind that might benefit from these lifelike ECAs include health-care support for elderly/disabled patients in extended home care, education/training, and knowledge preservation. Another more personal application would be to posthumously preserve elements of the persona of a loved one by family members. We apply this approach to a Q/a system for knowledge preservation and dissemination, where the specific individual who had this knowledge was to retire from the US National Science Foundation. The system is described in detail, and evaluations were performed to determine how well the system was perceived by users. (shrink)

This book provides a novel framework for understanding and revising labor markets and education policies in an era of machine learning. It posits that while learning and knowing both require thinking, learning is fundamentally different than knowing because it results in cognitive processes that change over time. Learning, in contrast to knowing, requires time and agency. Therefore, “learning algorithms”—that enable machines to modify their actions based on real-world experiences—are a fundamentally new form of artificial intelligence that have potential to be (...) even more disruptive to labor markets than prior introductions of digital technology. To explore the difference between knowing and learning, Turing’s “Imitation Game,”—that he proposed as a test for machine thinking—is expanded to include time dependence. The arguments presented in the book introduce three novel concepts: (1) Comparative learning advantage: This is a concept analogous to comparative labor advantage but arises from the disparate times required to learn new knowledge bases/skillsets. It is argued that in the future, comparative learning advantages between humans and machines will determine their division of labor. (2) Two dimensions of job performance—expertise and interpersonal: Job tasks can be sorted into two broad categories. Tasks that require expertise have stable endpoints, which makes these tasks inherently repetitive and subject to automation. Tasks that are interpersonal are highly context-dependent and lack stable endpoints, which makes these tasks inherently non-routine. Humans compared to machines have a comparative learning advantage along the interpersonal dimension, which is increasing in value economically. (3) The Learning Game is a time-dependent version of Turing’s “Imitation Game.” It is more than a thought experiment. The “Learning Game” provides a mathematical framework with quantitative criteria for training and assessing comparative learning advantages. The book is highly interdisciplinary—presenting philosophical arguments in economics, artificial intelligence, and education. It also provides data, mathematical analysis, and testable criteria that researchers in these fields will find of practical use. The book calls for a rethinking of how labor markets operate and how the education system should prepare students for future jobs. It concludes with a list of counterintuitive recommendations for future education and labor policies that all stakeholders—employers, employees, educators, students, and political leaders—should heed. (shrink)

My basic argument is that a computer cannot distinguish between metaphor and nonsense. This would be my new “Turing Test.” I was very fond of a particular Italian poem, but I was told by an Italian friend that it was a hackneyed poem of little worth. I then taught myself to experience the poem alternately, as real poetry and as the silly nonsense that my friend claimed it really was. Having done so, I realized that I could do (...) the same with any metaphor, such as “having a green thumb.” Thinking about the nature of this switch, from the literal to the metaphorical, I also realized that it was the sort of change that could not be prescribed or even described: this, the basic aesthetic gesture, remains beyond the boundary of logical definition. It then dawned on me that it might provide a test for the validity of a “Turing Test,” by any definition. Can a computer track a mind as it goes through this transformation? I could not envisage such a possibility. This would be a “Turing Test” based on the discipline of aesthetics rather than on technology. It may even be argued that the ability to experience metaphor is the very definition of the human. (shrink)

Anti-behaviorist arguments against the validity of the Turing Test as a sufficient condition for attributing intelligence are based on a memorizing machine, which has recorded within it responses to every possible Turing Test interaction of up to a fixed length. The mere possibility of such a machine is claimed to be enough to invalidate the Turing Test. I consider the nomological possibility of memorizing machines, and how long a Turing Test they can (...) pass. I replicate my previous analysis of this critical Turing Test length based on the age of the universe, show how considerations of communication time shorten that estimate and allow eliminating the sole remaining contingent assumption, and argue that the bound is so short that it is incompatible with the very notion of the Turing Test. I conclude that the memorizing machine objection to the Turing Test as a sufficient condition for attributing intelligence is invalid. (shrink)