Striking similarities across ape gestural repertoires suggest shared phylogenetic origins that likely provided a foundation for the emergence of language. We pilot a novel approach for exploring possible semantic universals across human and nonhuman ape species. In a forced‐choice task, n = 300 participants watched 10 chimpanzee gesture forms performed by a human and chose from responses that paralleled inferred meanings for chimpanzee gestures. Participants agreed on a single meaning for nine gesture forms; in six of these the (...) agreed form‐meaning pair response(s) matched those established for chimpanzees. Such shared understanding suggests apes' (including humans') gesturing shares deep evolutionary origins. (shrink)

Frege held that the bare demonstrative ‘that’ is incomplete, and that it is the word together with a gesture that serves as the designating expression, and likewise that it is the word ‘yesterday’ together with the time of utterance that designates the relevant day. David Kaplan’s original theory of indexicals holds that Frege’s supplementation thesis is correct about demonstratives but incorrect about ‘yesterday’. Kaplan’s account of demonstratives deviates from Frege’s in treating supplemented demonstratives as directly referential, hence rigid. It (...) is argued here that the gesture or other demonstration that accompanies an utterance of ‘that’ is not part of the designating expression but instead part of the utterance context. (shrink)

We present a formal analysis of iconic coverbal gesture. Our model describes the incomplete meaning of gesture that’s derivable from its form, and the pragmatic reasoning that yields a more specific interpretation. Our formalism builds reported.

Silent gesture is not considered to be linguistic, on par with spoken and sign languages. It is claimed that silent gestures, unlike language, represent events holistically, without compositional structure. However, recent research has demonstrated that gesturers use consistent strategies when representing objects and events, and that there are behavioral and clinically relevant limits on what form a gesture may take to effect a particular meaning. This systematicity challenges a holistic interpretation of silent gesture, which predicts that there (...) should be no stable form‐meaning correspondence across event representations. Here, we demonstrate to the contrary that untrained gesturers systematically manipulate the form of their gestures when representing events with and without a theme (e.g., Someone popped the balloon vs. Someone walked), that is, transitive and intransitive events. We elicited silent gestures and annotated them for manual features active in coding transitivity distinctions in sign languages. We trained linear support vector machines to make item‐by‐item transitivity predictions based on these features. Prediction accuracy was good across the entire dataset, thus demonstrating that systematicity in silent gesture can be explained with recourse to subunits. We argue that handshape features are constructs co‐opted from cognitive systems subserving manual action production and comprehension for communicative purposes, which may integrate into the linguistic system of emerging sign languages. We further suggest that nonsigners tend to map event participants to each hand, a strategy found across genetically and geographically distinct sign languages, suggesting the strategy's cognitive foundation. (shrink)

We examined whether children's ability to integrate speech and gesture follows the pattern of a broader developmental shift between 3- and 5-year-old children regarding the ability to process two pieces of information simultaneously. In Experiment 1, 3-year-olds, 5-year-olds, and adults were presented with either an iconic gesture or a spoken sentence or a combination of the two on a computer screen, and they were instructed to select a photograph that best matched the message. The 3-year-olds did not integrate (...) information in speech and gesture, but 5-year-olds and adults did. In Experiment 2, 3-year-old children were presented with the same speech and gesture as in Experiment 1 that were produced live by an experimenter. When presented live, 3-year-olds could integrate speech and gesture. We concluded that development of the integration ability is a part of the broader developmental shift; however, live-presentation facilitates the nascent integration ability in 3-year-olds. (shrink)

When young children learn to use language, they start to use their hands in co‐verbal gesturing. There are, however, considerable differences between children, and it is not completely understood what these individual differences are due to. We studied how children at 4 years of age employ speech and iconic gestures to convey meaning in different kinds of spatial event descriptions, and how this relates to their cognitive abilities. Focusing on spontaneous illustrations of actions, we applied a semantic feature (SF) analysis (...) to characterize combinations of speech and gesture meaning and related them to the child's visual‐spatial abilities or abstract/concrete reasoning abilities (measured using the standardized SON‐R test). Results show that children with higher cognitive abilities convey significantly more meaning via gesture and less solely via speech. These findings suggest that young children's use of cospeech representational gesturing is positively related to their mental representation and reasoning abilities. (shrink)

Using a cross‐modal semantic priming paradigm, both experiments of the present study investigated the link between the mental representations of iconic gestures and words. Two groups of the participants performed a primed lexical decision task where they had to discriminate between visually presented words and nonwords (e.g., flirp). Word targets (e.g., bird) were preceded by video clips depicting either semantically related (e.g., pair of hands flapping) or semantically unrelated (e.g., drawing a square with both hands) gestures. The duration of gestures (...) was on average 3,500 ms in Experiment 1 but only 1,000 ms in Experiment 2. Significant priming effects were observed in both experiments, with faster response latencies for related gesture–word pairs than unrelated pairs. These results are consistent with the idea of interactions between the gestural and lexical representational systems, such that mere exposure to iconic gestures facilitates the recognition of semantically related words. (shrink)

Gestures and speech are clearly synchronized in many ways. However, previous studies have shown that the semantic similarity between gestures and speech breaks down as people approach transitions in understanding. Explanations for these gesture–speech mismatches, which focus on gestures and speech expressing different cognitive strategies, have been criticized for disregarding gestures’ and speech's integration and synchronization. In the current study, we applied three different perspectives to investigate gesture–speech synchronization in an easy and a difficult task: temporal alignment, semantic (...) similarity, and complexity matching. Participants engaged in a simple cognitive task and were assigned to either an easy or a difficult condition. We automatically measured pointing gestures, and we coded participant's speech, to determine the temporal alignment and semantic similarity between gestures and speech. Multifractal detrended fluctuation analysis was used to determine the extent of complexity matching between gestures and speech. We found that task difficulty indeed influenced gesture–speech synchronization in all three domains. We thereby extended the phenomenon of gesture–speech mismatches to difficult tasks in general. Furthermore, we investigated how temporal alignment, semantic similarity, and complexity matching were related in each condition, and how they predicted participants’ task performance. Our study illustrates how combining multiple perspectives, originating from different research areas (i.e., coordination dynamics, complexity science, cognitive psychology), provides novel understanding about cognitive concepts in general and about gesture–speech synchronization and task difficulty in particular. (shrink)

Using a cross-modal semantic priming paradigm, both experiments of the present study investigated the link between the mental representations of iconic gestures and words. Two groups of the participants performed a primed lexical decision task where they had to discriminate between visually presented words and nonwords (e.g., flirp). Word targets (e.g., bird) were preceded by video clips depicting either semantically related (e.g., pair of hands flapping) or semantically unrelated (e.g., drawing a square with both hands) gestures. The duration of gestures (...) was on average 3,500 ms in Experiment 1 but only 1,000 ms in Experiment 2. Significant priming effects were observed in both experiments, with faster response latencies for related gesture–word pairs than unrelated pairs. These results are consistent with the idea of interactions between the gestural and lexical representational systems, such that mere exposure to iconic gestures facilitates the recognition of semantically related words. (shrink)

People frequently gesture when a word is on the tip of their tongue (TOT), yet research is mixed as to whether and why gesture aids lexical retrieval. We tested three accounts: the lexical retrieval hypothesis, which predicts that semantically related gestures facilitate successful lexical retrieval; the cognitive load account, which predicts that matching gestures facilitate lexical retrieval only when retrieval is hard, as in the case of a TOT; and the motor movement account, which predicts that any motor (...) movements should support lexical retrieval. In Experiment 1 (a between‐subjects study; N = 90), gesture inhibition, but not neck inhibition, affected TOT resolution but not overall lexical retrieval; participants in the gesture‐inhibited condition resolved fewer TOTs than participants who were allowed to gesture. When participants could gesture, they produced more representational gestures during resolved than unresolved TOTs, a pattern not observed for meaningless motor movements (e.g., beats). However, the effect of gesture inhibition on TOT resolution was not uniform; some participants resolved many TOTs, while others struggled. In Experiment 2 (a within‐subjects study; N = 34), the effect of gesture inhibition was traced to individual differences in verbal, not spatial short‐term memory (STM) span; those with weaker verbal STM resolved fewer TOTs when unable to gesture. This relationship between verbal STM and TOT resolution was not observed when participants were allowed to gesture. Taken together, these results fit the cognitive load account; when lexical retrieval is hard, gesture effectively reduces the cognitive load of TOT resolution for those who find the task especially taxing. (shrink)

Speakers frequently produce elaborate hand movements during talk that have been shown to serve a communicative function. Nevertheless, two-thirds of the semantic content of these hand movements is encoded linguistically elsewhere in the discourse . The present experiment demonstrated that while 62.9% of semantic information in gesture was elsewhere, most gestures retained at least one semantic feature that was never represented linguistically. Semantic features were more explicit when they occurred in gesture than when represented linguistically. Even in cases (...) where the imagistic gesture appeared somewhat redundant, gesture at the narrative level preserves a discernable communicative function. (shrink)

During face‐to‐face conversation, transitions between speaker turns are incredibly fast. These fast turn exchanges seem to involve next speakers predicting upcoming semantic information, such that next turn planning can begin before a current turn is complete. Given that face‐to‐face conversation also involves the use of communicative bodily signals, an important question is how bodily signals such as co‐speech hand gestures play into these processes of prediction and fast responding. In this corpus study, we found that hand gestures that depict or (...) refer to semantic information started before the corresponding information in speech, which held both for the onset of the gesture as a whole, as well as the onset of the stroke (the most meaningful part of the gesture). This early timing potentially allows listeners to use the gestural information to predict the corresponding semantic information to be conveyed in speech. Moreover, we provided further evidence that questions with gestures got faster responses than questions without gestures. However, we found no evidence for the idea that how much a gesture precedes its lexical affiliate (i.e., its predictive potential) relates to how fast responses were given. The findings presented here highlight the importance of the temporal relation between speech and gesture and help to illuminate the potential mechanisms underpinning multimodal language processing during face‐to‐face conversation. (shrink)

Natural languages make prolific use of conventional constituent‐ordering patterns to indicate “who did what to whom,” yet the mechanisms through which these regularities arise are not well understood. A series of recent experiments demonstrates that, when prompted to express meanings through silent gesture, people bypass native language conventions, revealing apparent biases underpinning word order usage, based on the semantic properties of the information to be conveyed. We extend the scope of these studies by focusing, experimentally and computationally, on the (...) interpretation of silent gesture. We show cross‐linguistic experimental evidence that people use variability in constituent order as a cue to obtain different interpretations. To illuminate the computational principles that govern interpretation of non‐conventional communication, we derive a Bayesian model of interpretation via biased inductive inference and estimate these biases from the experimental data. Our analyses suggest people's interpretations balance the ambiguity that is characteristic of emerging language systems, with ordering preferences that are skewed and asymmetric, but defeasible. (shrink)

Natural languages make prolific use of conventional constituent‐ordering patterns to indicate “who did what to whom,” yet the mechanisms through which these regularities arise are not well understood. A series of recent experiments demonstrates that, when prompted to express meanings through silent gesture, people bypass native language conventions, revealing apparent biases underpinning word order usage, based on the semantic properties of the information to be conveyed. We extend the scope of these studies by focusing, experimentally and computationally, on the (...) interpretation of silent gesture. We show cross‐linguistic experimental evidence that people use variability in constituent order as a cue to obtain different interpretations. To illuminate the computational principles that govern interpretation of non‐conventional communication, we derive a Bayesian model of interpretation via biased inductive inference and estimate these biases from the experimental data. Our analyses suggest people's interpretations balance the ambiguity that is characteristic of emerging language systems, with ordering preferences that are skewed and asymmetric, but defeasible. (shrink)

This study investigated gesture–speech integration among adolescents who are deaf or hard of hearing and those with typical hearing. Thirty-eight adolescents performed a Stroop-like task in which they watched 120 short video clips of gestures and actions twice at random. Participants were asked to press one button if the visual content of the speaker’s movements was related to a written word and to press another button if it was unrelated to a written word while accuracy rates and response times (...) were recorded. We found stronger GSI effects among DHH participants than hearing participants. The semantic congruency effect was significantly larger in DHH participants than in hearing participants, and results of our experiments indicated a significantly larger gender congruency effect in DHH participants as compared to hearing participants. Results of this study shed light on GSI among DHH individuals and suggest future avenues for research examining the impact of gesture on language processing and communication in this population. (shrink)

The article argues in favour of a pragmatist enactive interpretation of the emergence of the symbolic and contentful mind from a basic form of social communicative interaction in which basic cognitive capacities are involved. Through a critical overview of Radical Enactivists (RECers)’ view about language, the article focuses on Mead’s pragmatist behavioural theory of meaning that refers to the gestural conversation as the origin of the evolution of linguistic conversation. The article develops as follows. After exposing the main elements of (...) REC’s theory of cognition and language that involve the construction of a theory of natural signs (teleosemiotics) and basic directionality (Ur-intentionality), some critical points of Hutto and Myin’s proposal will be highlighted. To foster a continuist perspective of language, the behavioural theory of meaning and language that Mead develops from the notion of gesture will be analysed. His theory is akin to REC and could augment the bare bones of REC’s sketched perspective, helping to include Ur-intentionality in a broader non-dualistic phylogenetic and ontogenetic theory of symbolic language from gestural communication, thus helping to overcome the distinction between a content-less intentionality and a content-involving intentionality, i.e., a semantic propositional intentionality. Furthermore, a recent revival of Mead’s theory testifies to its up-to-date relevance to explain the innate social dimension of human and non-human animals, and the human communicative capacity through the conditioning of bio-social canons and structures. (shrink)

Developed through a series of conceptual analyses (Edmund Husserl, Vilém Flusser, Ludwig Wittgenstein and Walter Benjamin) and case studies (Fernando Lopes’s Belarmino and Jeff Wall’s Mimic), this article delves into the relationship between gesture, attunement and atmosphere and how it unfolds in photographic works dealing with urban space. The first section focuses on the role played by photography in the film Belarmino, which raises questions about both the representation of urban phenomena and issues related to expression and gesture (...) in boxing. The second section discusses Husserl’s thinking on image consciousness and his surprising reference to a “photograph” of a boxer, which reveals the relevance of his phenomenological approach when it comes to defining the aesthetic properties of gesture-images. The third section examines the principles of Flusser’s philosophy of gestures, focusing on the semantic field of attunement and its connection with various elements related to photography, gesture, moods and affects. The question of gesture in photography—both the gestures of the photographed and those of the photographer—can be articulated with the notions of attunement and atmosphere, which go beyond semiological, psychological and communicational approaches and are important for our understanding of aesthetic and artistic experiences. Finally, if photography is a privileged way of studying atmospheres and gestures (as suggested by the Benjaminian notion of optical unconscious) and their connection with the inner life of the subject, in relation to urban space this study often acquires an intersubjective, social and political dimension, as in Jeff Wall’s Mimic. (shrink)

In his paper ‘Gesture Projection and Cosuppositions,’ Philippe Schlenker argues that co-verbal gestures convey not at-issue content by default and in particular, that they trigger conditional presuppositions. In this commentary, I take issue with both of these claims. Conditional presuppositions do not supply a systematic means for capturing the semantic contribution of a co-verbal gesture. Some gestures appear to contribute content inside of a negation when their associated speech content is likewise embedded; in other cases, co-verbal gestures arguably (...) contribute unconditional content to the global level. When this happens, we can infer what might look like a conditional presupposition, but this inference follows naturally from general principles already at work in purely verbal discourse and does not justify the claim that gesture content is contributed to a conditional presupposition. Problems exposed in the discussion of conditional presuppositions show that we are not yet in a position to make a general claim about the at-issue status of co-verbal gestures. (shrink)

In previous analyses of the influence of language on cognition, speech has been the main channel examined. In studies conducted among Yucatec Mayas, efforts to determine the preferred frame of reference in use in this community have failed to reach an agreement (Bohnemeyer & Stolz, 2006; Levinson, 2003 vs. Le Guen, 2006, 2009). This paper argues for a multimodal analysis of language that encompasses gesture as well as speech, and shows that the preferred frame of reference in Yucatec Maya (...) is only detectable through the analysis of co-speech gesture and not through speech alone. A series of experiments compares knowledge of the semantics of spatial terms, performance on nonlinguistic tasks and gestures produced by men and women. The results show a striking gender difference in the knowledge of the semantics of spatial terms, but an equal preference for a geocentric frame of reference in nonverbal tasks. In a localization task, participants used a variety of strategies in their speech, but they all exhibited a systematic preference for a geocentric frame of reference in their gestures. (shrink)

This paper pursues the idea that human and non-human great apes share a common set of directive (imperative) gestures and their meanings. We investigate gestures that are multifunctional, in that they have different effects in different contexts, focusing on non-human ape gestures that communicate “Stop that” in some contexts, and “Move away” in others. What may superficially appear to be lexical ambiguity can be derived from a single abstract lexical entry, “Not X!”, concluded to be a candidate for a universal (...) building block of meaning, shared by human and non-human great apes, reflections of which may also be found in the pragmatic gestures in humans. (shrink)

The last decades have seen major advances in the study of gestures both in humans and non-human primates. In this paper, we seriously examine the idea that there may be gestural form types that are shared across great ape species, including humans, which may underlie gestural universals, both in form and meaning. We focus on one case study, the hand fling gesture common to chimpanzees and humans, and provide a semantic analysis of this gesture.

The neuroscience of dance is a vibrant, fast growing field which embodies the promise of a genuine and productive interdisciplinary rapprochement between neuroscience and art. The strength of this field lies in the way it ties the experience of dance to sensorimotor processes that underwrite our ordinary perceptual engagement with the environment. Motor simulation and mimicry enhance our capacity to interpret the goals, motives, and emotions of others. Recent studies demonstrate that these same processes enable us to recognize abstract dance (...) movements as intentional actions with emotionally expressive and semantic content constitutive of their meaning. In what follows I examine this research with an eye to the potential contribution neuroscience can make to our understanding of dance as an artform. (shrink)

Silent gestures consist of complex multi‐articulatory movements but are now primarily studied through categorical coding of the referential gesture content. The relation of categorical linguistic content with continuous kinematics is therefore poorly understood. Here, we reanalyzed the video data from a gestural evolution experiment (Motamedi, Schouwstra, Smith, Culbertson, & Kirby, 2019), which showed increases in the systematicity of gesture content over time. We applied computer vision techniques to quantify the kinematics of the original data. Our kinematic analyses demonstrated (...) that gestures become more efficient and less complex in their kinematics over generations of learners. We further detect the systematicity of gesture form on the level of thegesture kinematic interrelations, which directly scales with the systematicity obtained on semantic coding of the gestures. Thus, from continuous kinematics alone, we can tap into linguistic aspects that were previously only approachable through categorical coding of meaning. Finally, going beyond issues of systematicity, we show how unique gesture kinematic dialects emerged over generations as isolated chains of participants gradually diverged over iterations from other chains. We, thereby, conclude that gestures can come to embody the linguistic system at the level of interrelationships between communicative tokens, which should calibrate our theories about form and linguistic content. (shrink)

This paper addresses the question of how emojis are integrated into the text that they occur with. I use the typology of gestural iconic enrichments proposed by Schlenker (2018a, 2018b) to investigate the hypothesis that emojis denoting objects (e.g., ????) and activities (e.g., ????) project (i.e., interact with logical operators) when co-occurring with text in a similar way as gestures do with speech. In particular, I claim that [i.] emojis generate co-suppositions, i.e., assertion-dependent presuppositions, when immediately following text (e.g., the (...) sentence “John didn’t train today ????” gives rise to the inference that “if John had trained today, weightlifting would have been involved”); [ii.] emojis generate supplements (in the sense of Potts, 2005), just like appositive relative clauses, when they are separated from the accompanying text by a pause (e.g., the sentence “John trained today...????” gives rise to the inference that “John trained today, which involved weightlifting”); [iii.] when used as replacements for words (e.g., “Yesterday, John didn’t ???? ”), emojis have an at-issue semantics and can trigger standard presuppositions. (shrink)

Jody Azzouni argues that we involuntarily experience certain physical items, certain products of human actions, and certain human actions themselves as having meaning-properties. We understand these items as possessing meaning or as having truth values. For example, a sign on a door reading "Drinks Inside" strikes native English speakers as referring to liquids in the room behind the door. The sign has a truth value--if no drinks are found in the room, the sign is misleading. Someone pointing in a direction (...) has the same effect: we experience her gesture as significant. Azzouni does not suggest that we don't recognize the expectations or intentions of speakers ; we do recognize that the person pointing in a certain direction intends for us to understand her gesture's significance. Nevertheless, Azzouni asserts that we experience that gesture as having significance independent of her intentions. The gesture is meaningful on its own. The same is true of language, both spoken and written. We experience the meanings of language artifacts as independent of their makers' intentions in the same way that we experience an object's shape as a property independent of the object's color. There is a distinctive phenomenology to the experience of understanding language, and Semantic Perception shows how this phenomenology can be brought to bear as evidence for and against competing theories of language. (shrink)

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic (...) expressions are addressed: proximal (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance. (shrink)

The article seeks a rapprochement between pragmatic and semantic theories of language by returning to a breaking point in the history of philosophy, the middle of the twentieth century, when these theoretical models began to evolve into distinct schools of thought. Philosophical accounts of this period explore various and intertwined dependencies between semantics and context; however, they only implicitly examine the potential of sounds and bodily gestures in bringing descriptive clarity to the modes and limits of such dependencies. The (...) article first investigates the way W. V. Quine conceptualized linguistic reference by combining behavioral models of language acquisition with more systematic explorations of syntax. It then turns to P. F. Strawson’s revisionary accounts of Kantian philosophy which reassess the silent strains of empiricism in Kant’s framework in order to identify possible grounds of reconciliation between pragmatic and semantic theories of language. Sound and gesture, two aspects that supplement language and give it an embodied feeling, are suggested as possible devices for formalizing intersections between context and semantic-oriented approaches. Poems by Wallace Stevens are used to think about the endurance of reference and how this endurance can find more compelling demonstration through an investigation of language as an embodied phenomenon. (shrink)

Sighted speakers of different languages vary systematically in how they package and order components of a motion event in speech. These differences influence how semantic elements are organized in gesture, but only when those gestures are produced with speech, not without speech. We ask whether the cross-linguistic similarity in silent gesture is driven by the visuospatial structure of the event. We compared 40 congenitally blind adult native speakers of English or Turkish to 80 sighted adult speakers as they (...) described three-dimensional motion scenes. We found an effect of language on co-speech gesture, not on silent gesture—blind speakers of both languages organized their silent gestures as sighted speakers do. Humans may have a natural semantic organization that they impose on events when conveying them in gesture without language—an organization that relies on neither visuospatial cues nor language structure. (shrink)

This paper explores the use of Laban’s effort actions from the field of dance and drama as a means to document user responses to physical product interaction. A range of traditional and modern product pairs were identified and reviewed in two workshops, where participants were asked to discuss and complete worksheets on their emotional response. The results provide qualitative feedback on their reactions to the different movements, and form the beginnings of an ‘emotional vocabulary’ that we plan to use in (...) the development of semantic differentials for future studies. Key factors in emotional response to gesture have been identified, including aligning movement to product function, emotional conflicts in movement, and user readiness and framing a sequence of movements. Keywords: Design research; gesture; human factors; interface design. (shrink)

This paper explores the use of Laban’s effort actions from the field of dance and drama as a means to document user responses to physical product interaction. A range of traditional and modern product pairs were identified and reviewed in two workshops, where participants were asked to discuss and complete worksheets on their emotional response. The results provide qualitative feedback on their reactions to the different movements, and form the beginnings of an ‘emotional vocabulary’ that we plan to use in (...) the development of semantic differentials for future studies. Key factors in emotional response to gesture have been identified, including aligning movement to product function, emotional conflicts in movement, and user readiness and framing a sequence of movements. Keywords: Design research; gesture; human factors; interface design. (shrink)

This paper points out that a major shift of paradigm is currently going on in the study of the human face and it seeks to articulate and to develop the fundamental assumptions underlying this shift. The main theses of the paper are: 1) Facial expressions can convey meanings comparable to the meanings of verbal utterances. 2) Semantic analysis (whether of verbal utterances or of facial expressions) must distinguish between the context-independent invariant and its contextual interpretations. 3) Certain components of facial (...) behavior (¿facial gestures¿) do have constant context-independent meanings. 4) The meanings of facial components and configurations of components have an inherent first-person and present tense orientation. 5) The basis for the interpretation of facial gestures is, above all, experiential. 6) The meanings of some facial expressions are universally intelligible and can be interpreted without reference to any local conventions. 7) To be fruitful, the semantic analysis of facial expressions needs a methodology. This can be derived from the methodological experience of linguistic semantics. The author illustrates and supports these theses by analyzing a range of universally interpretable facial expressions such as the following ones: ¿brow furrowed¿ (i.e. eyebrows drawn together); eyebrows raised; eyes wide open; corners of the mouth raised; corners of the mouth lowered; mouth open (while not speaking); lips pressed together; upper lip and nose ¿raised¿ (and, consequently, nose wrinkled). (shrink)

Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non‐native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye‐tracking to investigate whether and how native and highly proficient non‐native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear or (...) degraded (6‐band noise‐vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued‐recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non‐native listeners mostly gazed at the face during comprehension, but non‐native listeners gazed more often at gestures than native listeners. However, only native but not non‐native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non‐native listeners might gaze at gesture more as it might be more challenging for non‐native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non‐native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non‐native listeners. (shrink)

Many human languages have complex grammatical machinery devoted to temporality, but very little is known about how this came about. This paper investigates how people convey temporal information when they cannot use any conventional languages they know. In a laboratory experiment, adult participants were asked to convey information about simple events taking place at a given time, in spoken language and in silent gesture. It was shown that in spoken language, participants formed utterances according to the rules of their (...) native language, but in silent gesture, the temporal information was presented initially, and structurally separately, from the other information in the utterance. The experimental results are consistent with findings from natural systems emerging in situations of communicative stress: unsupervised adult second language learning and homesign. This confirms that presenting temporal information separately and initially is a robust strategy to talk about past and future when only sparse communicative means are available. (shrink)

This paper points out that a major shift of paradigm is currently going on in the study of the human face and it seeks to articulate and to develop the fundamental assumptions underlying this shift. The main theses of the paper are: 1) Facial expressions can convey meanings comparable to the meanings of verbal utterances. 2) Semantic analysis must distinguish between the context-independent invariant and its contextual interpretations. 3) Certain components of facial behavior do have constant context-independent meanings. 4) The (...) meanings of facial components and configurations of components have an inherent first-person and present tense orientation. 5) The basis for the interpretation of facial gestures is, above all, experiential. 6) The meanings of some facial expressions are universally intelligible and can be interpreted without reference to any local conventions. 7) To be fruitful, the semantic analysis of facial expressions needs a methodology. This can be derived from the methodological experience of linguistic semantics. The author illustrates and supports these theses by analyzing a range of universally interpretable facial expressions such as the following ones: “brow furrowed” ; eyebrows raised; eyes wide open; corners of the mouth raised; corners of the mouth lowered; mouth open ; lips pressed together; upper lip and nose “raised”. (shrink)

The present study claims that the use of spontaneous gestures can contribute to accomplishing coherence in sequential conversational exchanges. Three types of speech-accompanying gestures were analyzed in Chinese conversation. Based on the syntactic-semantic relation with the co-occurring utterance, the first type is associated with words that do not convey explicit meaning; the second type is associated with covert constituents; the third type does not have any linguistic affiliates. They suggest different ways in which gesture adds information to the propositional (...) content of the associated utterance or to the content of the speech event under discussion. They also play a role in achieving conversational coherence: on the part of the speaker, it is only by using the various types of gesture that a current speaker can make the message to be conveyed in the utterance complete. On the part of the next speaker, the recognition and negotiation with the former speaker of the gestured information is a way to acknowledge or confirm mutual knowledge. The collaborative efforts contribute to the coherence of the talk in the subsequent interaction. (shrink)

Past research has sought to elucidate how speakers and addressees establish common ground in conversation, yet few studies have focused on how visual cues such as co-speech gestures contribute to this process. Likewise, the effect of cognitive constraints on multimodal grounding remains to be established. This study addresses the relationship between the verbal and gestural modalities during grounding in referential communication. We report data from a collaborative task where repeated references were elicited, and a time constraint was imposed to increase (...) cognitive load. Our results reveal no differential effects of repetition or cognitive load on the semantic-based gesture rate, suggesting that representational gestures and speech are closely coordinated during grounding. However, gestures and speech differed in their execution, especially under time pressure. We argue that speech and gesture are two complementary streams that might be planned in conjunction but that unfold independently in later stages of language production, with speakers emphasizing the form of their gestures, but not of their words, to better meet the goals of the collaborative task. (shrink)

We argue that the pictorial nature of certain constructions in signs and in gestures explains surprising properties of their syntax. In several sign languages, the standard word order (e.g. SVO) gets turned into SOV (with preverbal arguments) when the predicate is a classifier, a distinguished construction with highly iconic properties (e.g. Pavlič, 2016). In silent gestures, participants also prefer an SOV order in extensional constructions, irrespective of the word order of the language they speak (Goldin-Meadow et al., 2008). But in (...) silent gestures and in Brazilian Sign Language (Libras), intensional constructions can override these SOV preferences, yielding SVO instead (Schouwstra & de Swart, 2014; Napoli et al., 2017). This distinction was argued to be due to iconicity: arguments are expressed before the verb if they correspond to entities that are present before the action, otherwise they follow the verb. While agreeing with this intuition, we propose that the extensional/intensional distinction is neither empirically nor theoretically appropriate. In new data from American Sign Language, we replicate the distinction among extensional classifier predicates: for _x ate up the ball_, the ball is typically seen before the eating and a preposed object is preferred; but for _x spit out the ball_, the ball is typically seen after the spitting and a postposed object is preferred, although both _eat up_ and _spit out_ are used extensionally. We extend this finding to data involving pro-speech (= speech-replacing) gestures embedded in French sentences. We argue for a Visibility Generalization according to which arguments appear before the verb if their denotations are typically visible before the action, and we develop a new formal account within a pictorial semantics for visual animations (inspired by Greenberg and Abusch). It derives the observed word order preferences, it explains how the semantics of classifier predicates combines iconic and conventional properties, and it makes a more general point: sign language semantics combines logical semantics with pictorial semantics. (shrink)

For humans, the ability to communicate and use language is instantiated not only in the vocal modality but also in the visual modality. The main examples of this are sign languages and gestures. Sign languages, the natural languages of Deaf communities, use systematic and conventionalized movements of the hands, face, and body for linguistic expression. Co-speech gestures, though non-linguistic, are produced in tight semantic and temporal integration with speech and constitute an integral part of language together with speech. The articles (...) in this issue explore and document how gestures and sign languages are similar or different and how communicative expression in the visual modality can change from being gestural to grammatical in nature through processes of conventionalization. As such, this issue contributes to our understanding of how the visual modality shapes language and the emergence of linguistic structure in newly developing systems. Studying the relationship between signs and gestures provides a new window onto the human ability to recruit multiple levels of representation in the service of using or creating conventionalized communicative systems. (shrink)

It is argued that none of the speaker's referential intentions accompanying his utterance of a demonstrative are semantically significant but rather the associated demonstration (or some other source of salience). It is constitutive of the speaker's having the specifically referential intention - held by Kent Bach to be semantically significant - that the speaker is taking, and relying upon, his accompanying gesture (or some other source of salience) as semantically significant, making it the case that this intention is not (...) even partly semantically significant. The same is then shown to hold for the speaker's remaining referential intentions: his intention aimed at a perceived object, believed by David Kaplan to be semantically significant, as well as the intention to refer to the object that he has in mind. (shrink)

Deictic (or pointing) gestures are traditionally known to have a simple function: to supply something as the referent of a demonstrative linguistic expression. I argue that deixis can have a more complex function. A deictic gesture can be used to _say something_ in conversation and can thereby become a full discourse move in its own right. To capture this phenomenon, which I call _rich demonstration_, I present an update semantics on which deictic gestures can indicate situations from a (...) conversation’s context and those situations coherently connect to prior discourse to generate information. (shrink)

Word learning is basic to foreign language acquisition, however time consuming and not always successful. Empirical studies have shown that traditional (visual) word learning can be enhanced by gestures. The gesture benefit has been attributed to depth of encoding. Gestures can lead to depth of encoding because they trigger semantic processing and sensorimotor enrichment of the novel word. However, the neural underpinning of depth of encoding is still unclear. Here, we combined an fMRI and a behavioral study to investigate (...) word encoding online. In the scanner, participants encoded 30 novel words of an artificial language created for experimental purposes and their translation into the subjects’ native language. Participants encoded the words three times visually, audiovisually, and by additionally observing semantically related gestures performed by an actress. Hemodynamic activity during word encoding revealed the recruitment of cortical areas involved in stimulus processing. In this study, depth of encoding can be spelt out in terms of sensorimotor brain networks that grow larger the more sensory modalities are linked to the novel word. Word retention outside the scanner documented a positive effect of gestures in a free recall test in the short-term. (shrink)

As part of a recent attempt to extend the methods of formal semantics beyond language, it has been claimed that music has an abstract truth-conditional semantics, albeit one that has more in common with iconic semantics than with standard compositional semantics. After summarizing this approach and addressing a common objection, we argue that music semantics should be enriched in three directions by incorporating insights of other areas of Super Semantics. First, it has been claimed (...) by Abusch 2013 that visual narratives make use of discourse referents akin to those we find in language. We argue that a similar conclusion extends to music, and we highlight it by investigating ways in which orchestration and dance may make cross-referential dependencies more explicit. Second, we show that by bringing music semantics closer to the semantics of visual narratives, we can give an account of the semantics of mixed visual and musical sequences. Third, it has been claimed that co-speech gestures trigger characteristic conditionalized presuppositions, called ‘cosuppositions’, and that their semantic status derives from their parasitic character relative to words. We argue that the same conclusion extends to some instances of film and cartoon music: it may trigger cosuppositions that can be revealed by embedding film excerpts or gifs in sentences so as to test presupposition projection. We further argue that under special discourse conditions, pro-speech gestures and pro-speech music alike can trigger cosuppositions as well. These results suggest that new insights can be gained not just by extending the methods of semantics to new objects, but also by drawing new connections among them. (shrink)

Much evidence suggests that semantic characteristics of a message (e.g., the extent to which the message evokes thoughts of spatial or motor properties) and social characteristics of a speaking situation (e.g., whether there is a listener who can see the speaker) both influence how much speakers gesture. However, the Gesture as Simulated Action (GSA) framework (Hostetter & Alibali, ) predicts that these effects should not be independent but should interact such that the effect of visibility is lessened when (...) a message evokes strong thoughts of action. This study tested this claim by comparing the gesture rates produced by speakers as they described 24 nouns that vary in how strongly they evoke thoughts of action. Further, half of the words were described with visibility between speaker and listener blocked. The results demonstrated a significant interaction as predicted by the GSA framework. (shrink)