I argue that emojis are essentially little pictures, rather than words, gestures, expressives, or diagrams. ???? means that the world looks like that, from some viewpoint. I flesh out a pictorial semantics in terms of geometric projection with abstraction and stylization. Since such a semantics delivers only very minimal contents I add an account of pragmatic enrichment, driven by coherence and nonliteral interpretation. The apparent semantic distinction between emojis depicting entities (like ????) and those depicting facial expressions (like ????) I (...) analyze as a difference between truth-conditional and use-conditional pictorial content: ???? depicts what the world of evaluation looks like, while ???? depicts what the utterance context looks like. (shrink)

This introduction aims to familiarize readers with basic dimensions of variation among pictorial and diagrammatic representations, as we understand them, in order to serve as a backdrop to the articles in this volume. Instead of trying to canvas the vast range of representational kinds, we focus on a few important axes of difference, and a small handful of illustrative examples. We begin in Section 1 with background: the distinction between pictures and diagrams, the concept of systems of representation, and that (...) of the properties of usage associated with signs. In Section 2 we illustrate these ideas with a case study of diagrammatic representation: the evolution from Euler diagrams to Venn diagrams. Section 3 is correspondingly devoted to pictorial representation, illustrated by the comparison between parallel and linear perspective drawing. We conclude with open questions, and then briefly summarize the articles to follow. (shrink)

In demonstration, speakers use real-world activity both for its practical effects and to help make their points. The demonstrations of origami mathematics, for example, reconfigure pieces of paper by folding, while simultaneously allowing their author to signal geometric inferences. Demonstration challenges us to explain how practical actions can get such precise significance and how this meaning compares with that of other representations. In this paper, we propose an explanation inspired by David Lewis’s characterizations of coordination and scorekeeping in conversation. In (...) particular, we argue that words, gestures, diagrams and demonstrations can function together as integrated ensembles that contribute to conversation, because interlocutors use them in parallel ways to coordinate updates to the conversational record. (shrink)

This study examines the form and function of gestural depictions that develop over extended stretches of concept explanation by a philosopher. Building onStreeck’s (2009)explorations of depiction by gesture, we examine how this speaker’s process of exposition involves sequences of multimodal, analogical depiction by which the philosophical concepts are not only expressed through gesture forms, but also dynamically analyzed and construed through gestural activity. Drawing on perspectives of gesture as active meaning making (Müller 2014,2016,Streeck 2009), we argue that the build-up of (...) gestures in depiction sequences, activated through a multimodal metaphor (Müller & Cienki 2009), engages the wider philosophical standpoint of the speaker. Using video analysis supported by interview data, we demonstrate how examination of gestures within and across discourse can lead to understanding of how dynamic, embodied, and subjective processes of conceptualization contribute to philosophical theorizing. (shrink)

As part of a recent attempt to extend the methods of formal semantics beyond language, it has been claimed that music has an abstract truth-conditional semantics, albeit one that has more in common with iconic semantics than with standard compositional semantics. After summarizing this approach and addressing a common objection, we argue that music semantics should be enriched in three directions by incorporating insights of other areas of Super Semantics. First, it has been claimed by Abusch 2013 that visual narratives (...) make use of discourse referents akin to those we find in language. We argue that a similar conclusion extends to music, and we highlight it by investigating ways in which orchestration and dance may make cross-referential dependencies more explicit. Second, we show that by bringing music semantics closer to the semantics of visual narratives, we can give an account of the semantics of mixed visual and musical sequences. Third, it has been claimed that co-speech gestures trigger characteristic conditionalized presuppositions, called ‘cosuppositions’, and that their semantic status derives from their parasitic character relative to words. We argue that the same conclusion extends to some instances of film and cartoon music: it may trigger cosuppositions that can be revealed by embedding film excerpts or gifs in sentences so as to test presupposition projection. We further argue that under special discourse conditions, pro-speech gestures and pro-speech music alike can trigger cosuppositions as well. These results suggest that new insights can be gained not just by extending the methods of semantics to new objects, but also by drawing new connections among them. (shrink)

In dynamic theories of presupposition, a trigger pp′ with presupposition p and at-issue component p′ comes with a requirement that p should be entailed by the local context of pp′. We argue that some co-speech gestures should be analyzed within a presuppositional framework, but with a twist: an expression p co-occurring with a co-speech gesture G with content g comes with the requirement that the local context of p should guarantee that p entails g; we call such assertion-dependent presuppositions ‘cosuppositions’. (...) We show that this analysis can be combined with earlier theories of local contexts to account for complex patterns of gesture projection in quantified and in attitudinal contexts, and we compare our account to two potential alternatives: one based on supervaluations, and one, due to Cornelia Ebert, that treats co-speech gestures as supplements. We argue that the latter is correct, but for ‘post-speech’ gestures, rather than for co-speech gestures. (shrink)

There is a widely held view that visual representations (images) do not depict negation, for example, as expressed by the sentence, “the train is not coming.” The present study focuses on the real-world visual representations of photographs and comic (manga) illustrations and empirically challenges the question of whether humans and machines, that is, modern deep neural networks, can recognize visual representations as expressing negation. By collecting data on the captions humans gave to images and analyzing the occurrences of negation phrases, (...) we show some evidence that humans recognize certain images as expressing negation. Furthermore, based on this finding, we examined whether or not humans and machines can classify novel images as expressing negation. The humans were able to correctly classify images to some extent, as expected from the analysis of the image captions. On the other hand, the machine learning model of image processing was only able to perform this classification at about the chance level, not at the same level of performance as the human. Based on these results, we discuss what makes humans capable of recognizing negation in visual representations, highlighting the role of the background commonsense knowledge that humans can exploit. Comparing human and machine learning performances suggests new ways to understand human cognitive abilities and to build artificial intelligence systems with more human-like abilities to understand logical concepts. (shrink)

In this paper, I argue that an important component of the language-ready brain is the ability to recognize and conceptualize events. By ‘event’, I mean any situation or activity in the world or our mental life, that we find salient enough to individuate as a thought or word. While this may sound either trivial or non-unique to humans, I hope to show that abstracting away events and their participants from the embodied flow of experience is a characteristic unique to humans. (...) This ability is enabled, I will argue, by two critical competencies that act as scaffolds for language-ready thought in the prehuman brain. The first, as argued by Arbib (2006,2012,2016) and others, is a sophisticated system of gesture production and understanding in prehumans, which provided a template for schema-like sequencing and slot-filling of information units. The second involves the integration of multiple modalities of expression in the communicative act, in particular, the alignment of co-gestural speech and co-speech gesture. With such computational facilities, action-based gestures can be abstracted away from their associated objects and become full event representations. This view supports the MSH argument for the emergence of more complex linguistic expressions from initially holophrastic units. In particular, actions can be thought of as protoverbs, which through this process are abstracted to full event descriptions, i.e., verbs. (shrink)

In this paper, I discuss some of Maximilian de Gaynesford’s arguments regarding indexicals. Although I agree with his treatment of the first singular personal pronoun as a prototype of demonstrative expressions, I challenge his refusal to treat indexicals as complex demonstratives. To offer an alternative to this refusal I try to develop a common ground from different theories that consider indexicals as linguistic constructions that embed a nonlinguistic element, following an original idea in Frege’s latest writings. These views form the (...) backdrop on which we can put forward the claim of treating all indexicals as complex demonstratives. In the central part of the paper, I criticize each of de Gaynesford’s arguments against the reduction of indexicals to complex demonstratives. Besides, I propose a new definition of the concept of “demonstration” as a nonlinguistic feature of all indexicals in their referential uses, to contrast de Gaynesford’s rejection of the idea that demonstrations are an essential feature of indexicals. Eventually, I strengthen my claim by distinguishing indexicals from proper names and definite descriptions on the ground that only perceptual indexicals necessarily require an accompanying demonstration. However, the main point of the paper is a negative one, that is the rejection of de Gaynesford’s arguments against the reduction of indexicals to complex demonstratives. More work is needed to reach a positive conclusion on this topic. (shrink)

We argue that formal linguistic theory, properly extended, can provide a unifying framework for diverse phenomena beyond traditional linguistic objects. We display applications to pictorial meanings, visual narratives, music, dance, animal communication, and, more abstractly, to logical and non-logical concepts in the ‘language of thought’ and reasoning. In many of these cases, a careful analysis reveals that classic linguistic notions are pervasive across these domains, such as for instance the constituency (or grouping) core principle of syntax, the use of logical (...) variables (for object tracking), or the variety of inference types investigated in semantics/pragmatics. The aim of this overview is to show how the application of formal linguistic concepts and methodology to non-linguistic objects yields non-trivial insights, thus opening the possibility of a general, precise theory of signs. (An appendix, found in the online supplements to this article, surveys applications of Super Linguistics to animal communication.). (shrink)

We argue that formal linguistic theory, properly extended, can provide a unifying framework for diverse phenomena beyond traditional linguistic objects. We display applications to pictorial meanings, visual narratives, music, dance, animal communication, and, more abstractly, to logical and non-logical concepts in the ‘language of thought’ and reasoning. In many of these cases, a careful analysis reveals that classic linguistic notions are pervasive across these domains, such as for instance the constituency (or grouping) core principle of syntax, the use of logical (...) variables (for object tracking), or the variety of inference types investigated in semantics/pragmatics. The aim of this overview is to show how the application of formal linguistic concepts and methodology to non-linguistic objects yields non-trivial insights, thus opening the possibility of a general, precise theory of signs. (An appendix, found in the online supplements to this article, surveys applications of Super Linguistics to animal communication.). (shrink)

Stojnić et al. (2013, 2017) argue that the reference of demonstratives is fixed without any contribution from the extra-linguistic context. On their `prominence/coherence' theory, the reference of a demonstrative expression depends only on its context-independent linguistic meaning. Here, we argue that Stojnić et al.’s striking claims can be maintained in only the thinnest technical sense. Instead of eliminating appeals to the extra-linguistic context, we show how the prominence/coherence theory merely suppresses them. Then we ask why one might be tempted to (...) try and offer such a view. Since we are rather sympathetic to the motivations we find, we close by sketching a more plausible alternative. (shrink)

According to what is perhaps the dominant picture of reference, what a referential term refers to in a context is determined by what the speaker intends for her audience to identify as the referent. I argue that this sort of broadly Gricean view entails, counterintuitively, that it is impossible to knowingly use referential terms in ways that one expects or intends to be misunderstood. Then I sketch an alternative which can better account for such opaque uses of language, or what (...) I call “sneaky reference.” I close by reflecting on the ramifications of these arguments for the theory of meaning more broadly. (shrink)

In his paper ‘Gesture Projection and Cosuppositions,’ Philippe Schlenker argues that co-verbal gestures convey not at-issue content by default and in particular, that they trigger conditional presuppositions. In this commentary, I take issue with both of these claims. Conditional presuppositions do not supply a systematic means for capturing the semantic contribution of a co-verbal gesture. Some gestures appear to contribute content inside of a negation when their associated speech content is likewise embedded; in other cases, co-verbal gestures arguably contribute unconditional (...) content to the global level. When this happens, we can infer what might look like a conditional presupposition, but this inference follows naturally from general principles already at work in purely verbal discourse and does not justify the claim that gesture content is contributed to a conditional presupposition. Problems exposed in the discussion of conditional presuppositions show that we are not yet in a position to make a general claim about the at-issue status of co-verbal gestures. (shrink)

This paper presents an analysis of face emoji (disc-shaped pictograms with stylized facial expressions) that accompany written text. We propose that there is a use of face emoji in which they comment on a target proposition expressed by the accompanying text, as opposed to making an independent contribution to discourse. Focusing on positively valenced and negatively valenced emoji (which we gloss as _happy_ and _unhappy_, respectively), we argue that the emoji comment on how the target proposition bears on a contextually (...) provided discourse value endorsed by the author. Discourse values embody what an author desires, aspires to, wishes for, or hopes for. Our analysis derives a range of non-trivial generalizations, including (i) ordering restrictions with regards to the placement of emoji and text, (ii) cases of apparent mixed emotions, and (iii) cases where the lexical content of the accompanying text influences the acceptability of a face emoji. (shrink)

Lexical innovations (e.g., zero-derivations coined on the fly by a speaker) seem to bear semantic content. Yet, such expressions cannot bear semantic content as a function of the conventions of meaning in force in the language, since they are not part of its lexicon. This is in tension with the commonplace view that the semantic content of lexical expressions is constituted by linguistic conventions. The conventionalist has two immediate ways out of the tension. The first is to preserve the conventionalist (...) assumption and deny that lexical innovations bear semantic content. The second is to dynamicize the conventionalist assumption, that is, argue that presentations of unattested expressions trigger an augmentation of the standing semantic resources of the language and instantiate content as a result of this underlying update. Building on a comparison with the production of novel onomatopoeic words, iconic pseudowords and pro-speech gestures, the paper argues that the issue is best addressed by suspending the conventionalist assumption, and describes the metasemantic implications of the claim. (shrink)

Deictic (or pointing) gestures are traditionally known to have a simple function: to supply something as the referent of a demonstrative linguistic expression. I argue that deixis can have a more complex function. A deictic gesture can be used to _say something_ in conversation and can thereby become a full discourse move in its own right. To capture this phenomenon, which I call _rich demonstration_, I present an update semantics on which deictic gestures can indicate situations from a conversation’s context (...) and those situations coherently connect to prior discourse to generate information. (shrink)

Humans communicate with different modalities. We offer an account of multi-modal meaning coordination, taking speech-gesture meaning coordination as a prototypical case. We argue that temporal synchrony (plus prosody) does not determine how to coordinate speech meaning and gesture meaning. Challenging cases are asynchrony and broadcasting cases, which are illustrated with empirical data. We propose that a process algebra account satisfies the desiderata. It models gesture and speech as independent but concurrent processes that can communicate flexibly with each other and exchange (...) the same information more than once. The account utilizes the ψ-calculus, allowing for agents, input-output-channels, concurrent processes, and data transport of typed λ-terms. A multi-modal meaning is produced integrating speech meaning and gesture meaning into one semantic package. Two cases of meaning coordination are handled in some detail: the asynchrony between gesture and speech, and the broadcasting of gesture meaning across several dialogue contributions. This account can be generalized to other cases of multi-modal meaning. (shrink)

This paper examines the interplay of semantics and pragmatics within the domain of film. Films are made up of individual shots strung together in sequences over time. Though each shot is disconnected from the next, combinations of shots still convey coherent stories that take place in continuous space and time. How is this possible? The semantic view of film holds that film coherence is achieved in part through a kind of film language, a set of conventions which govern the relationships (...) between shots. In this paper, we develop and defend a new version of the semantic view. We articulate it for a pair of conventions that govern spatial relations between viewpoints. One such rule is already well-known; sometimes called the "180° Rule," we term it the X-Constraint; to this we add a previously unrecorded rule, the T-Constraint. As we show, both have the effect, in different ways, of limiting the way that viewpoint can shift through space from shot to shot over the course of a film sequence. Such constraints, we contend, are analogous to relations of discourse coherence that are widely recognized in the linguistic domain. If film is to have a language, it is a language made up of rules like these. (shrink)

Paulina Sliwa (2015) argues that knowing why p is necessary and sufficient for understanding why p. She tries to rebut recent attacks against the necessity and sufficiency claims, and explains the gradability of understanding why in terms of knowledge. I argue that her attempts do not succeed, but I indicate more promising ways to defend reductionism about understanding why throughout the discussion.