Taste is a central sense for humans and animals, and it has been largely studied either from physiological and neurological approaches or from socio-cultural ones. This paper adopts another view, focused on the activity of tasting rather than on the sense of taste, approached within the perspective of ethnomethodology and multimodal conversation analysis. This view addresses the activity of tasting as it is interactionally organized in specific social settings, observed in a naturalistic way, on the basis of video recordings. (...) Focusing on video recorded improvised tastings of cheese in gourmet shop encounters, the paper offers a systematic analysis of the way in which tasting is orderly achieved in an intersubjective way. It follows the various steps characterizing tasting, from the invitation to taste, to the grasping of a bit to taste, which is put in the mouth, chewed, and swallowed; it details how an interactional moment offering the taster a priviledged, individual, focused space in which to devote exclusive attention to the object tasted is actively tailored by all parties. By contrast, the completion of tasting is marked by a return to mutual gaze, the animation of facial expressions and nods, and the final production of a judgment of taste. By offering a systematic reconstruction of how these tasting moments are organized, the paper invites to a multimodal approach of sensoriality in social interaction. (shrink)

When people are engaged in social interaction, they can repeat aspects of each other’s communicative behavior, such as words or gestures. This kind of behavioral alignment has been studied across a wide range of disciplines and has been accounted for by diverging theories. In this paper, we review various operationalizations of lexical and gestural alignment. We reveal that scholars have fundamentally different takes on when and how behavior is considered to be aligned, which makes it difficult to compare findings (...) and draw uniform conclusions. Furthermore, we show that scholars tend to focus on one particular dimension of alignment (traditionally, whether two instances of behavior overlap in form), while other dimensions remain understudied. This hampers theory testing and building, which requires a well‐defined account of the factors that are central to or might enhance alignment. To capture the complex nature of alignment, we identify five key dimensions to formalize the relationship between any pair of behavior: time, sequence, meaning, form, and modality. We show how assumptions regarding the underlying mechanism of alignment (placed along the continuum of priming vs. grounding) pattern together with operationalizations in terms of the five dimensions. This integrative framework can help researchers in the field of alignment and related phenomena (including behavior matching, mimicry, entrainment, and accommodation) to formulate their hypotheses and operationalizations in a more transparent and systematic manner. The framework also enables us to discover unexplored research avenues and derive new hypotheses regarding alignment. (shrink)

Everyday media consumption leaves no doubt that the perception of moving images from various media is characterized by experience and understanding. Corresponding research in this field has shown that the stimulus patterns flooding in on us are not only processed mentally, but also bodily. Building on this, the following study argues that incoming stimuli are processed not only visually, but multimodally, with all senses, and moreover affectively. The classical binding of a sensory organ to a medium, on whose delimitation the (...) common understanding of multimodality is based, is thus abolished. Can arguments for this assumption be found? Building on the research of cultural anthropologist Ernst Cassirer and his successors, the two American philosophers Susanne K. Langer and John M. Krois, and supported by research from the life sciences in which, in parallel, the arts also play an important role in understanding image formation, as in the work of Heinz Werner and, following on from this, in the more recent research of the child psychologist Daniel N. Stern in collaboration with mirror neuron research by the group of neuroscientists around Giacomo Rizzolatti in Parma, justifications for this will be provided. What the multimodal processing of film images is concretely based on will finally be made clear by means of an exemplary analysis of a video film by the Swiss artist Pipilotti Rist. It aims to prove, that the prerequisites for the thesis can only lie in the fact, that both the modes of perception (visual, tactile, auditory, olfactory, gustatory) as well as the medially conditioned modes of image formation are based on non-discursive, formal-abstract structures that are also processed affectively. At the same time, the analysis thus shows how the selection and composition of the vital experienced forms can provide information about the socio-culturally shaped self-image of the producer and the viewer. (shrink)

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, ), we focus on measuring the temporal patterns of (...) specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog. (shrink)

Journal Name: Cognitive Linguistics Issue: Ahead of print.

The framework of depicting put forward byClark (2016)offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: “speech-embedded nonverbal depictions,” namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples from American (...) TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance. (shrink)

This article examines the interaction between grammar and multimodal resources by analyzing reported speech in Korean multiparty face-to-face interaction. The operation of two relevancy rules — minimization and recognition in interaction — is examined together with how the absence or presence of grammar is complemented by multimodal resources of various sorts. For the analysis, three categories are posited depending on who the quoted character is in the talk. In quoting oneself or co-participants in the talk, (...) syntactic resources, prosody, sequential organization and embodied participant frameworks collaborate in the interactional management of the quote without overt realization of grammatical resources traditionally proposed for building quotations in Korean. The multimodal character of quoted speech is most weakly realized when quoting a non-present character, not only because of the limitation in body and spatial resources available in the local context to recover various referential terms, but more importantly, because the interactional project that the parties are engaged in makes a distancing device such as the quotative particle -ko relevant. This study provides evidence for the intricate interaction between multimodal resources and grammar through investigation of an important realm of human interaction, that of reported speech. (shrink)

The analysis of video-recorded interaction consists of various professionalized ways of seeing participant behavior through multimodal, co-operative, or intercorporeal lenses. While these perspectives are often adopted simultaneously, each creates a different view of the human body and interaction. Moreover, microanalysis is often produced through local practices of sense-making that involve the researchers’ bodies. It has not been fully elaborated by previous research how adopting these different ways of seeing human behavior influences both what is seen from a (...) video and how it is seen, as well as the way the interpretation of the data ultimately unfolds in the interaction between researchers. In this article, we provide a theoretical-methodological discussion of the microanalytic research process. We explore how it differs from “seeing” affect in interaction either as a co-operative and multimodal action or as an intercorporeal experience. First, we introduce the multimodal conversation analytic, co-operative, and intercorporeal approaches to microanalysis. Second, we apply and compare these practices to a video-recorded interaction of a romantic couple. Furthermore, we examine a video-recorded episode of us, the researchers, reflecting on our analytic observations about this interaction. We suggest that adopting a multimodal and co-operative perspective constructs affect as co-produced and displayed through observable action, while an intercorporeal perspective produces affect as an embodied and experienced phenomenon. While the former enables locating affect in a specific moment and identifiable body parts, the latter facilitates recognizing the experienced side of affect. These different modes of professional vision complement one another in capturing affect in interaction while being fundamentally used in local interactions between the researchers. (shrink)

This article starts with the environmental changes in human cognition, analyzes the virtual as the main feature of visual perception under digital technology, and explores the transition from passive to active human cognitive activities. With the diversified understanding of visual information, human contradiction of memory also began to become prominent. Aiming at the problem that the existing multimodal TV media recognition methods have low recognition rate of unknown application layer protocols, an adaptive clustering method for identifying unknown application layer (...) protocols is proposed. This method clusters application layer protocols based on similarity of the load characteristics of network stream application layer protocol data. The method divides the similarity calculation in the clustering algorithm to improve the clustering efficiency of the algorithm. Experimental results show that the proposed method can efficiently and accurately recognize unknown visual communication. This article proposes that, in the interactive multimodal visual information transmission, human visual perception experience has changed, the diversity of visual information content expression makes the aesthetic subject more personalized and stylized. (shrink)

Dans cette contribution, nous présentons la base CLAPI développée au laboratoire ICAR dans le contexte de l’évolution des bases de données de langues parlées en France au cours des trente dernières années. Nous détaillons les deux composantes de CLAPI, l’archive de corpus de langue parlée en interaction audio et vidéo enregistrés dans des situations sociales naturelles variées, et la plateforme d’outils.L’usage et l’apport de CLAPI sont illustrés par deux études. L’une décrit comment la base peut être utilisée pour des (...) travaux de linguistique interactionnelle intégrant la multimodalité (« oh là là ») ; l’autre concerne une recherche combinant données et métadonnées (« trop »). L’article est aussi l’occasion d’un bilan plus général. La mise en perspective montre en effet qu’après la période des questions est venue celle des dilemmes. La période des questions, choix et décisions à toutes sortes de niveaux a accompagné la mise en place des bases de données. L’expérience permet maintenant de mesurer leurs indéniables apports en termes non seulement de quantité de données disponibles (et traitables grâce aux outils), mais aussi de qualité (comme conséquence des exigences de standardisation liées au partage des données). La période des dilemmes nous conduit à nous interroger sur les meilleurs choix à opérer aujourd’hui dans les relations entre la poursuite des recherches sur des corpus variés (et parfois sensibles) et les exigences des bases de données ouvertes. (shrink)

This article investigates a multiparty interaction in an accounting office by applying a multimodal approach to discourse. This approach allows the incorporation of all relevant communicative modes and is based on the following three notions: 1) the notion of mediated action; 2) the notion of modal density; and 3) the notion of a foreground– background continuum of attention/awareness. The article illustrates that a social actor in a multiparty interaction simultaneously co-constructs several higher-level actions with the various participants (...) on different levels of their attention/awareness. On a theoretical level, the article argues that traditional approaches to discourse analysis, with their unconditional focus on language as the primary mode, misconstrue the multiple higher-level actions that a social actor is engaged in simultaneously as dyadic or triadic interactions in quick succession. (shrink)

Adult-infant early dyadic interactions have been extensively explored by developmental psychologists. Around the age of 2 months, infants already demonstrate complex, delicate and very sensitive behaviors that seem to express their ability to interact and share emotions with their caregivers. This paper presents 3 pilot studies of parent-infant dyadic interaction in various set-ups. The first two present longitudinal data collected on two infants aged between 1 and 6 months and their mothers. We analyzed the development of coordination between them, (...) at the motor and at the vocal level. The 3rd pilot study aims to explore interpersonal coordination in both vocal behavior and motor activity for one infant and his mother at 2, 4 and 6 months. These pilot studies however leave a number of questions open concerning developmental changes and infants’ progressive mastery of interaction. We identify areas worth examining and try to tease out specific issues that may help develop new methodological pathways for the study of early naturalistic social interaction. We assume that a continuous, rather than discrete, approach would better capture the changes taking place in the various communicative modalities, while also displaying each dyad’s specificity and the narrative dimension of social engagement between infants and caregivers. (shrink)

The aim of our study is to show the ways in which family members coordinate their minds, bodies and language in a functional and goal-oriented manner when they are jointly remembering shared events that they had experienced together as a group. So far, little attention has been paid to the influence that the interplay of multiple behavioral channels have in collaborative remembering in small groups. Our goal is to specifically examine the central role that direct questions have when they act (...) as embodied reminders through the interanimation of multiple behavioral channels in family interactions. The video data for analysis comes from an ongoing project on how collaborative remembering takes places among small groups of Argentinean Spanish speakers as each group recalls a vacation taken together several years ago. (shrink)

This article aims to describe the construction and annotation of a multimodal and multilingual (French, English, and Mandarin Chinese) corpus for the study of second language acquisition and professional development in language teaching. The corpus was built within a research project whose main objective is to determine which techno-semio-pedagogical competencies are developed informally and which ones require formal training. In this paper, we explain the theoretical framework adopted, characterized by an ecological approach to the interactive environment. We subsequently illustrate (...) the procedures for the collection of audio, video and eye-tracking data. We also provide details of the annotation of raw data to produce a corpus of analysis drawing on different automatic and semi-automatic annotation tools. Finally, we explain how such a corpus enables us to study the development of learners’ multimodal competence and tutors’ techno-pedagogical competence and point to some other possible research questions. (shrink)

This article analyses the sequential ordering of multi-modal discussions in real-time online classes in postgraduate education contexts. The article explores the ways that text and verbal talk are organized by the participants as inter-connecting modes of interaction. Focusing on Initiation, Response and Feedback sequences as an example of a form of exchange, the article shows that the interaction was comparatively disorderly where conducted across talk and text modes. For instance, written responses to questions or to encouragement turns often (...) overlapped with verbal answers from other students or with encouragement turns from the tutor. However, through the withholding of speech turns, the close latching of speech to written talk, and the ceasing of written turns at points where the topics or speakers changed, the participants showed a concern with re-establishing sequential order across the modalities. (shrink)

Irregular self-selection is a demonstration of active involvement in interaction. English as a foreign language learners’ talk-in-interaction is one of such cases. Yet, little research has explored when, how, and why learners implement this action. The aim of this article is to address these issues in Chinese postgraduate EFL learners’ conversations from the perspective of multimodal interaction. To this end, we provide descriptive statistics and use multimodal conversation analysis to investigate the detailed process of irregular (...) self-selection. The results show the interactional sensitivity of learners, and all the successful irregular self-selection can be divided into the following three types: turn interruption, turn competition TC), and turn holding abortion. Learners implement this action by using multimodal resources, including lexis, syntax, pitch reset, intensity enhancement, gaze, and so forth. However, their body movements lack diversity, causing behaviors to be constrained and inactive. The main purpose of irregular self-selection is to provide knowledge that contributes to topical development. This study reveals that Chinese postgraduate EFL learners are interactionally competent members. They are able to achieve communicative goals but in a low diversity of body movements. The findings help to understand the detailed process of speakership claiming in EFL learners’ conversations. (shrink)

The article investigates a multimodal practice for self-selecting observed in a video-taped corpus of work meetings: the use of pointing gestures predicting possible turn completions and projecting the emergence of possible next speakers. This practice is analyzed in various sequential positions, namely at turn beginnings and at pre-beginnings. It displays recipients' practical online turn parsing, and their orientation to transition spaces, and to TCU, completions in a visible, recognizable, public way. It shows the emergent and progressive establishment of speakership, (...) exploiting both systematic features of turn-taking and specific features of the interactional space. The article explores the emic, locally situated and contingent definition of speakership by considering not only where pointing gestures begin but also where they end: pointing does not only end in pre-completion positions, projecting turn end, but can also persist through sequences, showing speaker's orientations to her turn's sequential implicativeness. Thus, by observing a particular multimodal practice within a specific interactional setting, the article explores participants' orientations to the rights and obligations associated with talk-related categories such as `non current speaker', `possible next speaker', `incipient speaker' and `current speaker'. (shrink)

According to the shared signal hypothesis the impact of facial expressions on emotion processing partially depends on whether the gaze is directed toward or away from the observer. In autism spectrum disorder several aspects of face processing have been found to be atypical, including attention to eye gaze and the identification of emotional expressions. However, there is little research on how gaze direction affects emotional expression processing in typically developing individuals and in those with ASD. This question is investigated here (...) in two multimodal experiments. Experiment 1 required processing eye gaze direction while faces differed in emotional expression. Forty-seven children participated. Their Autism Diagnostic Observation Schedule scores ranged from 0 to 6 in the experiment. Event-related potentials were sensitive to gaze direction and emotion, but emotion processing did not depend on gaze direction. However, for angry faces the gaze direction effect on the N170 amplitude, as typically observed in TD individuals, diminished with increasing ADOS score. For neutral expressions this correlation was not significant. Experiment 2 required explicit emotion classifications in a facial emotion composite task while eye gaze was manipulated incidentally. A group of 22 children with ASD was compared to a propensity score-matched group of TD children. The same comparison was carried out for a subgroup of nine children with ASD who were less trained in social cognition, according to clinician’s report. The ASD group performed overall worse in emotion recognition than the TD group, independently of emotion or gaze direction. However, for disgust expressions, eye tracking data revealed that TD children fixated relatively longer on the eyes of the stimulus face with a direct gaze as compared with averted gaze. In children with ASD we observed no such modulation of fixation behavior as a function of gaze direction. Overall, the present findings from ERPs and eye tracking confirm the hypothesis of an impaired sensitivity to gaze direction in children with ASD or elevated autistic traits, at least for specific emotions. Therefore, we conclude that multimodal investigations of the interaction between emotional processing and stimulus gaze direction are promising to understand the characteristics of individuals differing along the autism trait dimension. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the (...) class='Hi'>multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

ABSTRACTThis article offers the first large-scale study of a multimodal corpus of 210 advertisements. First, the reader is presented with a description of the corpus in terms of the distribution of conceptual operations and use of modal cues. Subsequently, the weight of mode and marketing strategy to trigger more or less amounts of conceptual complexity is analyzed. This corpus-based survey is complemented with the qualitative analysis of three novel metaphor–metonymy interactions that stem from the data and that have not (...) yet been surveyed in multimodal use. The results show that metaphtonymy is the most frequet conceptual operation in the corpus; that there is a significant effect of the use of modes in the activation of different amounts of conceptual complexity; and that the type of advertised product and the marketing strategy has no significant effect on the number and complexity of conceptual mappings in the advert... (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) (...) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

Visual (facial), tactile, and gestural, as well as vocal, elements of mother-infant interactions are each formalizations, repetitions, exaggerations, and elaborations of ordinary adult communicative signals of affiliation – suggesting ritualization. They are temporally organized and enable emotional coordination of the interacting pair. This larger view of motherese supports Falk's claim that the social-emotional elements of language are primary and suggests that language and music have common evolutionary foundations.

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /, *, \, together with a package of structural postulates characterizing the resource management properties of the * connective. Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke (...) style interpretation in terms of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems each have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems.The first type is obtained by combining a number of unimodal systems into one multimodal logic. The combined multimodal logic is set up in such a way that the individual resource management properties of the constituting logics are preserved. But the inferential capacity of the mixed logic is greater than the sum of its component parts through the addition of interaction postulates, together with the corresponding interpretive constraints on frames, regulating the communication between the component logics.The second type of mixed system is obtained by generalizing the residuation scheme for binary connectives to families of n-ary connectives, and by putting together families, of different arities in one logic. We focus on residuation for unary connectives, hence on mixed frames, as these already represent the complexities in full. We prove a number of elementary logical results for unary families of residuated connectives and their combination with binary families. The existing proposals for unary ‘structural modalities’ are situated within this general framework, and a number of new linguistic applications is given. (shrink)

ContextBehavioral observation scales are important for understanding and assessing social skills. In the context of collaborative problem-solving skills, considered essential in the 21st century, there are no validated scales in French that can be adapted to different CPS tasks. The aim of this study is to adapt and validate, by annotating a new video corpus of dyadic interactions that we have collected, two observational scales allowing us to qualitatively assess CPS skills: the Social Performance Rating Scale and the Social Skills (...) of Collaboration Scale.MethodThe construct validity of these two scales was assessed by exploratory factor analysis and inter-item correlations. We also checked inter-judge agreement using inter-class correlation coefficients. Internal consistency was determined using Cronbach’s alpha and convergent and divergent validity by assessing correlations between the two scales and measures of depression and alexithymia. Finally, the discriminative properties of the two scales were analyzed by comparing the scores obtained by a group of anxious individuals and a non-anxious control group.ResultsThe results show that our two scales have excellent inter-item correlations. Internal consistency is excellent. Inter-rater agreement ranged from moderate to high. Finally, convergent validity was significant with the alexithymia scale, as was divergent validity with the depression scale. Anxious individuals had lower scores on both scales than non-anxious individuals.ConclusionBoth scales show good psychometric properties for assessing social skills relevant to different collaborative tasks. They also identify individuals with difficulties in social interaction. Thus, they could allow monitoring the effectiveness of training social skills useful in CPS. (shrink)

In this commentary we raise three issues: (1) Is it motherese or song that sets the stage for very early mother-infant interaction? (2) Does the infant play a pivotal role in the complex temporal structure of social interaction? (3) Is the vocal channel primordial or do other modalities play an equally important role in social interaction?

A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several (...) lines of interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual and multimodal. Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario. (shrink)