In the rapidly evolving landscape of multimodal communication research, this paper explores the transformative role of machine learning (ML), particularly using multimodal large language models, in tracking, augmenting, annotating, and analyzing multimodal data. Building upon the foundations laid in our previous work, we explore the capabilities that have emerged over the past years. The integration of ML allows researchers to gain richer insights from multimodal data, enabling a deeper understanding of human (and non-human) communication (...) across modalities. In particular, augmentation methods have become indispensable because they facilitate the synthesis of multimodal data and further increase the diversity and richness of training datasets. In addition, ML-based tools have accelerated annotation processes, reducing human effort while improving accuracy. Continued advances in ML and the proliferation of more powerful models suggest even more sophisticated analyses of multimodal communication, e.g., through models like ChatGPT, which can now “understand” images. This makes it all the more important to assess what these models can achieve now or in the near future, and what will remain unattainable beyond that. We also acknowledge the ethical and practical challenges associated with these advancements, emphasizing the importance of responsible AI and data privacy. We must be careful to ensure that benefits are shared equitably and that technology respects individual rights. In this paper, we highlight advances in ML-based multimodal research and discuss what the near future holds. Our goal is to provide insights into this research stream for both the multimodal research community, especially in linguistics, and the broader ML community. In this way, we hope to foster collaboration in an area that is likely to shape the future of technologically mediated human communication. (shrink)

Multimodal communication research focuses on how different means of signalling coordinate to communicate effectively. This line of research is traditionally influenced by fields such as cognitive and neuroscience, human-computer interaction, and linguistics. With new technologies becoming available in fields such as natural language processing and computer vision, the field can increasingly avail itself of new ways of analyzing and understanding multimodal communication. As a result, there is a general hope that multimodal research may be at (...) the “precipice of greatness” due to technological advances in computer science and resulting extended empirical coverage. However, for this to come about there must be sufficient guidance on key (theoretical) needs of innovation in the field of multimodal communication. Absent such guidance, the research focus of computer scientists might increasingly diverge from crucial issues in multimodal communication. With this paper, we want to further promote interaction between these fields, which may enormously benefit both communities. The multimodal research community (represented here by a consortium of researchers from the Visual Communication [ViCom] Priority Programme) can engage in the innovation by clearly stating which technological tools are needed to make progress in the field of multimodal communication. In this article, we try to facilitate the establishment of a much needed common ground on feasible expectations (e.g., in terms of terminology and measures to be able to train machine learning algorithms) and to critically reflect possibly idle hopes for technical advances, informed by recent successes and challenges in computer science, social signal processing, and related domains. (shrink)

A variety of theoretical frameworks predict the resemblance of behaviors between two people engaged in communication, in the form of coordination, mimicry, or alignment. However, little is known about the time course of the behavior matching, even though there is evidence that dyads synchronize oscillatory motions (e.g., postural sway). This study examined the temporal structure of nonoscillatory actions—language, facial, and gestural behaviors—produced during a route communication task. The focus was the temporal relationship between matching behaviors in the interlocutors (...) (e.g., facial behavior in one interlocutor vs. the same facial behavior in the other interlocutor). Cross-recurrence analysis revealed that within each category tested (language, facial, gestural), interlocutors synchronized matching behaviors, at temporal lags short enough to provide imitation of one interlocutor by the other, from one conversational turn to the next. Both social and cognitive variables predicted the degree of temporal organization. These findings suggest that the temporal structure of matching behaviors provides low-level and low-cost resources for human interaction. (shrink)

Journal Name: Cognitive Linguistics Issue: Ahead of print.

Pantomime has long been considered distinct from co-speech gesture. It has therefore been argued that pantomime cannot be part of gesture-speech integration. We examine pantomime as distinct from silent gesture, focusing on non-co-speech gestures that occur in the midst of children’s spoken narratives. We propose that gestures with features of pantomime are an infrequent but meaningful component of a multimodal communicative strategy. We examined spontaneous non-co-speech representational gesture production in the narratives of 30 monolingual English-speaking children between the ages (...) of 8- and 11-years. We compared the use of co-speech and non-co-speech gestures in both autobiographical and fictional narratives and examined viewpoint and the use of non-manual articulators, as well as the length of responses and narrative quality. The use of non-co-speech gestures was associated with longer narratives of equal or higher quality than those using only co-speech gestures. Non-co-speech gestures were most likely to adopt character-viewpoint and use non-manual articulators. The present study supports a deeper understanding of the term pantomime and its multimodal use by children in the integration of speech and gesture. (shrink)

Name der Zeitschrift: Cognitive Linguistics Jahrgang: 27 Heft: 4 Seiten: 603-618.

Name der Zeitschrift: Semiotica Jahrgang: 2018 Heft: 225 Seiten: 77-102.

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, ), we focus on measuring the temporal (...) patterns of specific communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog. (shrink)

This book positions itself at the intersection of the key areas of the modern humanities. Different authors from a variety of countries take innovative approaches to investigating multimodal communication, adapting pedagogical design to digital environments and enhancing cognitive skills through transformations in teaching and learning practices. The eclectic forms under study require eclectic approaches and methodologies, and the authors cross disciplinary boundaries drawing on philosophy, linguistics, semiotics, computational linguistics, mathematics, cognitive studies and neuroaesthetics. Part I presents methods of (...) analysing multimodal communication in its different displays, covering promotional video in crowdfunding project presentations, multimodal public signs of prohibition and visuals as arguments. Part II explores varied teaching methodologies that have emerged as a result of and in response to modern technological changes and contains some practical hints for educators. It demonstrates the pedagogical potential of video games, virtual worlds, linguistic corpora and online dictionaries. Part III focuses on psychological and cognitive factors influencing success in the classroom, primarily, ways of developing students’ and teachers’ personalities. The volume sits at the intersection between Communication Studies, Digital Humanities, Discourse Analysis, Education Theory and Cognitive Studies and is useful to scholars and students of communication, languages, education and other areas of the humanities. This book should trigger scholarly discussions as well as stimulating practitioners’ interest in these fields. (shrink)

This article starts with the environmental changes in human cognition, analyzes the virtual as the main feature of visual perception under digital technology, and explores the transition from passive to active human cognitive activities. With the diversified understanding of visual information, human contradiction of memory also began to become prominent. Aiming at the problem that the existing multimodal TV media recognition methods have low recognition rate of unknown application layer protocols, an adaptive clustering method for identifying unknown application layer (...) protocols is proposed. This method clusters application layer protocols based on similarity of the load characteristics of network stream application layer protocol data. The method divides the similarity calculation in the clustering algorithm to improve the clustering efficiency of the algorithm. Experimental results show that the proposed method can efficiently and accurately recognize unknown visual communication. This article proposes that, in the interactive multimodal visual information transmission, human visual perception experience has changed, the diversity of visual information content expression makes the aesthetic subject more personalized and stylized. (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, (...) pragmatics, and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

Irregular self-selection is a demonstration of active involvement in interaction. English as a foreign language learners’ talk-in-interaction is one of such cases. Yet, little research has explored when, how, and why learners implement this action. The aim of this article is to address these issues in Chinese postgraduate EFL learners’ conversations from the perspective of multimodal interaction. To this end, we provide descriptive statistics and use multimodal conversation analysis to investigate the detailed process of irregular self-selection. The results (...) show the interactional sensitivity of learners, and all the successful irregular self-selection can be divided into the following three types: turn interruption, turn competition TC), and turn holding abortion. Learners implement this action by using multimodal resources, including lexis, syntax, pitch reset, intensity enhancement, gaze, and so forth. However, their body movements lack diversity, causing behaviors to be constrained and inactive. The main purpose of irregular self-selection is to provide knowledge that contributes to topical development. This study reveals that Chinese postgraduate EFL learners are interactionally competent members. They are able to achieve communicative goals but in a low diversity of body movements. The findings help to understand the detailed process of speakership claiming in EFL learners’ conversations. (shrink)

Open peer commentary on the article “A Cybernetic Approach to Contextual Teaching and Learning” by Philip Baron. Upshot: Universities in South Africa and elsewhere are being challenged to prepare graduates for the world of work and to recognize the multitude of resources that students bring with them to the higher-education experience. A multimodal approach to pedagogy may assist in both of these regards.

This book envisions a relational and multimodal conceptualisation of higher education, which advocates for knowledge to be analyzed under three relational dimensions: social, digital and environmental. The volume draws on interdisciplinary approaches grounded in Peirceian semiotics, in exploring an integration of these dimensions in higher education theory and practice. The book situates learning in an awareness of the environment, grounded in principles of interconnectedness and flattened hierarchies. The volume features practical case studies through dialogues with higher education teachers, presenting (...) different ways of practicing multimodal and relational higher education which prioritize and embrace expression and creation through more than one dominant modality of communication. The book highlights how a relational multimodal paradigm can serve as a way forward for universities to address global challenges concerning social, technological, and environmental futures through a social justice lens. This innovative book will be of interest to students and scholars in semiotics, multimodality, and higher education, as well as stakeholders in these fields. (shrink)

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /, *, \, together with a package of structural postulates characterizing the resource management properties of the * connective. Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke (...) style interpretation in terms of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems each have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems.The first type is obtained by combining a number of unimodal systems into one multimodal logic. The combined multimodal logic is set up in such a way that the individual resource management properties of the constituting logics are preserved. But the inferential capacity of the mixed logic is greater than the sum of its component parts through the addition of interaction postulates, together with the corresponding interpretive constraints on frames, regulating the communication between the component logics.The second type of mixed system is obtained by generalizing the residuation scheme for binary connectives to families of n-ary connectives, and by putting together families, of different arities in one logic. We focus on residuation for unary connectives, hence on mixed frames, as these already represent the complexities in full. We prove a number of elementary logical results for unary families of residuated connectives and their combination with binary families. The existing proposals for unary ‘structural modalities’ are situated within this general framework, and a number of new linguistic applications is given. (shrink)

In this article I analyse object labelling image macro internet memes as multimodal metaphors, taking the Distracted Boyfriend meme as a case study. Object labelling memes are multimodal texts in which users add labels to a stock photograph to convey messages that are often humorous or satirical in nature. Using the relevance-theoretic account of metaphor, I argue that object labelling memes are multimodal metaphors which are interpreted using the same processes as verbal metaphors. The labelling of the (...) image guides the viewer in the construction of ad hoc concepts, and it is these ad hoc concepts that contribute to the overall meaning that is communicated. The analysis in this article is rooted in the relevance-theoretic claim that pragmatic interpretive processes are triggered by all and any ostensive acts of communication. I also draw heavily on Deirdre Wilson’s work on lexical pragmatics to show how this plays out in the case of a multimodal digital text. Memes, like verbal metaphors, do not require a special theory or framework. They can be understood as ostensive stimuli which trigger the search for an optimally relevant interpretation. (shrink)

Metaphor pervades discourse and may govern how we think and act. But most studies only discuss its verbal varieties. This book examines metaphors drawing on combinations of visuals, language, gestures, sound, and music. Investigated texts include advertising, political cartoons, comics, film, songs, and oral communication. Where appropriate, the influence of genre and cultural factors is thematized.

In this article I analyse object labelling image macro internet memes as multimodal metaphors, taking the Distracted Boyfriend meme as a case study. Object labelling memes are multimodal texts in which users add labels to a stock photograph to convey messages that are often humorous or satirical in nature. Using the relevance-theoretic account of metaphor, I argue that object labelling memes are multimodal metaphors which are interpreted using the same processes as verbal metaphors. The labelling of the (...) image guides the viewer in the construction of ad hoc concepts, and it is these ad hoc concepts that contribute to the overall meaning that is communicated. The analysis in this article is rooted in the relevance-theoretic claim that pragmatic interpretive processes are triggered by all and any ostensive acts of communication. I also draw heavily on Deirdre Wilson’s work on lexical pragmatics to show how this plays out in the case of a multimodal digital text. Memes, like verbal metaphors, do not require a special theory or framework. They can be understood as ostensive stimuli which trigger the search for an optimally relevant interpretation. (shrink)

In academic oral genres such as conference presentations, speakers resort to more than words to convey meaning. Research also suggests that persuasion, an important element of the communicative purpose of conference presentations, is frequently achieved through a combination of semiotic modes. Therefore, a skilful orchestration of these modes can be considered key to achieving effective communication in this genre. However, our understanding of persuasion has often focused on specific elements of the message considered in isolation and mainly from the (...) linguistic perspective. Relatively little attention has been paid to the overall persuasive effect achieved by the complex multimodal ensemble. This study approaches the analysis of persuasive strategies in conference presentations combining multimodal discourse analysis and ethnographic methods. It focuses on a particular attention-getting technique: enactment of characters, or acting the part of a person that is being referred to. Our analysis shows how it is achieved through the orchestration of different modes such as words, intonation, gestures, head movements, gaze and facial expression. (shrink)

The analysis of legal communication has almost exclusively been the domain of discourse analysts focusing on the ways that the linguistic system is used to realise legal meanings. Multimodal discourse analysis, where visual forms in combination with traditional linguistic expressions co-occur, is now also an area of expanding interest. Taking a Systemic Functional Linguistics “social semiotic” perspective, this paper applies and critiques an analytical framework that has been used for examining intersemiotic complementarity in various types of page-based (...) class='Hi'>multimodal texts by analysing a cartoon satirizing the demands that verdict deliberations can make on a citizen jury seated in a Crown Court. While the multimodal analysis reveals that a straightforward application of this analytical framework is useful in and of itself, it is argued that the framework needs to be extended to also account for further and more complex layers of represented meaning. The analysis is also situated in the wider legal social semiotic, and draws attention to international debates on the efficacy of the jury system itself. (shrink)

Concern for global warming, climate change and pollution has grown in recent years, with countries across the world facing natural disasters on unprecedented scales. The communication of environmental protection is therefore a necessary area of enquiry, especially from a Conceptual Metaphor Theory perspective. The present article explores (1) how the themes of global warming, climate change, pollution and activism are conceptualized in a corpus of 51 noncommercial advertisements, (2) the interaction of metonymy with metaphor, (3) the distribution across verbal (...) and visual modes of metaphoric source and target domains, and (4) how value is evoked. Findings show that half of the corpus frames environmental themes through source domains such as weapons, predators and natural disasters. The other half triggers incongruous mappings, such as between concrete entities, and relies on metonymic processes of inference to access the main rhetorical message. Among the most frequent metonymies, CAUSE-EFFECT and CATEGORY FOR SALIENT PROPERTY highlight the negative effects of the represented phenomena. Multimodality usually occurs within source and/or target domains rather than across the metaphoric mapping. Most of the campaigns project mixed value, where a negative evaluation of a theme is accompanied by a positive message, inviting the audience to take action. (shrink)

This book shares the recent debates by systemic functional linguistics and other linguistic forums. Its principal focus is on how we use language to make meaning of the world, on how the systems and structures of the ideational function of language represent the realisation of our experiences of the world around us. The volume captures the endeavours of scholars working in different contexts, disciplines and languages around the world. Their contributions explore what underlies experiential and logical meaning-making through specific analyses (...) of recently-created, contextually diverse, single texts or collections of texts, from mono- to multimodal texts. The issues addressed are: layers of meaning through the transitivity system; agency and subjectivity; what kinds of participants and circumstances are associated with various processes and how these vary across languages; new ways of researching and capturing the interaction of the experiential function with the other functions of language - interpersonal, textual and logical - in communicative contexts; how multimodality and new ways of modelling experience semiotically influence the work of linguists, linguistic description and application. The book displays the dynamic dialogue on theoretical and applied interests of scholars interested in functional linguistics and working in a wide range of academic contexts. At post-graduate level advanced students will benefit from new perspectives, the innovative thinking and research accounts that make up the collection. The papers highlight the flexibility of systemic functional linguistic approach and exemplify how it can offer deeper and further insights into potential ways of exploring meaning-making by drawing on recent seminal developments in ideation. (shrink)

Investigations of iconicity in language, whereby interactants coordinate meaningful bodily actions to create resemblances, are prevalent across the human communication sciences. However, when it comes to analysing and comparing iconicity across different interactions and modes of communication, it is not always clear we are looking at the same thing. For example, tokens of spoken ideophones and manual depicting actions may both be analysed as iconic forms. Yet spoken ideophones may signal depictive and descriptive qualities via speech, while manual (...) actions may signal depictive, descriptive, and indexical qualities via the shape, movement, and placement of the hands in space. Furthermore, each may co-occur with other semiotics articulated with the face, hands, and body within composite utterances. The paradigm of iconicity as a single property is too broad and coarse for comparative semiotics, as important details necessary for understanding the range of human communicative potentialities may be masked. Here, we draw on semiotic approaches to language and communication, including the model of language as signalled via describing, indicating and/or depicting and the notion of non-referential indexicality, to illustrate the multidimensionality of iconicity in co-present interactions. This builds on our earlier proposal for analysing how different methods of semiotic signalling are combined in multimodal language use. We discuss some implications for the language and communication sciences and explain how this approach may inform a theory of biosemiotics. (shrink)

Research has shown that suicide rate in Australia is on the rise and that most people who die by suicide are not in contact with mental health services. They most likely communicate their suicidal thoughts to family members or close friends, whose responses may sound unhelpful and/or dismissive, thus reinforcing suicidal ideation. This national emergency has been tackled via a social media campaign, #YouCan Talk, launched by a government-supported digital platform, BeyondBlue. This article adopts a multimodal discourse analysis approach (...) to investigate how peer support is encouraged and articulated in the context of mental health discourse for suicide prevention. The two case studies selected for analysis from the BeyondBlue platform are the #YouCanTalk social media campaign, designed to teach carers to identify severe depression and effectively respond to suicide warning signs and a sample of posts from a thread for peer support in a monitored online forum devoted to help carers who seek peer advice. Unlike previous research, the article focuses on how pronouns and verbs index interpersonal relations in a systemic-functional perspective as well as other multimodal resources, such as visuals, layout and hyperlinking, to understand how identities are entextualised by both professional health providers and peer carers in digital platforms that address mental health issues. (shrink)

The paper examines selected aspects of the defence closing argument in a highly publicised criminal trial to illustrate the orchestration of various semiotic resources in legal persuasion and to explain their role in the creation of meaning. The study demonstrates that closing arguments are multimodal performances whose persuasiveness results from the combination of modes (speech, image, video, gaze, gesture, posture, proxemics) which contextualise and strengthen one another, rather than language alone. Drawing on earlier research into multimodality, courtroom rhetoric and (...) proximity in disciplinary genres, the analysis centres on the ways in which the defence counsel controls the rhetorical features of his narrative and constructs himself and the audience as people with similar understandings and goals. The study specifically demonstrates how the counsel constructs the proximity of commitment, the proximity of membership and the proximity of experience. It explores such facets of proximity as: organisation, argument structure, credibility, stance and engagement, and identifies key rhetorical strategies used to achieve the intended communicative effect. The analysis clearly shows that the persuasiveness of the counsel’s performance depends on the synchronisation of a range of meaning-making resources, which, if used in isolation, would result in a much less engaging argument. (shrink)

This article investigates a multiparty interaction in an accounting office by applying a multimodal approach to discourse. This approach allows the incorporation of all relevant communicative modes and is based on the following three notions: 1) the notion of mediated action; 2) the notion of modal density; and 3) the notion of a foreground– background continuum of attention/awareness. The article illustrates that a social actor in a multiparty interaction simultaneously co-constructs several higher-level actions with the various participants on different (...) levels of their attention/awareness. On a theoretical level, the article argues that traditional approaches to discourse analysis, with their unconditional focus on language as the primary mode, misconstrue the multiple higher-level actions that a social actor is engaged in simultaneously as dyadic or triadic interactions in quick succession. (shrink)

Adult-infant early dyadic interactions have been extensively explored by developmental psychologists. Around the age of 2 months, infants already demonstrate complex, delicate and very sensitive behaviors that seem to express their ability to interact and share emotions with their caregivers. This paper presents 3 pilot studies of parent-infant dyadic interaction in various set-ups. The first two present longitudinal data collected on two infants aged between 1 and 6 months and their mothers. We analyzed the development of coordination between them, at (...) the motor and at the vocal level. The 3rd pilot study aims to explore interpersonal coordination in both vocal behavior and motor activity for one infant and his mother at 2, 4 and 6 months. These pilot studies however leave a number of questions open concerning developmental changes and infants’ progressive mastery of interaction. We identify areas worth examining and try to tease out specific issues that may help develop new methodological pathways for the study of early naturalistic social interaction. We assume that a continuous, rather than discrete, approach would better capture the changes taking place in the various communicative modalities, while also displaying each dyad’s specificity and the narrative dimension of social engagement between infants and caregivers. (shrink)

Improvising selected tools from Kress and Van Leeuwen’s inter-semiosis framework, this study explores how, between global and local, TV commercials in India often reframe a cultural third space producing new discursive forms and identities. Three commercials from the food and beverage category are selected on the basis of the country of origin of the endorsing company and the patterns of glocalization. Multimodal discourse analysis reveals that the commercials construct the glocal identity in several ways. In the Knorr Soups commercial, (...) the melody of the title song is contemporary pop, whereas the delivery style and crooning are distinctively Western. The language is predominantly Hindi with deft use of selected English words. The Funda Mint commercial casts a rebel voice-support executive who rejects the pseudo-glocal identity and asserts his true identity. In the Cadbury Bournvita Folk commercial, the two boys represent the global and the local in the realm of creativity. The study advances the knowledge of product positioning and brand identity involving the intersemiosis of discursive elements from different cultural spaces within the contemporary Indian consumerist culture. (shrink)

Multimodal discourse analysis offers a novel lens for the study of legal discourse. Within the closing arguments of a trial, prosecution and defense attorneys utilize various multimodal resources to present evidence, articulate opinions, and adopt stances to achieve their communicative goals. This research focuses on the closing arguments in the criminal trial concerning the death of George Floyd to analyze the multimodal representation of the closing arguments delivered by both prosecution and defense attorneys. It employs the analytical (...) framework of the attitude system and paralinguistic typology proposed by Martin and colleagues, and uses the UAM Corpus Tool and Praat to qualitatively annotate and quantify the linguistic and paralinguistic information. The findings reveal that these attorneys employ different multimodal resources, including language and paralanguage, to convey their attitudes towards the case’s participants. This leads to the construction of competing narratives, shedding light on how justice is seen to be done. (shrink)

This article analyses the multimodal representations of public space in Chilean broadcast news reports on the figure of the hooded rioter and its alleged connections with the student movement. We seek to identify how space is constructed as a legitimation strategy in relation to the actors involved and the actions taking place across four different news broadcast pieces in the light of Multimodal Critical Discourse Analysis and Systemic Functional Linguistics. Results show that the multimodal representations of space (...) are crucial to identify and functionalize hooded rioters as belonging to the student movement. Actions are dependent on the spaces in which actors operate, restricting the occupation of certain spaces to specific actors. Thus, transgressive protest actions are to be contained to educational spaces and represented as naturalized vandalism, ignoring the students’/demonstrators’ motivations to recuperate/vindicate the public space. (shrink)

The aim of this study is to analyze the relationship between rhythmic movements and deictic gestures at the end of the first year of life, and to focus on their unimodal or multimodal character. We hypothesize that multimodal rhythmic movement performed with an object in the hand can facilitate the transition to the first deictic gestures. Twenty-three children were observed at 9 and 12 months of age in a naturalistic play situation with their mother or father. Results showed (...) that rhythmic movements with objects in the hand are a frequent behavior in children's repertoires. Rhythmic behaviors tend to decrease from 9 to 12 months, specifically when they are unimodal. Multimodal rhythmic behavior production at 9 months is positively related with proximal deictic gestures 3 months later. Multimodal rhythmic movements are not directly related to distal deictic gestures, but are indirectly related via proximal deictic gestures. These results highlight the relevance of multimodal behaviors in the transition to the use of early gestures, and can be considered as a transitional phenomenon between the instrumental action and early communicative gestures. (shrink)

Doing theology with children, in a systematic and focused way, is a new practice. This article contributes to this theological practice by examining its emergence, nature and mission and proposing multimodal narrativity as a practical theological methodology for doing such theology. The article argues, from a practical theological standpoint, that doing theology with children should be understood as a synodal event – a journeying together with children about their faith; as a way of seeing the mysteries of God through (...) the eyes of the child and of encouraging a robust understanding of God and God’s plan for humanity and the entire creation; and as a way of seeing the faces of children and hearing their voices, thereby accounting for their suffering and hope. Children themselves are sources and disruptors of hope. The use of multimodal narrativity creates space for these narratives and disruptions of hope and encourages children to theologise their experiences in the midst of their different communities. (shrink)

When people are engaged in social interaction, they can repeat aspects of each other’s communicative behavior, such as words or gestures. This kind of behavioral alignment has been studied across a wide range of disciplines and has been accounted for by diverging theories. In this paper, we review various operationalizations of lexical and gestural alignment. We reveal that scholars have fundamentally different takes on when and how behavior is considered to be aligned, which makes it difficult to compare findings and (...) draw uniform conclusions. Furthermore, we show that scholars tend to focus on one particular dimension of alignment (traditionally, whether two instances of behavior overlap in form), while other dimensions remain understudied. This hampers theory testing and building, which requires a well‐defined account of the factors that are central to or might enhance alignment. To capture the complex nature of alignment, we identify five key dimensions to formalize the relationship between any pair of behavior: time, sequence, meaning, form, and modality. We show how assumptions regarding the underlying mechanism of alignment (placed along the continuum of priming vs. grounding) pattern together with operationalizations in terms of the five dimensions. This integrative framework can help researchers in the field of alignment and related phenomena (including behavior matching, mimicry, entrainment, and accommodation) to formulate their hypotheses and operationalizations in a more transparent and systematic manner. The framework also enables us to discover unexplored research avenues and derive new hypotheses regarding alignment. (shrink)

The phenomenon of multimodality is central to our everyday interaction. 'Hybrid' modes of communication that combine traditional uses of language with imagery, tagging, hashtags and voice-recognition tools have become the norm. Bringing together concepts of meaning and communication across a range of subject areas, including education, media studies, cultural studies, design and architecture, the authors uncover a multimodal grammar that moves away from rigid and language-centered understandings of meaning. They present the first framework for describing and analysing (...) different forms of meaning across text, image, space, body, sound and speech. Succinct summaries of the main thinkers in the fields of language, communications and semiotics are provided alongside rich examples to illustrate the key arguments. A history of media including the genesis of digital media, Unicode, Emoji, XML and HTML, MP3 and more is covered. This book will stimulate new thinking about the nature of meaning, and life itself, and will serve practitioners and theorists alike. (shrink)