This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results (...) show how the multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

ObjectiveWe wished to explore Parkinson's disease subtypes by clustering analysis based on the multimodal magnetic resonance imaging indices amplitude of low-frequency fluctuation and gray matter volume. Then, we analyzed the differences between PD subtypes.MethodsEighty-six PD patients and 44 healthy controls were recruited. We extracted ALFF and GMV according to the Anatomical Automatic Labeling partition using Data Processing and Analysis for Brain Imaging software. The Ward linkage method was used for hierarchical clustering analysis. DPABI was employed (...) to compare differences in ALFF and GMV between groups.ResultsTwo subtypes of PD were identified. The “diffuse malignant subtype” was characterized by reduced ALFF in the visual-related cortex and extensive reduction of GMV with severe impairment in motor function and cognitive function. The “mild subtype” was characterized by increased ALFF in the frontal lobe, temporal lobe, and sensorimotor cortex, and a slight decrease in GMV with mild impairment of motor function and cognitive function.ConclusionHierarchical clustering analysis based on multimodal MRI indices could be employed to identify two PD subtypes. These two PD subtypes showed different neurodegenerative patterns upon imaging. (shrink)

This study investigates the influences of action research on primary school English instruction from five dimensions in the classroom, viz., types of questions, language errors, gestures, facial expressions, and interpersonal distance. Four English teachers’ 9 real classroom teaching videos before and after action research are collected and annotated by using ELAN software. The results show that primary school English teachers in Chinese rural areas prefer closed questions to open questions; They make some language errors; Deictic gestures are the most (...) common gestures used, while metaphoric gestures, beat gestures and iconic gestures are rare; Teachers have the same preference for three types of facial expressions, and teachers’ serious expressions accounts for most of the time; They seldom keep an intimate distance or personal distance from their students. Second, Action research is effective to motivate teachers in rural areas, who make great progress in all five dimensions after AR: more open questions are asked; pragmatic errors and grammatical errors are reduced; deictic gestures increase; apathetic decrease, and more intimate distance is exhibited in the interpersonal distance dimension. Third, teacher’s English teaching competence is partly transferrable to her future professional development, and this is also the long-lasting effect of AR. (shrink)

Emotion recognition plays an important role in intelligent human–computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a (...) decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine and a subject-independent model based on long short-term memory network. In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals dataset and the Multimodal Database for Affect Recognition and Implicit Tagging dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated. (shrink)

In this article, we carry out a Multimodal Critical Discourse Analysis of a sample from a larger corpus of Romanian news articles that covered the controversial camp evictions and repatriation of Romanian Roma migrants from France that began in 2010 and continue to the time of writing in 2017. These French government policies have been highly criticized both within France and by international political and aid organizations. However, the analysis shows how these brutal, anti-humanitarian events became recontextualized (...) in the Romanian Press to represent the French government’s actions as peaceful and consensual. In addition, the demonization of the Roma in the press serves as a strategy to continuously disassociate them from their Romanian counterparts. While there is a long history of discrimination against the Roma in Romania, these particular recontextualizations can be understood in the context of the Romanian government’s need to gloss over its failure to comply with the Schengen accession requirements and acquire full European Union membership. (shrink)

With the continuous development of software, it is inevitable that there will be various unpredictable problems in computer software or programs that will damage the normal operation of the software. In the paper, static analysis software is taken as the research object, the errors or failures caused by the potential defects of the software modules are analyzed, and a software analysis method based on big data tendency prediction is proposed to use the (...) software defects of the stacked noise reduction sparse analyzer to predict. This method can learn features from original defect data, directly and efficiently extract required features of all levels from software defect data by setting different number of hidden layers, sparse regularization parameters, and noise ratio, and then classify and predict the extracted features by combining with big data. Through experimental tests, the performance of the presented method is better than that of the comparison method in correct rate, accuracy rate, recall rate, F1-measurement, AUC value, and running time, which proves that the research results in this paper have more accurate failure prediction effect and can timely eliminate software failures. (shrink)

The analysis of legal communication has almost exclusively been the domain of discourse analysts focusing on the ways that the linguistic system is used to realise legal meanings. Multimodal discourse analysis, where visual forms in combination with traditional linguistic expressions co-occur, is now also an area of expanding interest. Taking a Systemic Functional Linguistics “social semiotic” perspective, this paper applies and critiques an analytical framework that has been used for examining intersemiotic complementarity in various types of page-based (...) multimodal texts by analysing a cartoon satirizing the demands that verdict deliberations can make on a citizen jury seated in a Crown Court. While the multimodal analysis reveals that a straightforward application of this analytical framework is useful in and of itself, it is argued that the framework needs to be extended to also account for further and more complex layers of represented meaning. The analysis is also situated in the wider legal social semiotic, and draws attention to international debates on the efficacy of the jury system itself. (shrink)

“The journey (the [filmic] narrative) is made by the traveler (the viewer) step by step, pebble by pebble, cue by cue.”Flashbacks in Film: A Cognitive and Multimodal Analysis is devoted to explaini...

ABSTRACTIn this paper, I carry out an analysis of an event in Sweden called ‘the spring turnout’. It is a traditional event where cows are allowed out into the fields after the winter. I show how i...

Textual and visual analyses of nation-branding campaigns are rare but highly needed (Bolin and Ståhlberg, 2010; Hao, Paul, Trott, Guo, and Wu, 2019) as online media have become a popular tool for states to shape people’s perception (Volcic and Andrejevic, 2011). In Anholt’s much applied nation brand hexagon (2007), immigration and investment, society, governance, and culture and heritage are, along with tourism and export, the core aspects that build a country’s reputation. As the 2015 refugee peak situation resulted in a (...) more restrictive approach of the Swedish government towards asylum applicants, the country’s brand was put under pressure. How could Sweden’s values related to openness be highlighted while policymakers chose a deterrent road? In this study, we bridge streams of research on nation branding, framing, and migration studies by presenting a multimodal analysis of the “Portraits of migration” campaign as a strategic response to the refugee situation. (shrink)

Journal Name: Semiotica Issue: Ahead of print.

The British Miners’ Strike of 1984–1985 represents one of the most pivotal periods in British industrial relations. The significance of media stance towards the miners remains a controversial issue today, as attested by recent publications looking back at the strike. Here, authors including miners, journalists and other commentators argue that media coverage of the strike followed a consistently anti-trade union agenda in which the media sought to destabilise the strike. An internal British Broadcasting Corporation report, only recently made public, shows (...) that the BBC themselves had concerns over possible imbalances in their coverage of the so-called ‘Battle of Orgreave’. Despite the weight attached to media coverage in this context, however, surprisingly little research has been conducted from a discourse-analytical perspective to show systematically and empirically how such an agenda may have been manifested across media texts. In this article, drawing on Cognitive Linguistic Critical Discourse Studies, I show how one particular metaphorical framing of the strike, which construed the strike as a war between the State and the National Union of Miners, persisted through the year-long period and consider the potential ideological functions of this framing in media strategies of legitimation. I show how this metaphor featured in linguistic, visual and multimodal forms of media representation. (shrink)

This article analyzes the construction of female subjectivity in the specific context of audiovisual cyberspaces in Spain dedicated to the struggle against violence against women. Looking at the YouTube channels of two virtual feminist communities that deal with violence against women, the authors analyze how the victim-subject is configured in terms of agency and activism. The authors adopt a multimodal model of studying the sign complexes of the videos as semiotic artifacts that produce meaning. Sign complexes are always engaged (...) because representation is never neutral because what is represented in sign is meant to realize the values and positions of those who make the sign. In this article, the authors understand feminist activism in the fight against violence targeting women as constructing in its discourse not only the activist process, but also the subject of this activism: the victim-subject of gender-based violence. The analysis engages in the discussion about the various ways in which this subject is interpellated by the audiovisual texts in terms of agency. As a result, this study proposes the necessity to devise new ways of articulating this subject as a political and agential one. (shrink)

Interactions between an intelligent software agent and a human user are ubiquitous in everyday situations such as access to information, entertainment, and purchases. In such interactions, the ISA mediates the user’s access to the content, or controls some other aspect of the user experience, and is not designed to be neutral about outcomes of user choices. Like human users, ISAs are driven by goals, make autonomous decisions, and can learn from experience. Using ideas from bounded rationality, we frame these (...) interactions as instances of an ISA whose reward depends on actions performed by the user. Such agents benefit by steering the user’s behaviour towards outcomes that maximise the ISA’s utility, which may or may not be aligned with that of the user. Video games, news recommendation aggregation engines, and fitness trackers can all be instances of this general case. Our analysis facilitates distinguishing various subcases of interaction, as well as second-order effects that might include the possibility for adaptive interfaces to induce behavioural addiction, and/or change in user belief. We present these types of interaction within a conceptual framework, and review current examples of persuasive technologies and the issues that arise from their use. We argue that the nature of the feedback commonly used by learning agents to update their models and subsequent decisions could steer the behaviour of human users away from what benefits them, and in a direction that can undermine autonomy and cause further disparity between actions and goals as exemplified by addictive and compulsive behaviour. We discuss some of the ethical, social and legal implications of this technology and argue that it can sometimes exploit and reinforce weaknesses in human beings. (shrink)

Analysing film. Distinguishing the filmic contribution to meaning -- Examples of filmic "textual organisation" -- Redrawing boundaries -- Organisation of the book -- Semiotics and documents. Semiotics and its relations to film -- The nature of discourse semantics -- The film as cinematographic document -- A combined view: filmic documents for filmic discourse -- Constructing the semiotic mode of film. Semiotic multimodality -- The internal organisation of semiotic strata -- Composing and combining semiotic modes -- Materiality and "epistemological commitment" -- (...) Christian Metz and the grande syntagmatique of the image track. The original model -- Two examples of analysis with the grande syntagmatique -- Revisions and rebuttals -- Foundations for analysis: filmic units. The basic units of film: preliminaries -- Audiovisual iconic representations -- Perception, perceptual realism and reliable measurement -- Multiplicity: from perception to discourse -- Filmic units revisited: discourse-motivated definitions -- The paradigmatic organisation of film. Beyond Metz: towards a grande paradigmatique -- Capturing discourse dependency structures in film -- The paradigmatic dimensions of projection, taxis and plane -- Two examples of paradigmatic analysis -- Summary and conclusions. (shrink)

This paper aims at identifying socio-cultural portrayal of women through representational, interactive and compositional meanings with a focus on gender stereotypes propagated by media advertisements in Pakistan. Media adverts as such are an instrumental tool for manipulating attitudes and behavior of large and diverse audience for example, a large body of data reveals that women are portrayed in media to stylize their physical attributes to tempt and persuade customers. Therefore, advertisements are instrumental in creating a certain mind-set by shaping an (...) ideologythrough highlighting the hegemonic representation of men and sexual objectification of women for creating an erotic fantasy. The data for the study comprises print media adverts which were randomly collected to have primarily advertised Pakistan TV morning shows, home products and cosmetics and have been selected on an assumption that they embody a socio-cultural perspective. The findings show that the selected advertsproject the world of male chauvinism where women are shown as the facilitating sexual objects. (shrink)

The analysis of video-recorded interaction consists of various professionalized ways of seeing participant behavior through multimodal, co-operative, or intercorporeal lenses. While these perspectives are often adopted simultaneously, each creates a different view of the human body and interaction. Moreover, microanalysis is often produced through local practices of sense-making that involve the researchers’ bodies. It has not been fully elaborated by previous research how adopting these different ways of seeing human behavior influences both what is seen from a video (...) and how it is seen, as well as the way the interpretation of the data ultimately unfolds in the interaction between researchers. In this article, we provide a theoretical-methodological discussion of the microanalytic research process. We explore how it differs from “seeing” affect in interaction either as a co-operative and multimodal action or as an intercorporeal experience. First, we introduce the multimodal conversation analytic, co-operative, and intercorporeal approaches to microanalysis. Second, we apply and compare these practices to a video-recorded interaction of a romantic couple. Furthermore, we examine a video-recorded episode of us, the researchers, reflecting on our analytic observations about this interaction. We suggest that adopting a multimodal and co-operative perspective constructs affect as co-produced and displayed through observable action, while an intercorporeal perspective produces affect as an embodied and experienced phenomenon. While the former enables locating affect in a specific moment and identifiable body parts, the latter facilitates recognizing the experienced side of affect. These different modes of professional vision complement one another in capturing affect in interaction while being fundamentally used in local interactions between the researchers. (shrink)

The present article is devoted to a logical treatment of some fundamental concepts involved in responsibility attribution. We specify a theoretical framework based on a language of temporal deontic logic with agent-relative operators for deliberate causal contribution. The framework is endowed with a procedure to solve normative conflicts which arise from the assessment of different normative sources. We provide a characterization result for a basic system within this framework and illustrate how the concepts formalized can be put at work in (...) the analysis of examples of legal reasoning. (shrink)

Researchers in virtually every discipline rely on sophisticated proprietary software for their work. However, some researchers are unable to afford the licenses and instead procure the software illegally. We discuss the prohibition of software piracy by intellectual property laws, and argue that the moral basis for the copyright law offers the possibility of cases where software piracy may be morally justified. The ethics codes that scientific institutions abide by are informed by a rule-consequentialist logic: by preserving (...) personal rights to authored works, people able to do so will be incentivized to create. By showing that the law has this rule-consequentialist grounding, we suggest that scientists who blindly adopt their institutional ethics codes will commit themselves to accepting that software piracy could be morally justified, in some cases. We hope that this conclusion will spark debate over important tensions between ethics codes, copyright law, and the underlying moral basis for these regulations. We conclude by offering practical solutions for researchers. (shrink)

This book presents the methodological framework of combining Multimodal Conversation Analysis (MCA) with Interpretative Phenomenological Analysis (IPA) to interpretively analyse translanguaging practices in educational contexts. Beginning with an overview of the three uses of translanguaging - translanguaging as a theory of language, as a pedagogical practice and as an analytical perspective - the book goes on to critically examine the different methodological approaches for analysing translanguaging practices in multilingual classroom interactions. It explains how MCA and IPA are (...) useful methodologies for understanding how and why translanguaging practices are constructed by participants in the classroom and discusses types of data collected and data collection procedures. The author, Kevin W. H. Tai, shows how combining these approaches enables researchers to study how translanguaging practices are constructed in multilingual classrooms and how teachers make sense of their own translanguaging practices at particular moments of classroom interaction. This detailed and concise guide is indispensable for students, practitioners, policymakers, and researchers from across the globe, particularly those working in the fields of applied linguistics and language education. (shrink)

Musical performance is a multimodal experience, for performers and listeners alike. This paper reports on a pilot study which constitutes the first step toward a comprehensive approach to the experience of music as performed. We aim at bridging the gap between qualitative and quantitative approaches, by combining methods for data collection. The purpose is to build a data corpus containing multimodal measures linked to high-level subjective observations. This will allow for a systematic inclusion of the knowledge of music (...) professionals in an analytic framework, which synthesizes methods across established research disciplines. We outline the methods we are currently developing for the creation of a multimodal data corpus dedicated to the analysis and exploration of instrumental music performance from the perspective of embodied music cognition. This will enable the study of the multiple facets of instrumental music performance in great detail, as well as lead to the development of music creation techniques that take advantage of the cross-modal relationships and higher-level qualities emerging from the analysis of this multi-layered, multimodal corpus. The results of the pilot project suggest that qualitative analysis through stimulated recall is an efficient method for generating higher-level understandings of musical performance. Furthermore, the results indicate several directions for further development, regarding observational movement analysis, and computational analysis of coarticulation, chunking, and movement qualities in musical performance. We argue that the development of methods for combining qualitative and quantitative data are required to fully understand expressive musical performance, especially in a broader scenario in which arts, humanities, and science are increasingly entangled. The future work in the project will therefore entail an increasingly multimodal analysis, aiming to become as holistic as is music in performance. (shrink)

Improvising selected tools from Kress and Van Leeuwen’s inter-semiosis framework, this study explores how, between global and local, TV commercials in India often reframe a cultural third space producing new discursive forms and identities. Three commercials from the food and beverage category are selected on the basis of the country of origin of the endorsing company and the patterns of glocalization. Multimodal discourse analysis reveals that the commercials construct the glocal identity in several ways. In the Knorr Soups (...) commercial, the melody of the title song is contemporary pop, whereas the delivery style and crooning are distinctively Western. The language is predominantly Hindi with deft use of selected English words. The Funda Mint commercial casts a rebel voice-support executive who rejects the pseudo-glocal identity and asserts his true identity. In the Cadbury Bournvita Folk commercial, the two boys represent the global and the local in the realm of creativity. The study advances the knowledge of product positioning and brand identity involving the intersemiosis of discursive elements from different cultural spaces within the contemporary Indian consumerist culture. (shrink)

Ledin and Machin's critique of the use of some current approaches to multimodality for the purposes of critical discourse studies raises some important methodological concerns that need to be addressed. However, both the particular position they develop as well as some of the key points they raise are themselves problematic. In this response, I argue that Ledin and Machin misconstrue some significant aspects of the body of theory they critique and, as a consequence, offer a potentially misleading view of the (...) relevant state of the art in multimodality. I suggest that pursuing closer alignments with theoretically more nuanced accounts of multimodality would allow the account that Ledin and Machin propose to be revised in ways better suited to the task of conducting empirically-driven multimodal critical discourse studies. (shrink)

A number of studies have investigated and found a significant relationship among economic wealth, Hofstede’s national culture dimensions, and software piracy rates (SPR). No study, however, has examined the relationship between economic wealth, culture, and the fact that national SPRs have been declining steadily since 1994. Using a larger sample than has previously been available (57 countries), we confirm the expected negative relationship between economic wealth, culture (individualism and masculinity) and levels of software piracy. The rate of decline (...) in software piracy, however, is found to be a cultural phenomenon, with two factors (power distance (PDI) and uncertainty avoidance (UAI)) working in opposition. Similar results are found for a subset of 37 relatively poor countries. This suggests that, while the rise in economic wealth seen for most countries should lead to a reduction in software piracy, the rate of decline is determined by cultural factors. Global strategies for dealing with software piracy are discussed. (shrink)

We will present relational tools for analysing software systems, based on the Haskell datatype for relations defined in Chapter 5 of Doets and Van Eijck, The Haskell Road to Logic, Maths and Programming, King’s College Publications, London 2004 [DvE04]. The main purpose is to demonstrate some very concrete applications of abstract relations, and to make the point that functional programming is highly relevant to software engineering.

This paper uses van Leeuwen’s Authority Legitimation framework to examine government posters published in the 1950s–1980s in Hong Kong, which serve as a means of shaping public opinion and legitimate social action. Martin and White’s Appraisal framework is also applied to provide the study with relevant analytical tools by which to construct evaluatively coherent authorial reading positions propagated by the government in the posters as well as aligning viewers with these desired positions. The government posters being studied are concerned with (...) various aspects of social practices in the domain of public health communication. This paper argues that the representations of these social practices are realized as texts – both visual and verbal – in the posters, as part of a broader recontextualization process where legitimate ways of doing things as stipulated in the government guidelines are recontextualized in these posters and conveyed to members of the public. The aim is two-fold: to provide a systematic analysis of how visual and verbal resources construe evaluation in the government posters and the consequent legitimation discourse as viewers are being persuaded to align with legitimate social practices; to highlight various forms of legitimation, namely, role model authority legitimation, impersonal authority legitimation, and conformity authority legitimation, that are realized in these posters through visual-verbal articulation. (shrink)

Two property regimes for software development may be distinguished. Within corporations, on the one hand, a Private Regime obtains which excludes all outsiders from access to a firm's software assets. It is shown how the protective instruments of secrecy and both copyright and patent have been strengthened considerably during the last two decades. On the other, a Public Regime among hackers may be distinguished, initiated by individuals, organizations or firms, in which source code is freely exchanged. It is (...) argued that copyright is put to novel use here: claiming their rights, authors write `open source licenses' that allow public usage of the code, while at the same time regulating the inclusion of users. A `regulated commons' is created. The analysis focuses successively on the most important open source licenses to emerge, the problem of possible incompatibility between them (especially as far as the dominant General Public License is concerned), and the fragmentation into several user communities that may result. (shrink)

In this article I analyse object labelling image macro internet memes as multimodal metaphors, taking the Distracted Boyfriend meme as a case study. Object labelling memes are multimodal texts in which users add labels to a stock photograph to convey messages that are often humorous or satirical in nature. Using the relevance-theoretic account of metaphor, I argue that object labelling memes are multimodal metaphors which are interpreted using the same processes as verbal metaphors. The labelling of the (...) image guides the viewer in the construction of ad hoc concepts, and it is these ad hoc concepts that contribute to the overall meaning that is communicated. The analysis in this article is rooted in the relevance-theoretic claim that pragmatic interpretive processes are triggered by all and any ostensive acts of communication. I also draw heavily on Deirdre Wilson’s work on lexical pragmatics to show how this plays out in the case of a multimodal digital text. Memes, like verbal metaphors, do not require a special theory or framework. They can be understood as ostensive stimuli which trigger the search for an optimally relevant interpretation. (shrink)

It is argued in this paper that a multimodal analysis of turn-taking, one of the core areas of conversation analytic research, is needed and has to integrate gaze as one of the most central resources for allocating turns, and that new technologies are available that can provide a solid and reliable empirical foundation for this analysis. On the basis of eye-tracking data of spontaneous conversations, it is shown that gaze is the most ubiquitous next-speaker-selection technique. It can (...) function alone or enhance other techniques. I also discuss the interrelationship between the strength for sequential projection and the choice of next-speaker-selection techniques by a current speaker. The appropriate consideration of gaze leads to a revision of the turn-taking model in that it reduces the domain of self-selection and expands that of the current-speaker-selects-next sub-rule. It also has consequences for the analysis of “simultaneous starts”. (shrink)

Studies of talk-and-bodily-conduct-in-interaction have inspired new insights into the way in which language, interaction and cognition might be articulated. More particularly, they have shown that participants mutually orient to the finely tuned multimodal details by which talk and action in interaction are sequentially organized. This article deals with this form of ‘participants’ multimodal online analysis’ by focusing on a particular phenomenon - the methodical practices and resources by which the end of a turn and of an activity (...) phase is projected and collectively achieved - in a specific videorecorded setting - a meeting in an architect’s office. It aims at questioning both how these local orientations are systematically displayed and exploited by the participants for the sequential organization of their activity and how they can be demonstrably observed by the analyst. (shrink)

In this contribution we will approach language and images in different modalities: speech and face, anticipated and imagined movements, illusions on the sound by the image. It will be the opportunity for us to revisit the Gestalt concepts which were considered obsolete since structuralism in Humanities. As instantiated by Gilbert Durand in The Anthropological Structures of the Imaginary (1999, French 1st ed. 1960), we shall recall that Gestalt is not—even implicitly—an exclusively static approach to cognition. On the contrary we will (...) emphasize that forms can emerge from movements and stabilize in memory, thence morphology. And for speech, we will consider phonology as the stabilized outcome from audible and visible gestures of the mouth—caught in the time-course of a flow of some four in six syllables by second—via the perception of motor coordinations: which is necessary to master the most common expression of language between humans. We will adopt here also for the perceptual flows of spoken language, Multimodal Scene Analysis, the ongoing legacy of Gestalt Theory, including one of our main research topics: the gestural expression of the face and the hand for Deaf people, who practise as an augment French LPC (Langue française Parlée Complétée), adapted in 1967 from Cued Speech by its originator Dr Richard Cornett. (shrink)

This book contributes to the analysis of film from a multimodal and textual perspective by extending formal semantics into the realm of multimodal discourse analysis. It accounts for both the inferential as well as intersemiotic meaning making processes in filmic discourse and therefore addresses one of the main questions that have been asked within film theory and multimodal analysis: How do we understand film and multimodal texts? The book offers an analytical answer to (...) this question by providing a systematic tool for the description of this comprehension process. It aims to advance knowledge of the various resources in filmic texts, the ways the resources work together in constructing meaning and the ways people understand this meaning construction. This new approach to film interpretation is thus able to remodel and improve the classical paradigm of film text analysis. (shrink)

Multimodal Critical Discourse Analysis (MCDA) as a sub-discipline of Critical Discourse Analysis (CDA) emerged from the availability of social semiotic frameworks describing multimodal meaning making. However, weaknesses of these frameworks have raised concerns and prompted recent methodological reflections in MCDA. Inspired by these reflections, this paper critically assesses MCDA research on advertising persuasion and identifies a lack of attention in studies to account for the social and ideological impact of advertising. This shortcoming is argued to be (...) attributable to the weak empirical grounding of social semiotic frameworks which in turn is rooted in a ‘discursive gap’ (Bateman, Citation2019, p. 300) in multimodality as a discipline. The authors then propose methodological solutions involving eye tracking experiments and higher order data analytics which can strengthen the empirical basis of theoretical assumptions made in MCDA research on advertising. Drawing on methodological guidance for empirical multimodal research, the authors discuss specific issues in realizing this proposal, focusing on a strategy addressing operationalization challenges in experimental design and the affordance of a data cluster algorithm that calculates the common viewing path of a group of viewers. (shrink)

Software piracy is a damaging and important moral issue, which is widely believed to be unchecked in particular areas of the globe. This cross-cultural study examines differences in morality and behavior toward software piracy in Singapore versus the United States, and reviews the cultural histories of Asia versus the United States to explore why these differences occur. The paper is based upon pilot data collected in the U.S. and Singapore, using a tradeoff analysis methodology and analysis. (...) The data reveal some fascinating interactions between the level of ethical transgression and the rewards or consequences which they produce. (shrink)

Research has shown that suicide rate in Australia is on the rise and that most people who die by suicide are not in contact with mental health services. They most likely communicate their suicidal thoughts to family members or close friends, whose responses may sound unhelpful and/or dismissive, thus reinforcing suicidal ideation. This national emergency has been tackled via a social media campaign, #YouCan Talk, launched by a government-supported digital platform, BeyondBlue. This article adopts a multimodal discourse analysis (...) approach to investigate how peer support is encouraged and articulated in the context of mental health discourse for suicide prevention. The two case studies selected for analysis from the BeyondBlue platform are the #YouCanTalk social media campaign, designed to teach carers to identify severe depression and effectively respond to suicide warning signs and a sample of posts from a thread for peer support in a monitored online forum devoted to help carers who seek peer advice. Unlike previous research, the article focuses on how pronouns and verbs index interpersonal relations in a systemic-functional perspective as well as other multimodal resources, such as visuals, layout and hyperlinking, to understand how identities are entextualised by both professional health providers and peer carers in digital platforms that address mental health issues. (shrink)