This article analyses the multimodal representations of public space in Chilean broadcast news reports on the figure of the hooded rioter and its alleged connections with the student movement. We seek to identify how space is constructed as a legitimation strategy in relation to the actors involved and the actions taking place across four different news broadcast pieces in the light of Multimodal Critical Discourse Analysis and Systemic Functional Linguistics. Results show that the multimodal representations (...) of space are crucial to identify and functionalize hooded rioters as belonging to the student movement. Actions are dependent on the spaces in which actors operate, restricting the occupation of certain spaces to specific actors. Thus, transgressive protest actions are to be contained to educational spaces and represented as naturalized vandalism, ignoring the students’/demonstrators’ motivations to recuperate/vindicate the public space. (shrink)

Name der Zeitschrift: Semiotica - Journal of the International Association for Semiotic Studies / Revue de l'Association Internationale de Sémiotique Jahrgang: 2013 Heft: 197 Seiten: 79-100.

First of all I would like to describe inductive and abductive reasoning in the light of the agent–based framework to the aim of clarifying their fallacious character and the role of the so-called ideal systems (logical and computational). Then I will analyze some inductive and abductive types of reasoning that in the perspective of classical and informal logic are defined fallacies. I will describe how in an agent-based reasoning this kind of fallacious reasoning can in some cases be redefined and (...) considered as a good way of reasoning. Finally, I will illustrate how what I call manipulative abduction can be interpreted as a form of practical reasoning a better understanding of which can furnish a description of human beings as hybrid reasoners in so far they are users of ideal and computational agents. (shrink)

Our brains make up a series of signs and are engaged in making or manifesting or reacting to a series of signs: through this semiotic activity they are at the same time engaged in “being minds” and so in thinking intelligently. An important effect of this semiotic activity of brains is a continuous process of “externalization of the mind” that exhibits a new cognitive perspective on the mechanisms underling the semiotic emergence of abductive processes of meaning formation. To illustrate this (...) process I will take advantage of the analysis of some aspects of the cognitive interplay between internal and external representations. I consider this interplay critical in analyzing the relation between meaningful semiotic internal resources and devices and their dynamical interactions with the externalized semiotic materiality suitably stocked in the environment. Hence, minds are material, “extended” and artificial in themselves. A considerable part of human abductive thinking is occurring through an activity consisting in a kind of reification in the external environment and a subsequent re-projection and reinterpretation through new configurations of neural networks and chemical processes. I also illustrate how this activity takes advantage of hybrid representations and how it can nicely account for various processes of creative and selective abduction, bringing up the question of how multimodal aspects involving a full range of sensory modalities are important in hypothetical reasoning. (shrink)

One of the main limitations of natural language‐based approaches to meaning is that they do not incorporate multimodal representations the way humans do. In this study, we evaluate how well different kinds of models account for people's representations of both concrete and abstract concepts. The models we compare include unimodal distributional linguistic models as well as multimodal models which combine linguistic with perceptual or affective information. There are two types of linguistic models: those based on text (...) corpora and those derived from word association data. We present two new studies and a reanalysis of a series of previous studies. The studies demonstrate that both visual and affective multimodal models better capture behavior that reflects human representations than unimodal linguistic models. The size of the multimodal advantage depends on the nature of semantic representations involved, and it is especially pronounced for basic‐level concepts that belong to the same superordinate category. Additional visual and affective features improve the accuracy of linguistic models based on text corpora more than those based on word associations; this suggests systematic qualitative differences between what information is encoded in natural language versus what information is reflected in word associations. Altogether, our work presents new evidence that multimodal information is important for capturing both abstract and concrete words and that fully representing word meaning requires more than purely linguistic information. Implications for both embodied and distributional views of semantic representation are discussed. (shrink)

Analysing film. Distinguishing the filmic contribution to meaning -- Examples of filmic "textual organisation" -- Redrawing boundaries -- Organisation of the book -- Semiotics and documents. Semiotics and its relations to film -- The nature of discourse semantics -- The film as cinematographic document -- A combined view: filmic documents for filmic discourse -- Constructing the semiotic mode of film. Semiotic multimodality -- The internal organisation of semiotic strata -- Composing and combining semiotic modes -- Materiality and "epistemological commitment" -- (...) Christian Metz and the grande syntagmatique of the image track. The original model -- Two examples of analysis with the grande syntagmatique -- Revisions and rebuttals -- Foundations for analysis: filmic units. The basic units of film: preliminaries -- Audiovisual iconic representations -- Perception, perceptual realism and reliable measurement -- Multiplicity: from perception to discourse -- Filmic units revisited: discourse-motivated definitions -- The paradigmatic organisation of film. Beyond Metz: towards a grande paradigmatique -- Capturing discourse dependency structures in film -- The paradigmatic dimensions of projection, taxis and plane -- Two examples of paradigmatic analysis -- Summary and conclusions. (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, (...) pragmatics, and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a (...) set of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to (...) class='Hi'>multimodal experiences of pain. Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (...) (e.g. /gi/) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

Open peer commentary on the article “A Cybernetic Approach to Contextual Teaching and Learning” by Philip Baron. Upshot: Universities in South Africa and elsewhere are being challenged to prepare graduates for the world of work and to recognize the multitude of resources that students bring with them to the higher-education experience. A multimodal approach to pedagogy may assist in both of these regards.

This study proposes a multimodal fusion model to account for the cognitive mechanisms involving 56 political cartoons with regard to U.S. beef import issues as reported in two dominant Taiwanese newspapers, the Liberty Times and United Daily News. Specifically, this study claims that multimodal fusion model evolves from two metonymic-metaphoric networks, i.e., related metonymic network and diversified metaphoric network, and combines the conceptual, visual, and verbal modes. Our analysis demonstrates that multimodal fusion is a significant and recurrent (...) representation technique in the genre of political cartoon and has the cognitive function of encapsulating the abstract complex political debates efficiently with irony and humorous effect. Furthermore, our analysis shows the important role of metonymy and demonstrates how metonymies and metaphors are interwoven in the process of multimodal fusion, which underlies the metaphorical mappings of conceptual scenarios related to “POLITICS IS.. (shrink)

From the perspective of distributed cognition I will stress how abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc., but also kinesthetic – related to the ability to sense the position and location and orientation and movement of the body and its parts – and motor experiences and other feelings such as pain, and thus all sensory modalities. The presence of kinesthetic (...) and motor aspects demonstrates that abductive reasoning can be manipulative. We can also see, in this regard, how implicit factors take part in the abductive procedure, which consequently acquires the character of a kind of “thinking through doing”. This paper further describes 1) the fact that hypotheses in science can be built through different cognitive mediators and so they can also model the same cognitive aspect in different ways; how they can be carriers/ producers of knowledge in a multimodal way; 2) the problem of the possible non-explanatory and instrumental nature of abductive reasoning and the analysis of the consequences for induction; 3) the role of manipulative abduction in building new evidence/ experiments and how they trigger smart inductive inferences. (shrink)

Dancers and musicians are experts in spatial and temporal processing which allows them to coordinate movement with music. This high-level processing has been associated with structural and functional adaptation of the brain for high performance sensorimotor integration. For these integration processes, adaptation does not only take place in primary and secondary sensory and motor areas but tertiary brain areas, such as the lateral prefrontal cortex (lPFC) and the intraparietal sulcus (IPS), also provide vital resources for highly specialized performance. Here, we (...) review evidence for the role of these brain areas in multimodal training protocols and integrate these findings into a new model of sensorimotor processing in complex motor learning. (shrink)

In this paper I contend that abduction is essentially multimodal, in that both data and hypotheses can have a full range of verbal and sensory representations, involving words, sights, images, smells, etc. but also kinesthetic experiences and other feelings such as pain, and thus all sensory modalities. The kinesthetic aspects simply explain abductive reasoning is basically manipulative, both linguistic and non linguistic signs have an internal semiotic life, as particular configurations of neural networks and chemical distributions (and in (...) terms of their transformations) at the level of human brains, and as somatic expressions, but can also be delegated to many external objects and devices, for example written texts, diagrams, artifacts, etc. Still in this case unconscious aspects takepart of the abductive procedure, which acquires the features of a kind of "thinking through doing". (shrink)

Recent research on proprioception reveals that it relies on a systematically distorted model of bodily dimensions. This generates a puzzle about proprioception in action control: action requires accurate bodily parameters. Proprioception is crucial for ordinary action, but if it relies on a systematically distorted body model, then proprioception should contain systematic errors. But we cannot respond by discarding proprioception from motor control, since we know from the severe problems deafferented agents face in acting that ordinary action requires proprioception. The solution (...) is that the possibility of bodily action is provided for by multimodal body representations for action. (shrink)

This paper proposes a methodological synthesis in order to study multimodal media discourse and argumentation in the context of the so-called ‘refugee crisis’ in Greece. It follows the framework of Multimodal Critical Discourse Analysis, integrating this with argumentation studies, with a particular emphasis on the analysis of inference. Our data come from the Greek newspapers Kathimerini and Ta Nea. We contend that the proposed methodological synergy enables scrutiny of (a) racist conceptualizations cultivated by the representation of migrants and (...) refugees in headlines and photographs in newspaper articles along with (b) the argumentative potential that is implicitly sustained in these multimodal representations. (shrink)

This paper uses van Leeuwen’s Authority Legitimation framework to examine government posters published in the 1950s–1980s in Hong Kong, which serve as a means of shaping public opinion and legitimate social action. Martin and White’s Appraisal framework is also applied to provide the study with relevant analytical tools by which to construct evaluatively coherent authorial reading positions propagated by the government in the posters as well as aligning viewers with these desired positions. The government posters being studied are concerned with (...) various aspects of social practices in the domain of public health communication. This paper argues that the representations of these social practices are realized as texts – both visual and verbal – in the posters, as part of a broader recontextualization process where legitimate ways of doing things as stipulated in the government guidelines are recontextualized in these posters and conveyed to members of the public. The aim is two-fold: to provide a systematic analysis of how visual and verbal resources construe evaluation in the government posters and the consequent legitimation discourse as viewers are being persuaded to align with legitimate social practices; to highlight various forms of legitimation, namely, role model authority legitimation, impersonal authority legitimation, and conformity authority legitimation, that are realized in these posters through visual-verbal articulation. (shrink)

ABSTRACT The goal of this paper is to improve heuristics for hard news discourse by proposing a cognitive model of abstraction, regarding social media contexts. To this end, hard news is discussed as a genre, structure and metalanguage, under the formal definition of a semiotic mode. Annotation is a successful technology to control the effects of genre operations, reveal relations, and to inquire about data and documents. It proceeds to characterize Twitter’s interface, in terms of formal and material regularities, employed (...) recursively. It further demonstrates the formalization of a semantics for hard news discourse in the so-called logical forms, which has been adapted from analytical tools developed by the Segmented Theory of Discourse Representation (SDRT) in order to examine coherence in different levels of detail. Finally, the implication of this approach as a discipline is discussed, regarding the production of transversal knowledge aiming at digital literacy, which urges in present times.1. (shrink)

RESUMO O objetivo do artigo é aperfeiçoar heurísticas para o discurso das notícias por meio do desenvolvimento de um modelo cognitivo de abstração, em contextos de mídia social. Para tal, discute-se a notícia como gênero, estrutura e metalinguagem, partindo da noção formal de modo semiótico. A anotação sob essa visada é uma tecnologia bem-sucedida para controlar os efeitos das restrições de gênero, revelar relações e inquirir sobre dados e documentos. Procede-se à anotação da notícia na interface do Twitter, caracterizada em (...) termos de suas regularidades formais e materiais empregadas recursivamente. Demonstra-se a formalização de uma semântica para o discurso da notícia nas chamadas formas lógicas, adaptada de ferramentas desenvolvidas pela Segmented Discourse Representation Theory (SDRT), a fim de construir novas hipóteses sobre coerência em diferentes níveis de detalhe. Por fim, discute-se a implicação desta abordagem como disciplina na produção de conhecimento transversal com vistas à literacia digital, em tempos urgentes1. (shrink)

From the viewpoint of pattern recognition and computational learning, mirror neurons form an interesting multimodal representation that links action perception and planning. While it seems unlikely that all details of such representations are specified by the genetic code, robust learning of such complex representations likely requires an appropriate interplay between plasticity, generalization, and anatomical constraints of the underlying neural architecture.

Multimodal discourse analysis offers a novel lens for the study of legal discourse. Within the closing arguments of a trial, prosecution and defense attorneys utilize various multimodal resources to present evidence, articulate opinions, and adopt stances to achieve their communicative goals. This research focuses on the closing arguments in the criminal trial concerning the death of George Floyd to analyze the multimodal representation of the closing arguments delivered by both prosecution and defense attorneys. It employs the analytical (...) framework of the attitude system and paralinguistic typology proposed by Martin and colleagues, and uses the UAM Corpus Tool and Praat to qualitatively annotate and quantify the linguistic and paralinguistic information. The findings reveal that these attorneys employ different multimodal resources, including language and paralanguage, to convey their attitudes towards the case’s participants. This leads to the construction of competing narratives, shedding light on how justice is seen to be done. (shrink)

The representational dynamics of the brain is a subsymbolic process, and it has to be conceived as an "agent-free" type of dynamical self-organization. However, in generating a coherent internal world-model, the brain decomposes target space in a certain way. In doing so, it defines an "ontology": to have an ontology is to interpret a world. In this paper we argue that the brain, viewed as a representational system aimed at interpreting the world, possesses an ontology too. It decomposes target space (...) in a way that exhibits certain invariances, which in turn are functionally significant. A challenge for empirical research is to determine which are the functional regularities guiding this decomposition process. What are the explicit and implicit assumptions about the structure of reality, which at the same time shape the causal profile of the brain's motor output and the representational deep structure of the conscious mind arising from it (its "phenomenal output")? How do they constrain high-level phenomena like conscious experience, the emergence of a first-person perspective, or social cognition? By reviewing a series of neuroscientific results, we focus on the contribution the motor system makes to this process. As it turns out, the motor system constructs goals, actions, and intending selves as basic constituents of the world it interprets. It does so by assigning a single, unified causal role to them. Empirical evidence now clearly shows how the brain actually codes movements and action goals in terms of multimodal representations of organism-object relations. Under a representationalist analysis, this process can be interpreted as an internal representation of the intentionality relation itself. We try to show how such a more complex form of representational content, once it is in place, can later function as the building block for social cognition and a for more complex, consciously experienced representation of the first-person perspective as well. The motor system may therefore play a decisive role in understanding how the functional ontology of the human brain could be gradually extended into the subjective and social domains. (shrink)

Argumentation is a cognitive process of reconstruction that occurs in the discursive proximal space. Multimodal argumentation, from a cognitive-semiotic perspective, suggests that certain multimodal structures can lead audiences to an accurate mental representation of argumentation, depending on the knowledge possessed, codes and signs employed (modes), and context. It is not necessary to have a formal standardized and verbal structure. We are able to argue effectively with others without the need for verbal translation, taking into account the semiotic interpretation (...) of the message. (shrink)

The rise in Study Abroad participation among college students has increased interest among educationalists wondering about the impact of SA on students, particularly when students return home without evidence of deep engagement and understanding of other cultures and people. The purpose of this case study was to locate one potential source of the meanings ascribed to the SA experience, through analysis of multimodal representations on the institutional website of a popular SA program provider. In this study, Kress’ model (...) of multimodal social semiotic communication was employed in the analysis of the ensemble of modes of communication on the website, and their role in the production and dissemination of discourses of SA. Findings indicate that discourses of tourism prevail over discourses of education, and the representations enacted on the institutional website are mirrored in the discursive practices of students. (shrink)

This study investigates how COVID-19 advice is creatively delivered in a one-minute video produced by the Taiwan Centers for Disease Control in February 2022. It examines metaphors in verbal, visual, and multimodal modes, and illustrates how such metaphors interact with multimodal narrativity. Drawing on the insights of studies on multimodal metaphors, this paper seeks to identify which are the most pervasive metaphors or what dominant metaphor, if any, is used in the video. It finds that the metaphorical (...) mappings between coping with the covid pandemic and handling olympic weightlifting are represented across modes and that the creative delivery of COVID-19 advice is woven from a warp of metaphors in various modes and a weft of narrativity across modes. Such metaphorical creativity is underpinned by the semiotic flow that features four interconnected configurations: moving between metaphorization, literalization, and re-metaphorization; somehow blurring the source-target boundary; expanding the range of functions that the multimodal representation can serve; and initiating a metaphorical thought from one or more contextual factors. (shrink)

Les langues des signes sont des langues qui n’ont pas un système d’écriture qui leur est propre et qui, par leur nature visuo-gestuelle, ne peuvent pas être écrites phonographiquement. Les conséquences de cette « non‑scriptibilité » sont au centre des travaux menés à partir de 2000 par Elena Antinoro Pizzuto. EAP, décédée en 2011, mettait en évidence les biais que l’utilisation des « gloses », employées comme substituts de la transcription, induisent dans l’analyse linguistique des LS. Ces 10 dernières années, (...) les avancées technologiques et méthodologiques ont rendu plus aisée la saisie automatique des données corporelles, l’annotation multilinéaire de corpus multimodaux et le maintien des liens entre les annotations et les vidéos, ainsi que la diffusion des vidéos sur lesquelles se basent les analyses. Toutefois, ces technologies et bonnes pratiques — bien que fondamentales — ne font pas office de système de transcription : ce sont des solutions qui suppriment une partie des symptômes liés à la non‑scriptibilité des LS, mais ne comblent pas l’absence d’un système de représentation graphique de ces langues. Les questions soulevées par EAP sont donc encore d’actualité. Si différentes tentatives pour transcrire les LS ont vu le jour, leur utilisation pour exploiter de vastes corpus de LS s’est souvent heurtée à leur faible lisibilité et/ou requêtabilité. En comparant les avantages et les inconvénients de ces systèmes, il est possible d’établir une liste des caractéristiques que devrait posséder un système de transcription des LS. Mais développer un système qui réunit toutes ces spécificités est un défi que les linguistes ne peuvent pas relever seuls : c’est dans la pluridisciplinarité, et en particulier dans le travail conjoint avec les spécialistes des aspects graphiques de l’écriture, que semble résider la solution. Cette approche, dite « grapholinguistique », est suivie par l’équipe GestualScript pour développer Typannot, un système de transcription typographique des LS et de la gestualité co-verbale. (shrink)

Argumentation is the gear of a cognitive process of reconstruction when it manifests itself in the discursive proximal space. The cognitive-semiotic perspective of multimodal argumentation suggests that, depending on the knowledge, the codes and signs employed (modes) and the context, certain multimodal structures can lead some audiences towards an accurate mental representation of argumentation, without the need forany formal standardized and verbal structure. We are able to argue with other(s) effectively without the need for a verbal translation of (...) the contents in a space that includes the semiotic interpretation of the message. (shrink)

This paper is a multimodal critical discourse study of other-representation in ISIS’s magazine, Dabiq, It focuses on both the micro-level analysis of actor and action representation, and the macro-structure of negative other-depiction in Dabiq from both textual and visual perspectives. Through in-depth examination of linguistic and non-linguistic elements, the study aimed to unfold ISIS’s ideology at the global level, which is to construct its desired reality and eventually to recruit supporters. The analysis was carried out on fifteen issues of (...) Dabiq following a conceptual and analytical framework within Multimodal Critical Discourse Analysis employing Van Leeuwen’s Socio-semantic inventory. The results show a considerable number of systematic utilizations of discursive strategies by ISIS to belittle its enemies and downgrade their practices. Throughout Dabiq, the out-group participants are represented either as passive reactants who are afflicted by ISIS’s attacks or as powerless agents whose actions are trivial and carry no significant weight. Ineffectuality of ‘others’ has further been substantiated by the visual analysis. Deactivation of out-groups is closely tied to ISIS’s propaganda of self-alteration to achieve the double aim of casting fear among their adversaries and brainwashing the audience by twisting the reality, and persuading them that the enemies’ chance of winning this war is dim. (shrink)

The British Miners’ Strike of 1984–1985 represents one of the most pivotal periods in British industrial relations. The significance of media stance towards the miners remains a controversial issue today, as attested by recent publications looking back at the strike. Here, authors including miners, journalists and other commentators argue that media coverage of the strike followed a consistently anti-trade union agenda in which the media sought to destabilise the strike. An internal British Broadcasting Corporation report, only recently made public, shows (...) that the BBC themselves had concerns over possible imbalances in their coverage of the so-called ‘Battle of Orgreave’. Despite the weight attached to media coverage in this context, however, surprisingly little research has been conducted from a discourse-analytical perspective to show systematically and empirically how such an agenda may have been manifested across media texts. In this article, drawing on Cognitive Linguistic Critical Discourse Studies, I show how one particular metaphorical framing of the strike, which construed the strike as a war between the State and the National Union of Miners, persisted through the year-long period and consider the potential ideological functions of this framing in media strategies of legitimation. I show how this metaphor featured in linguistic, visual and multimodal forms of media representation. (shrink)

Name der Zeitschrift: Semiotica Jahrgang: 2018 Heft: 225 Seiten: 293-311.

In contrast to symbol-manipulation approaches, Cognitive Linguistics offers a modal rather than an amodal account of meaning in language. From this perspective, the meanings attached to linguistic expressions, in the form of conceptualisations, have various properties in common with visual forms of representation. This makes Cognitive Linguistics a potentially useful framework for identifying and analysing language-image relations in multimodal texts. In this paper, we investigate language-image relations with a specific focus on intersemiotic convergence. Analogous with research on gesture, we (...) extend the notion of co-text images and argue that images and language usages which are proximal to one another in a multimodal text can be expected to exhibit the same or consistent construals of the target scene. We outline some of the dimensions of conceptualisation along which intersemiotic convergence may be enacted in texts, including event-structure, viewpoint, distribution of attention and metaphor. We take as illustrative data photographs and their captions in online news texts covering a range of topics including immigration, political protests, and inter-state conflict. Our analysis suggests the utility of Cognitive Linguistics in allowing new potential sites of intersemiotic convergence to be identified and in proffering an account of language-image relations that is based in language cognition. (shrink)

Pantomime has long been considered distinct from co-speech gesture. It has therefore been argued that pantomime cannot be part of gesture-speech integration. We examine pantomime as distinct from silent gesture, focusing on non-co-speech gestures that occur in the midst of children’s spoken narratives. We propose that gestures with features of pantomime are an infrequent but meaningful component of a multimodal communicative strategy. We examined spontaneous non-co-speech representational gesture production in the narratives of 30 monolingual English-speaking children between the ages (...) of 8- and 11-years. We compared the use of co-speech and non-co-speech gestures in both autobiographical and fictional narratives and examined viewpoint and the use of non-manual articulators, as well as the length of responses and narrative quality. The use of non-co-speech gestures was associated with longer narratives of equal or higher quality than those using only co-speech gestures. Non-co-speech gestures were most likely to adopt character-viewpoint and use non-manual articulators. The present study supports a deeper understanding of the term pantomime and its multimodal use by children in the integration of speech and gesture. (shrink)

This paper examines political protest in 40 purposively sampled internet memes circulated among Nigerian WhatsApp users during the Covid-19 pandemic, with a view to exploring the thematic preoccupation, ideology, and the representation of participants and processes in the memes. The data, which were subjected to qualitative analysis, are examined from a multimodal critical discourse analytic approach. The analysis reveals that the memes are used to protest corruption, perceived government deceit, insecurity, hunger, and inadequate health facilities and other social amenities. (...) These are done in order to project and resist an anti-welfarist ideology and emphasise the negative representation of the government. (shrink)

From a marketing point of view, front labels play a crucial role in the consumers' relatively quick decisionmaking process when buying wine in retail stores. The aim of this paper was to reveal which semiotic code systems, in particular, colors, visual representations, and designs, as well as verbal representations, Austrian wine producers use on the front labels of their red wine bottles. For that purpose, the method of content analysis was applied on a representative corpus of red wine (...) labels. Our empirical study showed that the wine labels analyzed especially appeal to price sensitive customers through the employment of light colors. In addition, wine producers do not seem to apply the “visualtotaste lexicon” as the labels rarely show the color “red.” Furthermore, in a large number of cases the wine producers use their family names as brand names of their wines. Moreover, both the depiction of crests and landscapes and the application of indexical brand names, emphasize the Austrian origin of wines. Finally, the extensive use of English brand names appears to connect wines to the social stereotypes of modernity, prestige, and progress. (shrink)

This article analyzes the construction of female subjectivity in the specific context of audiovisual cyberspaces in Spain dedicated to the struggle against violence against women. Looking at the YouTube channels of two virtual feminist communities that deal with violence against women, the authors analyze how the victim-subject is configured in terms of agency and activism. The authors adopt a multimodal model of studying the sign complexes of the videos as semiotic artifacts that produce meaning. Sign complexes are always engaged (...) because representation is never neutral because what is represented in sign is meant to realize the values and positions of those who make the sign. In this article, the authors understand feminist activism in the fight against violence targeting women as constructing in its discourse not only the activist process, but also the subject of this activism: the victim-subject of gender-based violence. The analysis engages in the discussion about the various ways in which this subject is interpellated by the audiovisual texts in terms of agency. As a result, this study proposes the necessity to devise new ways of articulating this subject as a political and agential one. (shrink)

The aim of this article is to determine the representation of persons with disabilities in the print media advertisements of Costa Rica during the period 2014-2016. In the first place, we analyze the advertisements on disability that have appeared in the three most widely circulated newspapers in Costa Rica, following the Fairclough’s budgets about the representation of social actors, inclusion and exclusion strategies, legitimization and explanation that are made of the PCD in the ads. We determine the actions performed by (...) the PDC in the analyzed texts, the relationships with other people that appear in the announcements. Finally, from a multimodal perspective of Kress and Van Leeuwen, through the visual analysis and its relation with the linguistic one, we analyzed the congruent or incongruent images that are promoted in the advertising discourse. It is concluded that there is an invisibility of this population; since the number of ads on disability is less than 0.0015% and that the speech reproduces stigmas and stereotypes from a social- assistance perspective of the PCD, even though it is the State the main advertiser of the ads. (shrink)

This study shows how three Chilean media construct in their discourse the social actor ‘secondary students’, in a multimodal way. It focuses on the representation of Televisión Nacional de Chile (also known as TVN), El Dínamo and El Ciudadano of this social group which is often suppressed suppressed from the media sphere. It is a qualitative study with a social semiotic approach, following the guidelines of Visual Grammar and the typology of representation of social actors by van Leeuwen (2003). (...) The corpus comprised 15 multimodal texts of 2018 compiled from internet sites, which constitute a semiotic chain of a news event about education. The results suggest that TVN creates the meaning of an ‘aggressor student’, El Dínamo as a ‘student to be reformed’; whereas El Ciudadano, represents him as a social actor who is an ‘active participant’. (shrink)

The information in the working environment of industrial Internet is characterized by diversity, semantics, hierarchy, and relevance. However, the existing representation methods of environmental information mostly emphasize the concepts and relationships in the environment and have an insufficient understanding of the items and relationships at the instance level. There are also some problems such as low visualization of knowledge representation, poor human-machine interaction ability, insufficient knowledge reasoning ability, and slow knowledge search speed, which cannot meet the needs of intelligent and (...) personalized service. Based on this, this paper designs a cognitive information representation model based on a knowledge graph, which combines the perceptual information of industrial robot ontology with semantic description information such as functional attributes obtained from the Internet to form a structured and logically reasoned cognitive knowledge graph including perception layer and cognition layer. Aiming at the problem that the data sources of the knowledge base for constructing the cognitive knowledge graph are wide and heterogeneous, and there are entity semantic differences and knowledge system differences among different data sources, a multimodal entity semantic fusion model based on vector features and a system fusion framework based on HowNet are designed, and the environment description information such as object semantics, attributes, relations, spatial location, and context acquired by industrial robots and their own state information are unified and standardized. The automatic representation of robot perceived information is realized, and the universality, systematicness, and intuition of robot cognitive information representation are enhanced, so that the cognition reasoning ability and knowledge retrieval efficiency of robots in the industrial Internet environment can be effectively improved. (shrink)

Frame Semantics includes context as a central aspect of the theory. Frames themselves can be regarded as a representation of the immediate context against which meaning is to be construed. Moreover, the notion of frame invocation includes context as one possible source of information comprehenders use to construe meaning. As the original implementation of Frame Semantics, Berkeley FrameNet is capable of providing computational representations of some aspects of context, but not all of them. In this article, we present FrameNet (...) Brasil: a framenet enriched with qualia relations and capable of taking other semiotic modes as input data, namely pictures and videos. We claim that such an enriched model is capable of addressing other types of contextual information in a framenet, namely sentence-level cotext and commonsense knowledge. We demonstrate how the FrameNet Brasil software infrastructure addresses contextual information in both database construction and corpora annotation. We present the guidelines for the construction of two multimodal datasets whose annotations represent contextual information and also report on two experiments: the identification of frame-evoking lexical units in sentences and a methodology for domain adaptation in Neural Machine Translation that leverages frames and qualia for representing sentence-level context. Experimental results emphasize the importance of computationally representing contextual information in a principled structured fashion as opposed to trying to derive it from the manipulation of linguistic form alone. (shrink)

To this day, the study of the substratum of thought and its implied mechanisms is rarely directly addressed. Nowadays, systemic approaches based on introspective methodologies are no longer fashionable and are often overlooked or ignored. Most frequently, reductionist approaches are followed for deciphering the neuronal circuits functionally associated with cognitive processes. However, we argue that systemic studies of individual thought may still contribute to a useful and complementary description of the multimodal nature of perception, because they can take into (...) account individual diversity while still identifying the common features of perceptual processes. We propose to address this question by looking at one possible task for recognition of a “signifying sound”, as an example of conceptual grasping of a perceptual response. By adopting a mixed approach combining qualitative analyses of interviews based on introspection with quantitative statistical analyses carried out on the resulting categorization, this study describes a variety of mental strategies used by musicians to identify notes’ pitch. Sixty-seven musicians (music students and professionals) were interviewed, revealing that musicians utilize intermediate steps during note identification by selecting or activating cognitive bricks that help construct and reach the correct decision. We named these elements “mental anchorpoints”. Although the anchorpoints are not universal, and differ between individuals, they can be grouped into categories related to three main sensory modalities – auditory, visual and kinesthetic. Such categorization enabled us to characterize the mental representations that allow musicians to name notes in relationship to eleven basic typologies of anchorpoints. We propose a conceptual framework which summarizes the process of note identification in five steps, starting from sensory detection and ending with the verbalization of the note pitch, passing through the pivotal role of mental anchorpoints and mental representations. We found that musicians use multiple strategies and select individual combinations of mental anchorpoints belonging to these three different sensory modalities both in isolation and in combination. (shrink)

Kripke completeness of some infinitary predicate modal logics is presented. More precisely, we prove that if a normal modal logic above is -persistent and universal, the infinitary and predicate extension of with BF and BF is Kripke complete, where BF and BF denote the formulas pi pi and x x, respectively. The results include the completeness of extensions of standard modal logics such as , and its extensions by the schemata T, B, 4, 5, D, and their combinations. The proof (...) is obtained by extending the correspondence between the representation of modal algebras and the completeness of propositional modal logic to infinite. (shrink)