Analysing film. Distinguishing the filmic contribution to meaning -- Examples of filmic "textual organisation" -- Redrawing boundaries -- Organisation of the book -- Semiotics and documents. Semiotics and its relations to film -- The nature of discourse semantics -- The film as cinematographic document -- A combined view: filmic documents for filmic discourse -- Constructing the semiotic mode of film. Semiotic multimodality -- The internal organisation of semiotic strata -- Composing and combining semiotic modes -- Materiality and "epistemological commitment" -- (...) Christian Metz and the grande syntagmatique of the image track. The original model -- Two examples of analysis with the grande syntagmatique -- Revisions and rebuttals -- Foundations for analysis: filmic units. The basic units of film: preliminaries -- Audiovisual iconic representations -- Perception, perceptual realism and reliable measurement -- Multiplicity: from perception to discourse -- Filmic units revisited: discourse-motivated definitions -- The paradigmatic organisation of film. Beyond Metz: towards a grande paradigmatique -- Capturing discourse dependency structures in film -- The paradigmatic dimensions of projection, taxis and plane -- Two examples of paradigmatic analysis -- Summary and conclusions. (shrink)

When I am looking at my coffee machine that makes funny noises, this is an instance of multisensory perception – I perceive this event by means of both vision and audition. But very often we only receive sensory stimulation from a multisensory event by means of one sense modality, for example, when I hear the noisy coffee machine in the next room, that is, without seeing it. The aim of this paper is to bring together empirical findings about multimodal (...) perception and empirical findings about (visual, auditory, tactile) mental imagery and argue that on occasions like this, we have multimodal mental imagery: perceptual processing in one sense modality (here: vision) that is triggered by sensory stimulation in another sense modality (here: audition). Multimodal mental imagery is not a rare and obscure phenomenon. The vast majority of what we perceive are multisensory events: events that can be perceived in more than one sense modality – like the noisy coffee machine. And most of the time we are only acquainted with these multisensory events via a subset of the sense modalities involved – all the other aspects of these multisensory events are represented by means of multisensory mental imagery. This means that multisensory mental imagery is a crucial element of almost all instances of everyday perception. (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, (...) pragmatics, and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

This book envisions a relational and multimodal conceptualisation of higher education, which advocates for knowledge to be analyzed under three relational dimensions: social, digital and environmental. The volume draws on interdisciplinary approaches grounded in Peirceian semiotics, in exploring an integration of these dimensions in higher education theory and practice. The book situates learning in an awareness of the environment, grounded in principles of interconnectedness and flattened hierarchies. The volume features practical case studies through dialogues with higher education teachers, presenting (...) different ways of practicing multimodal and relational higher education which prioritize and embrace expression and creation through more than one dominant modality of communication. The book highlights how a relational multimodal paradigm can serve as a way forward for universities to address global challenges concerning social, technological, and environmental futures through a social justice lens. This innovative book will be of interest to students and scholars in semiotics, multimodality, and higher education, as well as stakeholders in these fields. (shrink)

We present an embedding of quantified multimodal logics into simple type theory and prove its soundness and completeness. A correspondence between QKπ models for quantified multimodal logics and Henkin models is established and exploited. Our embedding supports the application of off-the-shelf higher-order theorem provers for reasoning within and about quantified multimodal logics. Moreover, it provides a starting point for further logic embeddings and their combinations in simple type theory.

We introduce a multimodal logic for order of magnitude reasoning which considers a new logic-based alternative to the notion of closeness, we provide an axiom system and prove its soundness and completeness.

During the last decade, one source of debate in argumentation theory has been the notion that there are different modes of arguing that need to be distinguished when analyzing and evaluating arguments. Visual argument is often cited as a paradigm example. This paper discusses the ways in which it and modes of arguing that invoke non-verbal sounds, smells, tactile sensations, music and other non-verbal entities may be defined and conceptualized. Though some attempts to construct a ‘multimodal’ theory of argument (...) are criticized, it advocates for an argumentation theory that makes room for visual arguing and for other non-verbal modes that have not been explored in depth. In the process, the paper provides a method for identifying the structure of multimodal arguments and argues that adding modes to our theoretical tool box is an important step toward a comprehensive account of argument. (shrink)

We study straightforward embeddings of propositional normal multimodal logic and propositional intuitionistic logic in simple type theory. The correctness of these embeddings is easily shown. We give examples to demonstrate that these embeddings provide an effective framework for computational investigations of various non-classical logics. We report some experiments using the higher-order automated theorem prover LEO-II.

‘다중양상성’은 다양한 맥락에서 상이한 의미로 이해된다. 이 글에서 ‘다중양상성’은 음악의 미적 경험에서 지각과 의식이 신경 토대 연결망을 통해 인지적으로 동기화(synchronization)되는 방식을 가리킨다. 이 과정을 도식적으로 말하면, 우선 시각, 청각, 촉각 등 다양한 감각 자료들이 뇌의 피질 안에 분산된 전문 모듈들에 의해 신경적으로 처리‧저장된다. 심적 상태가 생물학적으로 저장되는 것이다. 이후 어떤 시점에서, 주체가 음악의 특수한 청각적 국면에 주목할 때 비청각적인 신경적 자료들은 활성화되어 중앙 의식에 합류한다. 즉 중앙 의식의 요구에 부응하여 생물학적 데이터가 심적 상태로 변환되고 현재화되며, 이에 따라 주체의 청각 경험은 (...) 상이한 지각 양상들과 앙상블을 이룬다. 이 단계에서 기억과 사고가 참여하여 주도적으로 서사적 해석을 구성할 수도 있으나 이것은 주체의 관심사에 달린 문제이다.BR 본 논의의 직접적인 초점은 음악에 있지만, 다중양상성 논제는 예술 일반에 확대 적용될 수 있다. 왜냐하면, 첫째로, 다중양상성 논제는 매체 지각의 다양성을 뜻하는 것이 아니라 예술 감상자가 작품 전체의 형식으로서 투사하는, 자기 표상적인 의식에 다름 아니기 때문이다. 둘째로, 예술 감상자가 미적 주목을 통해 가상적으로 투사하는 통합적인 작품 형식은 그가 현재 의식적으로 주목하는 작품의 지각적 면모들뿐만 아니라, 그 자료들을 추상하여 전체 형식으로 구성하는 과정에 개입하는 상이한 양태의 ‘지각 흔적’들과 개념적 사고틀이 산출하는 다중양상적 앙상블이기 때문이다. 이렇게 인지적으로 동기화된 앙상블을 가능하게 하는 것은 다름 아니라 의식 아래 차원에서 작동하는 다양한 지각 모듈들과 그것들을 연결해주는 비의식적인 신경 연결망이다. 따라서 비록 ‘신경미학’이 고차적 사고, 지향성, 예술사적 지식 등이 어떻게 예술 경험에 관여하는지를 직접 설명하기 어렵지만, 적어도 신경 토대가 어떻게 미적 경험의 대상인 자기 표상적 의식의 형성을 가능하게 하는지를 보여줄 수 있다. (shrink)

The aim of this paper is to argue that our experience of artworks is normally multimodal. It is the result of perceptual processing in more than one sense modality. In other words, multimodal experience of art is not the exception; it is the rule. I use the example of music in order to demonstrate the various ways in which the visual sense modality influences the auditory processing of music and conclude that this should make us look more closely (...) at our practices of engaging with artworks. (shrink)

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a set (...) of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition. (shrink)

Digitization of every daily procedure requires trustworthy verification schemes. People tend to overlook the security of the passwords they use, i.e. they use the same password on different occasions, they neglect to change them periodically or they often forget them. This raises a major security issue, especially for elderly people who are not familiar with modern technology and its risks and challenges. To overcome these drawbacks, biometric factors were utilized, and nowadays, they have been widely adopted due to their convenience (...) of use and ease of hardware installation. Many biometric-based authentication schemes were proposed, but despite the advantages that they offer, recent research has shown that biometrics by itself cannot be considered as an inviolable technique. Recently, we have proposed StegoPass, a novel method that obtains the 68 facial points of a user and utilizes them as a stego message to an image. This produced stego image was the password. Although the experiments conducted showed maximum security, it would be challenging to enhance the robustness of the proposed model for even more attacks. This paper examines the utilization of multimodal biometrics as the secret message embedded in the image. More specifically, besides the extraction of the facial points, we extract the unique minutiae moments and combine them in a feature vector. This feature vector is then embedded in the image. Two different datasets were used, and the security of the method was tested against state-of-the-art deep learning models, i.e. generative adversarial networks, to test whether the image could be digitally synthesized and fool the proposed verification scheme. The results proved that the new enhanced version of StegoPass offers an extremely secure method as its predecessor. (shrink)

One way to characterize the special relation that one has to one's own body is to say that only one's body appears to one from the inside. Although widely accepted, the nature of this specific experiential mode of presentation of the body is rarely spelled out. Most definitions amount to little more than lists of the various body senses (including senses of posture, movement, heat, pressure, and balance). It is true that body senses provide a kind of informational access to (...) one's own body, which one has to no other bodies, by contrast to external senses like vision, which can take many bodies as their object. But a theory of bodily awareness needs to take into account recent empirical evidence that indicates that bodily awareness is infected by a plague of multisensory effects, regardless of any dichotomy between body senses and external senses. Here I will argue in favour of a multimodal conception of bodily awareness. I will show that the body senses fail to fully account for the content of bodily experiences. I will then propose that vision helps compensate for the insufficiencies of the body senses in people who can see. I will finally argue that the multimodality of bodily experiences does not prevent privileged access to one's body. (shrink)

ABSTRACTThis article offers the first large-scale study of a multimodal corpus of 210 advertisements. First, the reader is presented with a description of the corpus in terms of the distribution of conceptual operations and use of modal cues. Subsequently, the weight of mode and marketing strategy to trigger more or less amounts of conceptual complexity is analyzed. This corpus-based survey is complemented with the qualitative analysis of three novel metaphor–metonymy interactions that stem from the data and that have not (...) yet been surveyed in multimodal use. The results show that metaphtonymy is the most frequet conceptual operation in the corpus; that there is a significant effect of the use of modes in the activation of different amounts of conceptual complexity; and that the type of advertised product and the marketing strategy has no significant effect on the number and complexity of conceptual mappings in the advert... (shrink)

This is a philosophical study by Lithuanian authors on issues related to how to teach philosophy, especially moral philosophy, through films, paintings, images, etc. The topics include multimodality as a synthesis; semiotics and language and image; cinema and philosophical education; postructuralism; film education; value education through spiritual cinema; Eastern Ethics for Western students through multimodal education; philosophy for children; sound and multimodality; Pedagogy of aesthetic to eco-pedagogy, etc.

The article investigates a multimodal practice for self-selecting observed in a video-taped corpus of work meetings: the use of pointing gestures predicting possible turn completions and projecting the emergence of possible next speakers. This practice is analyzed in various sequential positions, namely at turn beginnings and at pre-beginnings. It displays recipients' practical online turn parsing, and their orientation to transition spaces, and to TCU, completions in a visible, recognizable, public way. It shows the emergent and progressive establishment of speakership, (...) exploiting both systematic features of turn-taking and specific features of the interactional space. The article explores the emic, locally situated and contingent definition of speakership by considering not only where pointing gestures begin but also where they end: pointing does not only end in pre-completion positions, projecting turn end, but can also persist through sequences, showing speaker's orientations to her turn's sequential implicativeness. Thus, by observing a particular multimodal practice within a specific interactional setting, the article explores participants' orientations to the rights and obligations associated with talk-related categories such as `non current speaker', `possible next speaker', `incipient speaker' and `current speaker'. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to (...) class='Hi'>multimodal experiences of pain. Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

This book presents the methodological framework of combining Multimodal Conversation Analysis (MCA) with Interpretative Phenomenological Analysis (IPA) to interpretively analyse translanguaging practices in educational contexts. Beginning with an overview of the three uses of translanguaging - translanguaging as a theory of language, as a pedagogical practice and as an analytical perspective - the book goes on to critically examine the different methodological approaches for analysing translanguaging practices in multilingual classroom interactions. It explains how MCA and IPA are useful methodologies (...) for understanding how and why translanguaging practices are constructed by participants in the classroom and discusses types of data collected and data collection procedures. The author, Kevin W. H. Tai, shows how combining these approaches enables researchers to study how translanguaging practices are constructed in multilingual classrooms and how teachers make sense of their own translanguaging practices at particular moments of classroom interaction. This detailed and concise guide is indispensable for students, practitioners, policymakers, and researchers from across the globe, particularly those working in the fields of applied linguistics and language education. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the (...) class='Hi'>multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

This paper shows that some classes of multimodal paraconsistent logics endowed with weak forms of negation are incompletable with respect to Kripke semantics. The reach of such incompleteness is discussed, and we argue that this shortcoming, more than just a logical predicament, may be relevant for attempts to characterize quantum logics and to handle quantum information and quantum computation.

There has been a lot of discussion about how the cognitive penetrability of perception may or may not have important implications for understanding perceptual justification. The aim of this paper is to argue that a different set of findings in perceptual psychology poses an even more serious challenge to the very idea of perceptual justification. These findings are about the importance of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality (such as amodal completion (...) and multimodal completion). I argue that these findings show that everyday perception is in fact a mixture of sensory stimulation-driven perceptual processing and perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality and we have strong reasons to doubt the epistemic pedigree of the latter process. The implication of this is not that we should become skeptics or deny the possibility of perceptual justification. It is, rather, that the only way in which we can understand when and whether a perceptual state justifies beliefs is by paying close attention to empirical facts about the reliability of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality. In this sense (a very narrow sense) epistemology needs to be naturalized. (shrink)

Taste is a central sense for humans and animals, and it has been largely studied either from physiological and neurological approaches or from socio-cultural ones. This paper adopts another view, focused on the activity of tasting rather than on the sense of taste, approached within the perspective of ethnomethodology and multimodal conversation analysis. This view addresses the activity of tasting as it is interactionally organized in specific social settings, observed in a naturalistic way, on the basis of video recordings. (...) Focusing on video recorded improvised tastings of cheese in gourmet shop encounters, the paper offers a systematic analysis of the way in which tasting is orderly achieved in an intersubjective way. It follows the various steps characterizing tasting, from the invitation to taste, to the grasping of a bit to taste, which is put in the mouth, chewed, and swallowed; it details how an interactional moment offering the taster a priviledged, individual, focused space in which to devote exclusive attention to the object tasted is actively tailored by all parties. By contrast, the completion of tasting is marked by a return to mutual gaze, the animation of facial expressions and nods, and the final production of a judgment of taste. By offering a systematic reconstruction of how these tasting moments are organized, the paper invites to a multimodal approach of sensoriality in social interaction. (shrink)

Sensory Individuals: Unimodal and Multimodal Perspectives provides an interdisciplinary, well-balanced, and comprehensive look at different aspects of unisensory and multisensory objects, using both nuanced philosophical analysis and informed empirical work. The research presented in this book represents the field's progression from treating neural sensory processes as primarily modality-specific towards its current state of the art, according to which perception, and its supporting neural processes, are multi-modal, modality-independent, meta-modal, and task-dependent. Even within such approaches sensory stimuli, properties, brain activations, and (...) corresponding perceptual phenomenology can still be characterized in a modality-specific way. The book examines the basic building blocks of human perception, and whether they are best understood as sense modality dependent units of different forms or multimodal perceptual objects. The book combines a variety of innovative and integrative angles to explore the topic and acts as a catalyst for an increasingly diverse field of research, which is in an exciting phase of growth and advancement. New questions are arising as quickly as they are being answered, and the collection Sensory Individuals provides an original and up-to-date addition to the field. (shrink)

The 9/11 attacks in the USA had profound political consequences at both domestic and international levels. Specific and controversial policy developments were pursued requiring substantial legitimation to find acceptance. A prime example was the USA PATRIOT Act, which was passed in the immediate aftermath of 9/11 and subsequently received considerable critique due to the sweeping nature of its redefinition of what was acceptable in the cause of ‘fighting terror’. The media, and their construal of events and policies, played a significant (...) role in the necessary process of legitimation. In this paper, we explore how one particular area of the media, the superhero comic book, was also active in this regard. By focusing specifically on the multimodal meaning construction within Marvel's Civil War comics series, we suggest how a close multimodal discourse analysis can shed further light on the mechanisms by which particular discourse contexts of public opinion are propagated. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) (...) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /, *, \, together with a package of structural postulates characterizing the resource management properties of the * connective. Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke (...) style interpretation in terms of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems each have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems.The first type is obtained by combining a number of unimodal systems into one multimodal logic. The combined multimodal logic is set up in such a way that the individual resource management properties of the constituting logics are preserved. But the inferential capacity of the mixed logic is greater than the sum of its component parts through the addition of interaction postulates, together with the corresponding interpretive constraints on frames, regulating the communication between the component logics.The second type of mixed system is obtained by generalizing the residuation scheme for binary connectives to families of n-ary connectives, and by putting together families, of different arities in one logic. We focus on residuation for unary connectives, hence on mixed frames, as these already represent the complexities in full. We prove a number of elementary logical results for unary families of residuated connectives and their combination with binary families. The existing proposals for unary ‘structural modalities’ are situated within this general framework, and a number of new linguistic applications is given. (shrink)

A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several (...) lines of interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual and multimodal. Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario. (shrink)

Metaphor pervades discourse and may govern how we think and act. But most studies only discuss its verbal varieties. This book examines metaphors drawing on combinations of visuals, language, gestures, sound, and music. Investigated texts include advertising, political cartoons, comics, film, songs, and oral communication. Where appropriate, the influence of genre and cultural factors is thematized.

Recently, it has been observed that the usual type-theoretic restrictions are not enough to block certain paradoxes involving two or more predicates. In particular, when we have a self-referential language containing modal predicates, new paradoxes might appear even if there are type restrictions for the principles governing those predicates. In this article we consider two type-theoretic solutions to multimodal paradoxes. The first one adds types for each of the modal predicates. We argue that there are a number of problems (...) with most versions of this approach. The second one, which we favour, represents modal notions by using the truth predicate together with the corresponding modal operator. This way of doing things is not only useful because it avoids multimodal paradoxes, but also because it preserves the expressive capacity of the language. As an example of the sort of theory we have in mind, we provide a type-theoretic axiomatization that combines truth with necessity and knowledge. (shrink)

One of the main limitations of natural language‐based approaches to meaning is that they do not incorporate multimodal representations the way humans do. In this study, we evaluate how well different kinds of models account for people's representations of both concrete and abstract concepts. The models we compare include unimodal distributional linguistic models as well as multimodal models which combine linguistic with perceptual or affective information. There are two types of linguistic models: those based on text corpora and (...) those derived from word association data. We present two new studies and a reanalysis of a series of previous studies. The studies demonstrate that both visual and affective multimodal models better capture behavior that reflects human representations than unimodal linguistic models. The size of the multimodal advantage depends on the nature of semantic representations involved, and it is especially pronounced for basic‐level concepts that belong to the same superordinate category. Additional visual and affective features improve the accuracy of linguistic models based on text corpora more than those based on word associations; this suggests systematic qualitative differences between what information is encoded in natural language versus what information is reflected in word associations. Altogether, our work presents new evidence that multimodal information is important for capturing both abstract and concrete words and that fully representing word meaning requires more than purely linguistic information. Implications for both embodied and distributional views of semantic representation are discussed. (shrink)

Journal Name: Cognitive Linguistics Issue: Ahead of print.

Biometrics is essentially a pattern recognition system that recognizes an individual using their unique anatomical or behavioral patterns such as face, fingerprint, iris, signature etc. Recent researches have shown that many biometric traits are vulnerable to spoof attacks. Moreover, recent works showed that, contrary to a common belief, multimodal biometric systems in parallel fusion mode can be intruded even if only one trait is spoofed. However, most of the results were obtained using simulated spoof attacks, under the assumption that (...) the spoofed and genuine samples are indistinguishable, which may not be true for all biometric traits. In addition, so far vulnerability of multimodal biometric systems in serial fusion mode against spoof attacks has not been investigated. These issues raise a demand to investigate the robustness of multimodal systems under realistic spoof attacks. In this paper, we empirically investigate the performance of serial and parallel biometric fusion modes under realistic spoof attacks. Preliminary empirical results on real biometric systems made up of face, fingerprint and iris confirm that multimodal biometric systems in both fusion modes are not intrinsically robust against spoof attacks as believed so far. In particular, multimodal biometric systems in serial fusion mode can be even less robust than systems in parallel mode. We also experimentally found that incorporating the biometric sample quality in biometric fusion increases the robustness of the multimodal systems against spoof attacks. In the end, we study the trade-off between performance and robustness of the biometric systems under spoof attacks. (shrink)

According to the shared signal hypothesis the impact of facial expressions on emotion processing partially depends on whether the gaze is directed toward or away from the observer. In autism spectrum disorder several aspects of face processing have been found to be atypical, including attention to eye gaze and the identification of emotional expressions. However, there is little research on how gaze direction affects emotional expression processing in typically developing individuals and in those with ASD. This question is investigated here (...) in two multimodal experiments. Experiment 1 required processing eye gaze direction while faces differed in emotional expression. Forty-seven children participated. Their Autism Diagnostic Observation Schedule scores ranged from 0 to 6 in the experiment. Event-related potentials were sensitive to gaze direction and emotion, but emotion processing did not depend on gaze direction. However, for angry faces the gaze direction effect on the N170 amplitude, as typically observed in TD individuals, diminished with increasing ADOS score. For neutral expressions this correlation was not significant. Experiment 2 required explicit emotion classifications in a facial emotion composite task while eye gaze was manipulated incidentally. A group of 22 children with ASD was compared to a propensity score-matched group of TD children. The same comparison was carried out for a subgroup of nine children with ASD who were less trained in social cognition, according to clinician’s report. The ASD group performed overall worse in emotion recognition than the TD group, independently of emotion or gaze direction. However, for disgust expressions, eye tracking data revealed that TD children fixated relatively longer on the eyes of the stimulus face with a direct gaze as compared with averted gaze. In children with ASD we observed no such modulation of fixation behavior as a function of gaze direction. Overall, the present findings from ERPs and eye tracking confirm the hypothesis of an impaired sensitivity to gaze direction in children with ASD or elevated autistic traits, at least for specific emotions. Therefore, we conclude that multimodal investigations of the interaction between emotional processing and stimulus gaze direction are promising to understand the characteristics of individuals differing along the autism trait dimension. (shrink)

Philosophers and cognitive scientists of perception by custom have investigated individual sense modalities in relative isolation from each other. However, perceiving is, in a number of respects, multimodal. The traditional sense modalities should not be treated as explanatorily independent. Attention to the multimodal aspects of perception challenges common assumptions about the content and phenomenology of perception, and about the individuation and psychological nature of sense modalities. Multimodal perception thus presents a valuable opportunity for a case study in (...) mature interdisciplinary cognitive science. This chapter aims to raise these issues against the background of unimodal approaches in the study of perception. It presents some of the central empirical findings concerning multimodality, and it explains the philosophical implications of these findings. Foremost, it aims to encourage and open avenues for future research. (shrink)

Name der Zeitschrift: Semiotica Jahrgang: 2018 Heft: 220 Seiten: 41-67.

This study proposes a multimodal fusion model to account for the cognitive mechanisms involving 56 political cartoons with regard to U.S. beef import issues as reported in two dominant Taiwanese newspapers, the Liberty Times and United Daily News. Specifically, this study claims that multimodal fusion model evolves from two metonymic-metaphoric networks, i.e., related metonymic network and diversified metaphoric network, and combines the conceptual, visual, and verbal modes. Our analysis demonstrates that multimodal fusion is a significant and recurrent (...) representation technique in the genre of political cartoon and has the cognitive function of encapsulating the abstract complex political debates efficiently with irony and humorous effect. Furthermore, our analysis shows the important role of metonymy and demonstrates how metonymies and metaphors are interwoven in the process of multimodal fusion, which underlies the metaphorical mappings of conceptual scenarios related to “POLITICS IS.. (shrink)

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /,,\, together with a package of structural postulates characterizing the resource management properties of the connective.Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke style interpretation in terms (...) of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems earch have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems. (shrink)

Sensory Individuals: Unimodal and Multimodal Perspectives provides an interdisciplinary, well-balanced, and comprehensive look at different aspects of unisensory and multisensory objects, using both nuanced philosophical analysis and informed empirical work. The research presented in this book represents the field's progression from treating neural sensory processes as primarily modality-specific towards its current state of the art, according to which perception, and its supporting neural processes, are multi-modal, modality-independent, meta-modal, and task-dependent. Even within such approaches sensory stimuli, properties, brain activations, and (...) corresponding perceptual phenomenology can still be characterized in a modality-specific way. The book examines the basic building blocks of human perception, and whether they are best understood as sense modality dependent units of different forms or multimodal perceptual objects. The book combines a variety of innovative and integrative angles to explore the topic and acts as a catalyst for an increasingly diverse field of research, which is in an exciting phase of growth and advancement. New questions are arising as quickly as they are being answered, and the collection Sensory Individuals provides an original and up-to-date addition to the field. (shrink)

Many philosophers use findings about sensory substitution devices in the grand debate about how we should individuate the senses. The big question is this: Is “vision” assisted by (tactile) sensory substitution really vision? Or is it tactile perception? Or some sui generis novel form of perception? My claim is that sensory substitution assisted “vision” is neither vision nor tactile perception, because it is not perception at all. It is mental imagery: visual mental imagery triggered by tactile sensory stimulation. But it (...) is a special form of mental imagery that is triggered by corresponding sensory stimulation in a different sense modality, which I call “multimodal mental imagery.”. (shrink)

The question, “What norms are appropriate for the evaluation of the probative merits of visual arguments?” underlies the investigation of this paper. The notions of argument and of multimodal visual argument employed in the study are explained. Then four multimodal visual arguments are analyzed and their probative merits assessed. It turns out to be possible to judge these qualities using the same criteria that apply to verbally expressed arguments. Since the sample is small and not claimed to be (...) representative, this finding can at best be regarded as suggestive for the probative assessment of multimodal visual arguments in general. (shrink)

This paper explores a semiotic notion of body as starting point for bridging biosemiotic with social semiotic theory. The cornerstone of the argument is that the social semiotic criticism of the classic view of meaning as double articulation can support the criticism of language-centrism that lies at the foundation of biosemiotics. Besides the pragmatic epistemological advantages implicit in a theoretical synthesis, I argue that this brings a semiotic contribution to philosophy of mind broadly. Also, it contributes to overcoming the polemic (...) in linguistics between, loosely put, cognitive universalism and cultural relativism. This possibility is revealed by the recent convergence of various semiotic theories towards a criticism of the classic notion of meaning as double articulation. In biosemiotics, the interest to explicate meaning as multiply articulated stems from the construal of Umwelt as relying on the variety of sense perception channels and semiotic systems that a species has at its disposal. Recently, social semiotics developed an unexplored interest for embodiment by starting from the other end, namely the consideration of the modal heterogeneity of meaning. To bridge these notions, I employ the cognitive semantic notion of embodiment and Mittelberg’s cognitive semiotic notion of exbodiment. In light of these, I explore the possible intricacies between the biosemiotic notion of primary modeling system and concepts referring to preconceptual structures for knowledge organization stemming from cognitive linguistics. Further, Mittelberg’s concept of exbodiment allows for a construal of meaning articulation as mediation between the exbodying and embodying directions of mind. (shrink)

Improvising selected tools from Kress and Van Leeuwen’s inter-semiosis framework, this study explores how, between global and local, TV commercials in India often reframe a cultural third space producing new discursive forms and identities. Three commercials from the food and beverage category are selected on the basis of the country of origin of the endorsing company and the patterns of glocalization. Multimodal discourse analysis reveals that the commercials construct the glocal identity in several ways. In the Knorr Soups commercial, (...) the melody of the title song is contemporary pop, whereas the delivery style and crooning are distinctively Western. The language is predominantly Hindi with deft use of selected English words. The Funda Mint commercial casts a rebel voice-support executive who rejects the pseudo-glocal identity and asserts his true identity. In the Cadbury Bournvita Folk commercial, the two boys represent the global and the local in the realm of creativity. The study advances the knowledge of product positioning and brand identity involving the intersemiosis of discursive elements from different cultural spaces within the contemporary Indian consumerist culture. (shrink)