Name der Zeitschrift: Semiotica Jahrgang: 2018 Heft: 220 Seiten: 123-153.

ABSTRACTThis article offers the first large-scale study of a multimodal corpus of 210 advertisements. First, the reader is presented with a description of the corpus in terms of the distribution of conceptual operations and use of modal cues. Subsequently, the weight of mode and marketing strategy to trigger more or less amounts of conceptual complexity is analyzed. This corpus-based survey is complemented with the qualitative analysis of three novel metaphor–metonymy interactions that stem from the data and that have (...) not yet been surveyed in multimodal use. The results show that metaphtonymy is the most frequet conceptual operation in the corpus; that there is a significant effect of the use of modes in the activation of different amounts of conceptual complexity; and that the type of advertised product and the marketing strategy has no significant effect on the number and complexity of conceptual mappings in the advert... (shrink)

Concern for global warming, climate change and pollution has grown in recent years, with countries across the world facing natural disasters on unprecedented scales. The communication of environmental protection is therefore a necessary area of enquiry, especially from a Conceptual Metaphor Theory perspective. The present article explores (1) how the themes of global warming, climate change, pollution and activism are conceptualized in a corpus of 51 noncommercial advertisements, (2) the interaction of metonymy with metaphor, (3) the distribution across verbal (...) and visual modes of metaphoric source and target domains, and (4) how value is evoked. Findings show that half of the corpus frames environmental themes through source domains such as weapons, predators and natural disasters. The other half triggers incongruous mappings, such as between concrete entities, and relies on metonymic processes of inference to access the main rhetorical message. Among the most frequent metonymies, CAUSE-EFFECT and CATEGORY FOR SALIENT PROPERTY highlight the negative effects of the represented phenomena. Multimodality usually occurs within source and/or target domains rather than across the metaphoric mapping. Most of the campaigns project mixed value, where a negative evaluation of a theme is accompanied by a positive message, inviting the audience to take action. (shrink)

This study proposes a multimodal fusion model to account for the cognitive mechanisms involving 56 political cartoons with regard to U.S. beef import issues as reported in two dominant Taiwanese newspapers, the Liberty Times and United Daily News. Specifically, this study claims that multimodal fusion model evolves from two metonymic-metaphoric networks, i.e., related metonymic network and diversified metaphoric network, and combines the conceptual, visual, and verbal modes. Our analysis demonstrates that multimodal fusion is a significant and recurrent (...) representation technique in the genre of political cartoon and has the cognitive function of encapsulating the abstract complex political debates efficiently with irony and humorous effect. Furthermore, our analysis shows the important role of metonymy and demonstrates how metonymies and metaphors are interwoven in the process of multimodal fusion, which underlies the metaphorical mappings of conceptual scenarios related to “POLITICS IS.. (shrink)

The aim of the present paper is to contribute to a better understanding of the role of heraldry, in particular of para-heraldic devices known as “badges”, in 15th-century England. The case chosen for examination is that of the white rose, one of the major badges of Edward IV. The data consists of four contemporary texts in which Edward is referred to as the “rose”, analysed against the background of the use of the white rose of York as a heraldic device. (...) This includes surviving artefacts ranging from effigies to stained glass to seals and manuscript illuminations, as well as contemporary descriptions and depictions of those artefacts. Using the methodological apparatus of cognitive linguistics, specifically the multimodal metaphor and metonymy analysis, the author examines the interplay between language and heraldry. The results show that while the primary function of the white rose and of other badges employed by Edward IV was to emphasize his heritage and thus invite a metonymic reading, the badge inspired other, metaphorical readings, which were employed rhetorically by his supporters. In this context, the concept of the badge may be reinterpreted as a metaphtonymy. The analysis supports the view of heraldry as an integral element of medieval society. From a semiotic perspective, heraldry should be seen as a dynamic system that could be exploited creatively to suit the needs of its users, which in turn corresponds to the dynamic theory of metaphor. (shrink)

This study explores multimodal metaphors and metonymies in Faust, a German Expressionist silent fiction movie by Murnau. The article combines principles of psychocinematics, an interdisciplinary scientific field of enquiry, with the multimodal metaphor and expressive movement model, which looks into the temporal dynamics of metaphoric meaning-making by movie watchers. It is shown that interrelating both film-analytic approaches provides a deeper and more comprehensive insight into how figurative thought influences psycho-cognitive processes in the moviegoer’s mind as they dynamically unfold (...) in their cinematic contexts. Evidence is provided that Kappelhoff and co-workers’ proposal can be complemented and enriched by psychocinematics findings on the allocation of visual attention and eye fixation, motion and image frame sensing, and mental activity underlying emotional and diagetic experiences. (shrink)

TThis paper aims to evidence the inherently metonymic nature of co-speech gestures. Arguing that motivation in gesture involves iconicity (similarity), indexicality (contiguity), and habit (conventionality) to varying degrees, it demonstrates how a set of metonymic principles may lend a certain systematicity to experientially grounded processes of gestural abstraction and enaction. Introducing visuo-kinetic signs as an umbrella term for co-speech gestures and signed languages, the paper shows how a frame-based approach to gesture may integrate different cognitive/functional linguistic and semiotic accounts of (...) metonymy (e.g., experiential domains, frame metonymy, contiguity, and pragmatic inferencing). The guiding assumption is that gestures metonymically profile deeply embodied, routinized aspects of familiar scenes, that is, the motivating context of frames. The discussion shows how gestures may evoke frame structures exhibiting varying degrees of groundedness, complexity, and schematicity: basic physical action and object frames; more complex frames; and highly abstract, complex frame structures. It thereby provides gestural evidence for the idea that metonymy is more basic and more directly experientially grounded than metaphor and thus often feeds into correlated metaphoric processes. Furthermore, the paper offers some initial insights into how metonymy also seems to induce the emergence of schematic patterns in gesture which may result from action-based and discourse-driven processes of habituation and conventionalization. It exemplifies how these forces may engender grammaticalization of a basic physical action into a gestural marker that shows strong metonymic form reduction, decreased transitivity, and interacting pragmatic functions. Finally, addressing basic metonymic operations in signed lexemes elucidates certain similarities regarding sign constitution in gesture and sign. English and German multimodal discourse data as well as German Sign Language (DGS) are drawn upon to illustrate the theoretical points of the paper. Overall, this paper presents a unified account of metonymy’s role in underpinning forms, functions, and patterns in visuo-kinetic signs. (shrink)

This article focuses on how the human body and body parts such as the legs, abdomen, and torso are conceptualized in Spanish director Pedro Almodóvar's films. The comparative analysis follows a qualitative methodology in which instances of the use of body parts are analyzed within their multimodal, sociocultural, and cinematic contexts. Through camera work and mise-en-scène, the actors' bodies and bodily functions are mapped onto complex cultural, social, and filmic targets within the narrative structure. Ultimately, the author shows how (...) artistic creativity is an activity whose purpose is a reembodiment of the emotion by visual and physical alteration. The new body is an environment onto which traditional views of the body based on folk theories are mapped, and body parts and functions are metonymically highlighted. (shrink)

Journal Name: Cognitive Linguistics Issue: Ahead of print.

"The Body in Language: Comparative studies of Linguistic Embodiment provides new insights into the theory of linguistic embodiment in its universal and cultural aspects. The contributions of the volume offer theoretical reflections on grammaticalization, lexical semantics, philosophy, multimodal communication and - by discussing metaphorization and metonymy in figurative language - on cognitive linguistics in general. Case studies contribute first-hand data on embodiment from more than 15 languages and present findings on the body in language in diverse cultures from (...) various continents. Embodiment fundamentally underlies human conceptualization and the present discussions reveal a wide range of target domains in conceptual transfers with the body as the source domain."--PUblisher. (shrink)

When I am looking at my coffee machine that makes funny noises, this is an instance of multisensory perception – I perceive this event by means of both vision and audition. But very often we only receive sensory stimulation from a multisensory event by means of one sense modality, for example, when I hear the noisy coffee machine in the next room, that is, without seeing it. The aim of this paper is to bring together empirical findings about multimodal (...) perception and empirical findings about (visual, auditory, tactile) mental imagery and argue that on occasions like this, we have multimodal mental imagery: perceptual processing in one sense modality (here: vision) that is triggered by sensory stimulation in another sense modality (here: audition). Multimodal mental imagery is not a rare and obscure phenomenon. The vast majority of what we perceive are multisensory events: events that can be perceived in more than one sense modality – like the noisy coffee machine. And most of the time we are only acquainted with these multisensory events via a subset of the sense modalities involved – all the other aspects of these multisensory events are represented by means of multisensory mental imagery. This means that multisensory mental imagery is a crucial element of almost all instances of everyday perception. (shrink)

Logical metonymy resolution (begin a book begin reading a book or begin writing a book) has traditionally been explained either through complex lexical entries (qualia structures) or through the integration of the implicit event via post-lexical access to world knowledge. We propose that recent work within the words-as-cues paradigm can provide a more dynamic model of logical metonymy, accounting for early and dynamic integration of complex event information depending on previous contextual cues (agent and patient). We first present (...) a self-paced reading experiment on German subordinate sentences, where metonymic sentences and their paraphrased version differ only in the presence or absence of the clause-final target verb (Der Konditor begann die Glasur Der Konditor begann, die Glasur aufzutragen/The baker began the icing The baker began spreading the icing). Longer reading times at the target verb position in a high-typicality condition (baker + icing spread ) compared to a low-typicality (but still plausible) condition (child + icing spread) suggest that we make use of knowledge activated by lexical cues to build expectations about events. The early and dynamic integration of event knowledge in metonymy interpretation is bolstered by further evidence from a second experiment using the probe recognition paradigm. Presenting covert events as probes following a high-typicality or a low-typicality metonymic sentence (Der Konditor begann die Glasur AUFTRAGEN/The baker began the icing SPREAD), we obtain an analogous effect of typicality at 100 ms interstimulus interval. (shrink)

We examine metonymy at psycho- and neurolinguistic levels, seeking to adjudicate between two possible processing implementations. We compare highly conventionalized systematic metonymy to lesser-conventionalized circumstantial metonymy. Whereas these two metonymy types differ in terms of contextual demands, they each reveal a similar dependency between the named and intended conceptual entities. We reason that if each metonymy yields a distinct processing time course and substantially non-overlapping preferential localization pattern, it would not only support a two-mechanism view (...) but would suggest that conventionalization acts as a linguistic categorizer. By contrast, a similar behavior in time course and localization would support a one-mechanism view and the inference that conventionalization acts instead as a modulator of contextual felicitousness, and that differences in interpretation introduced by conventionalization are of degree, not of kind. Results from three paradigms: self-paced reading, event-related potentials, and functional magnetic resonance imaging, reveal the following: no main effect by condition for either metonymy type immediately after the metonymy trigger, and a main effect for only the Circumstantial metonymy one word post-trigger ; a N400 effect across metonymy types and a late positivity for Circumstantial metonymy ; and a highly overlapping activation connecting the left ventrolateral prefrontal cortex and the left dorsolateral prefrontal cortex. Altogether, the pattern observed does not reach the threshold required to justify a two-mechanism system. Instead, the pattern is more naturally understood as resulting from the implementation of a generalized referential dependency mechanism, modulated by degree of context dependence/conventionalization, thus supporting architectures of language whereby “lexical” and “pragmatic” meaning relations are encoded along a cline of contextual underspecification. (shrink)

During the last decade, one source of debate in argumentation theory has been the notion that there are different modes of arguing that need to be distinguished when analyzing and evaluating arguments. Visual argument is often cited as a paradigm example. This paper discusses the ways in which it and modes of arguing that invoke non-verbal sounds, smells, tactile sensations, music and other non-verbal entities may be defined and conceptualized. Though some attempts to construct a ‘multimodal’ theory of argument (...) are criticized, it advocates for an argumentation theory that makes room for visual arguing and for other non-verbal modes that have not been explored in depth. In the process, the paper provides a method for identifying the structure of multimodal arguments and argues that adding modes to our theoretical tool box is an important step toward a comprehensive account of argument. (shrink)

This study describes metonymic argument alternations, in which a constructional slot can be filled by any of a set of semantic roles that index one another, and provides a diachronic corpus analysis of two such alternations in French. In the Reveal secret frame and other communication frames, the Medium can indexically replace the Speaker and the Topic can indexically replace the Information. A regression analysis shows that while topic for information metonymy is more syntactically and pragmatically restricted, medium for (...) speaker metonymy has seen an increase in usage over time across the frame. This change is related to sociocultural developments and has implications for the study of figurative language and lexical semantic change. While it is well understood that figurative mechanisms such as metonymy affect language change, here it is demonstrated that cultural shifts drive changes in metonymies themselves, with corresponding impacts for linguistic structure. (shrink)

It is common to characterize pain with touch-related terms, like ‘cutting’, ‘pressing’, ‘sharp’, and ‘pulsing’, or temperature-related terms, like ‘hot’ or ‘burning’. This suggests that many pains are phenomenally multimodal because they are experienced as having some tactile-like or thermal-like character. The goal of this chapter is to investigate the structure of phenomenally multimodal pain experiences. It is argued that the usual accounts of multimodal structure proposed in investigations regarding exteroceptive experiences cannot be plausibly applied to (...) class='Hi'>multimodal experiences of pain. Instead, an alternative framework is proposed which characterizes the structure of tactile-like and thermal-like pains by referring to the notion of crossmodal correspondences. (shrink)

We present an embedding of quantified multimodal logics into simple type theory and prove its soundness and completeness. A correspondence between QKπ models for quantified multimodal logics and Henkin models is established and exploited. Our embedding supports the application of off-the-shelf higher-order theorem provers for reasoning within and about quantified multimodal logics. Moreover, it provides a starting point for further logic embeddings and their combinations in simple type theory.

One way to characterize the special relation that one has to one's own body is to say that only one's body appears to one from the inside. Although widely accepted, the nature of this specific experiential mode of presentation of the body is rarely spelled out. Most definitions amount to little more than lists of the various body senses (including senses of posture, movement, heat, pressure, and balance). It is true that body senses provide a kind of informational access to (...) one's own body, which one has to no other bodies, by contrast to external senses like vision, which can take many bodies as their object. But a theory of bodily awareness needs to take into account recent empirical evidence that indicates that bodily awareness is infected by a plague of multisensory effects, regardless of any dichotomy between body senses and external senses. Here I will argue in favour of a multimodal conception of bodily awareness. I will show that the body senses fail to fully account for the content of bodily experiences. I will then propose that vision helps compensate for the insufficiencies of the body senses in people who can see. I will finally argue that the multimodality of bodily experiences does not prevent privileged access to one's body. (shrink)

A foundational goal of cognitive linguistics is to explain linguistic phenomena in terms of general cognitive strategies rather than postulating an autonomous language module (Langacker 1987: 12–13). Metonymy is identified among the imaginative capacities of cognition (Langacker 1993: 30, 2009: 46–47). Whereas the majority of scholarship on metonymy has focused on lexical metonymy, this study explores the systematic presence of metonymy in word-formation. I argue that in many cases, the semantic relationships between stems, affixes, and the (...) words they form can be analyzed in terms of metonymy, and that this analysis yields a better, more insightful classification than traditional descriptions of word-formation. I present a metonymic classification of suffixal word-formation in three languages: Russian, Czech, and Norwegian. The system of classification is designed to maximize comparison between lexical and word-formational metonymy. This comparison supports another central claim of cognitive linguistics, namely that grammar (in this case word-formation) and lexicon form a continuum (Langacker 1987: 18–19), since I show that metonymic relationships in the two domains can be described in nearly identical terms. While many metonymic relationships are shared across the lexical and grammatical domains, some are specific to only one domain, and the two domains show different preferences for source and target concepts. Furthermore, I find that the range of metonymic relationships expressed in word-formation is more diverse than what has been found in lexical metonymy. There is remarkable similarity in word-formational metonymy across the three languages, despite their typological differences, though they all show some degree of language-specific behavior as well. Although this study is limited to three Indo-European languages, the goal is to create a classification system that could be implemented (perhaps with modifications) across a wider spectrum of languages. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the (...) class='Hi'>multimodal choices that political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

We study straightforward embeddings of propositional normal multimodal logic and propositional intuitionistic logic in simple type theory. The correctness of these embeddings is easily shown. We give examples to demonstrate that these embeddings provide an effective framework for computational investigations of various non-classical logics. We report some experiments using the higher-order automated theorem prover LEO-II.

As contact languages, pidgins and creoles arise in mixed linguistic environments. Drawing much of their vocabularies from one, frequently European, language and – to a lesser extent – from a number of indigenous languages, they have lexicons that are reduced in comparison with those of their lexifiers. To compensate for the poor lexification, pidgin and creoles create novel polysemy-based extensions of lexical items or develop periphrastic constructions equivalent of the missing lexical roots. Assuming a cognitive linguistic perspective, which emphasizes the (...) role of conceptualization in the construction of meaning and the figurative character of concepts, the paper deals with the lexicons of Nigerian Pidgin English and Tok Pisin, two contact languages representative for the Atlantic and the Pacific – the two major areas of linguistic contact. The analysis focuses on the first of the above-mentioned compensation strategies. It is argued that (i) the expressions borrowed from English and various indigenous languages acquire senses that are absent in English; (ii) the new senses are frequently based on metonymy, which serves as a major polysemy-based strategy of lexicon extension; (iii) most of the novel lexical extensions fall within the metonymic patterns that are well-established in English. (shrink)

The 9/11 attacks in the USA had profound political consequences at both domestic and international levels. Specific and controversial policy developments were pursued requiring substantial legitimation to find acceptance. A prime example was the USA PATRIOT Act, which was passed in the immediate aftermath of 9/11 and subsequently received considerable critique due to the sweeping nature of its redefinition of what was acceptable in the cause of ‘fighting terror’. The media, and their construal of events and policies, played a significant (...) role in the necessary process of legitimation. In this paper, we explore how one particular area of the media, the superhero comic book, was also active in this regard. By focusing specifically on the multimodal meaning construction within Marvel's Civil War comics series, we suggest how a close multimodal discourse analysis can shed further light on the mechanisms by which particular discourse contexts of public opinion are propagated. (shrink)

The aim of this paper is to argue that our experience of artworks is normally multimodal. It is the result of perceptual processing in more than one sense modality. In other words, multimodal experience of art is not the exception; it is the rule. I use the example of music in order to demonstrate the various ways in which the visual sense modality influences the auditory processing of music and conclude that this should make us look more closely (...) at our practices of engaging with artworks. (shrink)

There has been a lot of discussion about how the cognitive penetrability of perception may or may not have important implications for understanding perceptual justification. The aim of this paper is to argue that a different set of findings in perceptual psychology poses an even more serious challenge to the very idea of perceptual justification. These findings are about the importance of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality (such as amodal completion (...) and multimodal completion). I argue that these findings show that everyday perception is in fact a mixture of sensory stimulation-driven perceptual processing and perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality and we have strong reasons to doubt the epistemic pedigree of the latter process. The implication of this is not that we should become skeptics or deny the possibility of perceptual justification. It is, rather, that the only way in which we can understand when and whether a perceptual state justifies beliefs is by paying close attention to empirical facts about the reliability of perceptual processing that is not driven by corresponding sensory stimulation in the relevant sense modality. In this sense (a very narrow sense) epistemology needs to be naturalized. (shrink)

Analysing film. Distinguishing the filmic contribution to meaning -- Examples of filmic "textual organisation" -- Redrawing boundaries -- Organisation of the book -- Semiotics and documents. Semiotics and its relations to film -- The nature of discourse semantics -- The film as cinematographic document -- A combined view: filmic documents for filmic discourse -- Constructing the semiotic mode of film. Semiotic multimodality -- The internal organisation of semiotic strata -- Composing and combining semiotic modes -- Materiality and "epistemological commitment" -- (...) Christian Metz and the grande syntagmatique of the image track. The original model -- Two examples of analysis with the grande syntagmatique -- Revisions and rebuttals -- Foundations for analysis: filmic units. The basic units of film: preliminaries -- Audiovisual iconic representations -- Perception, perceptual realism and reliable measurement -- Multiplicity: from perception to discourse -- Filmic units revisited: discourse-motivated definitions -- The paradigmatic organisation of film. Beyond Metz: towards a grande paradigmatique -- Capturing discourse dependency structures in film -- The paradigmatic dimensions of projection, taxis and plane -- Two examples of paradigmatic analysis -- Summary and conclusions. (shrink)

Since the early 80s, metonymy has progressively gained central stage in linguistic investigations. The advent of cognitive linguistics marked a new turn in the study of this trope conceived, not as a deviation from semantic conventions (contra classical rhetorical theories), but as a phenomenon rooted in non-language-specific mechanisms of conceptualization and structuring of the world. Focusing on the particular case of whole-for-part (WP) metonymy, the general stand of this presentation will be to argue for the need to re-inject (...) properly semantic (descriptive and formal) considerations in the study of metonymy. It will be shown that semantic typologies of meronymic relations, as well as recent results in the semantics of partial predicates and colour predicates, can be jointly exploited to attain a better understanding of the mechanism of WP-metonymy. The resulting view will in particular connect such metonymies to mechanisms of reference and property attribution to functional parts for classificatory purposes. (shrink)

This paper explores the role of metonymy in determining a syntactic argument alternation (“conductor-vehiclealternation”) which occurs in English and Portuguese:o piloto acelerou a Ferrari“the driver speeded up the Ferrari”/a Ferrari acelerou“the Ferrari speeded up/sped away”. Since the verbs in theconductor-vehiclealternation haveconductorandvehiclearguments (controller and controlled entities), a metonymic process can occur, allowing thevehicleexpression to provide access to theconductorparticipant. To explain how metonymy allows a verb with two participants to be integrated into a construction with a single argument, we assume (...) that metonymy gathers information about both entities involved; thevehicleexpression provides mental access to bothvehicleandconductor(“fusion”). We also discuss cognitive and pragmatic factors involving the choice of a construction over another. Constructions withvehicleexpressions as subject are used when thevehicleis salient or theconductoris unknown. This also explains whydirigir“drive” does not alternate in Portuguese, contrarily to prediction and differently from Englishdrive. We provide a comparative account of the behavior of this verb in both languages.Dirigir, differently fromdrive, conceptualizes semantic components incompatible with a situation in which the agent/conductoris not salient or is unknown. This research adds to the ongoing body of literature on the role of metonymy in grammar and is a contribution to the understanding of the metonymic process, as a fusion, and also to argument alternation processes and lexical-constructional integration. (shrink)

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a set (...) of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition. (shrink)

This is a philosophical study by Lithuanian authors on issues related to how to teach philosophy, especially moral philosophy, through films, paintings, images, etc. The topics include multimodality as a synthesis; semiotics and language and image; cinema and philosophical education; postructuralism; film education; value education through spiritual cinema; Eastern Ethics for Western students through multimodal education; philosophy for children; sound and multimodality; Pedagogy of aesthetic to eco-pedagogy, etc.

Journal Name: Cognitive Linguistics Issue: Ahead of print.

What is the correct procedure for determining the contents of perception? Philosophers tackling this question increasingly rely on empirically-oriented procedures in order to reach an answer. I argue that this constitutes an improvement over the armchair methodology constitutive of phenomenal contrast cases, but that there is a crucial respect in which current empirical procedures remain limited: they are unimodal in nature, wrongly treating the senses as isolatable faculties. I thus have two aims: first, to motivate a reorientation of the admissible (...) contents debate into a multimodal framework, charting its various significances. The second is to explore whether any experimental studies of multimodal perception support a so-called Liberal (or ‘high-level’ or ‘rich’) account of perception’s admissible contents. I conclude that the McGurk effect and the ventriloquist effect are both explicable without the postulation of high-level content, but that at least one multimodal experimental paradigm may necessitate such content: the rubber hand illusion. One upshot of this argument is that Conservatives who claim that the Liberal view intolerably broadens the scope of perceptual illusions, particularly from the perspective of perceptual psychology, should pursue other arguments against that view. (shrink)

In this paper, I examine Kumārila Bha ṭṭ a's account of figurative language in Tantravārttika 1.4.11-17, arguing that, for him, both metonymy and metaphor crucially involve verbal postulation, a knowledge-conducive cognitive process which draws connections between concepts without appeal to speaker intention, but through compositional and contextual elements. It is with the help of this cognitive process that we can come to have knowledge of what is meant by a sentence in context. In addition, the paper explores the relationship (...) between metonymy and metaphor, the extent to which putatively literal language involves metonymy, and the objective constraints for metaphorical interpretation. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) (...) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

Critics in both philosophy and literary studies have rightly emphasized a “poetics of transition” relating the thought of Ralph Waldo Emerson to that of William James. However, less attention has been given to the ways that Emerson's philosophy of rhetoric correlates with James's rhetorical perspectives on psychology and philosophy. Fundamentally rhetorical interests in the contiguous circumstances and contingent reception of thinking link James to Emerson beyond matters of poetics and style. This article correlates Emerson's understanding of a rhetoric of (...) class='Hi'>metonymy as the basis of thinking with the principle of contiguity crucial to James's philosophy of mind. This relation between rhetoric and philosophy reiterates a rhetoric of mind that both Emerson and James associate with the older liberal education of the college just at the point that this curriculum is displaced by the professional, specialized disciplines of the emerging university in late nineteenth-century America. (shrink)

Digitization of every daily procedure requires trustworthy verification schemes. People tend to overlook the security of the passwords they use, i.e. they use the same password on different occasions, they neglect to change them periodically or they often forget them. This raises a major security issue, especially for elderly people who are not familiar with modern technology and its risks and challenges. To overcome these drawbacks, biometric factors were utilized, and nowadays, they have been widely adopted due to their convenience (...) of use and ease of hardware installation. Many biometric-based authentication schemes were proposed, but despite the advantages that they offer, recent research has shown that biometrics by itself cannot be considered as an inviolable technique. Recently, we have proposed StegoPass, a novel method that obtains the 68 facial points of a user and utilizes them as a stego message to an image. This produced stego image was the password. Although the experiments conducted showed maximum security, it would be challenging to enhance the robustness of the proposed model for even more attacks. This paper examines the utilization of multimodal biometrics as the secret message embedded in the image. More specifically, besides the extraction of the facial points, we extract the unique minutiae moments and combine them in a feature vector. This feature vector is then embedded in the image. Two different datasets were used, and the security of the method was tested against state-of-the-art deep learning models, i.e. generative adversarial networks, to test whether the image could be digitally synthesized and fool the proposed verification scheme. The results proved that the new enhanced version of StegoPass offers an extremely secure method as its predecessor. (shrink)

The analysis of multimodal argumentation in advertising is a crucial and problematic area of research. While its importance is growing in a time characterized by images and pictorial messages, the methods used for interpreting and reconstructing the structure of arguments expressed through verbal and visual means capture only isolated dimensions of this complex phenomenon. This paper intends to propose and illustrate a methodology for the reconstruction and analysis of “double-mode” arguments in advertisements, combining the instruments developed in social semiotics, (...) pragmatics, and argumentation theory. An advertisement is processed through a five-step path. The analysis of its context, text genre, and images leads to a first representation of the messages that it encodes both pictorially and verbally (step 1). These first semantic representations are further enriched by including their polyphonic articulations and presuppositions (step 2), their explicatures (step 3), and their dialogical functions and illocutionary forces (step 4). These pragmatic steps retrieve the commitment structure of the ad, which allows a further argument analysis conducted through argumentation schemes (step 5). (shrink)

In the first half of the paper I critically review some previous attempts to deal with metonymy. I focus in particular on the classical approach, the associationist approach and the Gricean approach. The main point of my criticisms is that the notion of empirical associations among objects is in itself inadequate for a complete descriptive and explanatory account of metonymy.

We introduce a multimodal logic for order of magnitude reasoning which considers a new logic-based alternative to the notion of closeness, we provide an axiom system and prove its soundness and completeness.

The article investigates a multimodal practice for self-selecting observed in a video-taped corpus of work meetings: the use of pointing gestures predicting possible turn completions and projecting the emergence of possible next speakers. This practice is analyzed in various sequential positions, namely at turn beginnings and at pre-beginnings. It displays recipients' practical online turn parsing, and their orientation to transition spaces, and to TCU, completions in a visible, recognizable, public way. It shows the emergent and progressive establishment of speakership, (...) exploiting both systematic features of turn-taking and specific features of the interactional space. The article explores the emic, locally situated and contingent definition of speakership by considering not only where pointing gestures begin but also where they end: pointing does not only end in pre-completion positions, projecting turn end, but can also persist through sequences, showing speaker's orientations to her turn's sequential implicativeness. Thus, by observing a particular multimodal practice within a specific interactional setting, the article explores participants' orientations to the rights and obligations associated with talk-related categories such as `non current speaker', `possible next speaker', `incipient speaker' and `current speaker'. (shrink)

Name der Zeitschrift: Semiotica Jahrgang: 2018 Heft: 220 Seiten: 41-67.

Irregular self-selection is a demonstration of active involvement in interaction. English as a foreign language learners’ talk-in-interaction is one of such cases. Yet, little research has explored when, how, and why learners implement this action. The aim of this article is to address these issues in Chinese postgraduate EFL learners’ conversations from the perspective of multimodal interaction. To this end, we provide descriptive statistics and use multimodal conversation analysis to investigate the detailed process of irregular self-selection. The results (...) show the interactional sensitivity of learners, and all the successful irregular self-selection can be divided into the following three types: turn interruption, turn competition TC), and turn holding abortion. Learners implement this action by using multimodal resources, including lexis, syntax, pitch reset, intensity enhancement, gaze, and so forth. However, their body movements lack diversity, causing behaviors to be constrained and inactive. The main purpose of irregular self-selection is to provide knowledge that contributes to topical development. This study reveals that Chinese postgraduate EFL learners are interactionally competent members. They are able to achieve communicative goals but in a low diversity of body movements. The findings help to understand the detailed process of speakership claiming in EFL learners’ conversations. (shrink)

Do metonymies play any role in the emergence of metaphors? There is a debate between scholars who suggest that many metaphors are based on, or derive from, metonymies, versus those who do not see such connection between the two. “Resemblance metaphors” do not seem to have anything to do with metonymy. However, in the case of “correlation metaphors” (see, e.g., CitationGrady, 1997a, Citation1997b, Citation1999; CitationLakoff & Johnson, 1980, Citation1999), several researchers argue that metaphors arise from, and are not independent (...) of, metonymies. My specific proposal in the article is that correlation-based metaphors emerge from frame-like mental representations through a metonymic stage. I suggest this happens when one of the elements of a frame-like mental structure is generalized (schematized) to a concept that lies outside the initial frame in a different part of the conceptual system. The generalization process leads to sufficient conceptual distance between the initial and the new frame on which metaphors can be based. (shrink)

Philosophers and cognitive scientists of perception by custom have investigated individual sense modalities in relative isolation from each other. However, perceiving is, in a number of respects, multimodal. The traditional sense modalities should not be treated as explanatorily independent. Attention to the multimodal aspects of perception challenges common assumptions about the content and phenomenology of perception, and about the individuation and psychological nature of sense modalities. Multimodal perception thus presents a valuable opportunity for a case study in (...) mature interdisciplinary cognitive science. This chapter aims to raise these issues against the background of unimodal approaches in the study of perception. It presents some of the central empirical findings concerning multimodality, and it explains the philosophical implications of these findings. Foremost, it aims to encourage and open avenues for future research. (shrink)