For humans, the ability to communicate and use language is instantiated not only in the vocal modality but also in the visual modality. The main examples of this are sign languages and gestures. Sign languages, the natural languages of Deaf communities, use systematic and conventionalized movements of the hands, face, and body for linguistic expression. Co-speech gestures, though non-linguistic, are produced in tight semantic and temporal integration with speech and constitute an integral part of language together with (...) speech. The articles in this issue explore and document how gestures and sign languages are similar or different and how communicative expression in the visual modality can change from being gestural to grammatical in nature through processes of conventionalization. As such, this issue contributes to our understanding of how the visual modality shapes language and the emergence of linguistic structure in newly developing systems. Studying the relationship between signs and gestures provides a new window onto the human ability to recruit multiple levels of representation in the service of using or creating conventionalized communicative systems. (shrink)

Although Pessoa et al. imply that many aspects of the filling-in debate may be displaced by a regard for active vision, they remain loyal to naive neural reductionist explanations of certain pieces of psychophysical evidence. Alternative interpretations are provided for two specific examples and a new category of filling-in (of visual detail) is proposed.

Modality specificity in priming is taken as evidence for independent perceptual systems. However, Easton, Greene, and Srinivas (1997) showed that visual and haptic cross-modal priming is comparable in magnitude to within-modal priming. Where appropriate, perceptual systems might share like information. To test this, we assessed priming and recognition for visual and auditory events, within- and across- modalities. On the visual test, auditory study resulted in no priming. On the auditory priming test, visual study resulted in (...) priming that was only marginally less than within-modal priming. The priming results show that visual study facilitates identification on both visual and auditory tests, but auditory study only facilitates performance on the auditory test. For both recognition tests, within-modal recognition exceeded cross-modal recognition. The results have two novel implications for the understanding of perceptual priming: First, we introduce visual and auditory priming for spatio-temporal events as a new priming paradigm chosen for its ecological validity and potential for information exchange. Second, we propose that the asymmetry of the cross-modal priming observed here may reflect the capacity of these perceptual modalities to provide cross-modal constraints on ambiguity. We argue that visual perception might inform and constrain auditory processing, while auditory perception corresponds to too many potential visual events to usefully inform and constrain visual perception. (shrink)

Three priming experiments were conducted to determine how information about the self from different sensory modalities/cognitive domains affects self-face recognition. Being exposed to your body odor, seeing your name, and hearing your name all facilitated self-face recognition in a reaction time task. No similar cross-modal facilitation was found among stimuli from familiar or novel individuals. The finding of a left-hand advantage for self-face recognition was replicated when no primes were presented. These data, along with other recent results suggest the brain (...) processes/represents information about the self in highly integrated ways. (shrink)

The aim of the present study is to evaluate Sonja Foss’s Rhetorical Schema for the Evaluation of Visual Imagery as well as reflect upon several points for further consideration; and finally suggest a renewed triadic approach as a method for analyzing art-relevant visual imagery. The triadic approach to be discussed assumes three correlative layers: the intellectual knowledge, function of the artistic content as the visual representational component, and aesthetics as modality. This study will include the analysis (...) of a print advertisement that used an artwork as its content of visual rhetoric, and this will inform further discussions on the proposed approach. The contribution of this renewed triadic approached to the field of visual rhetoric has the advantage of expanding and possibly improving rhetorical analysis methods of visual imagery. (shrink)

Training can influence behavioral performance and lead to brain reorganization. In particular, training in one modality, for example, auditory, can improve performance in another modality, for example, visual. Previous research suggests that one of the mechanisms behind this phenomenon could be the cross-modal recruitment of the sensory areas, for example, the auditory cortex. Studying expert musicians offers a chance to explore this process. Rhythm is an aspect of music that can be presented in various modalities. We designed (...) an fMRI experiment in which professional pianists and non-musicians discriminated between two sequences of rhythms presented auditorily or visually. Behavioral results showed that musicians performed in both visual and auditory rhythmic tasks better than non-musicians. We found no significant between-group differences in fMRI activations within the auditory cortex. However, we observed that musicians had increased activation in the right Inferior Parietal Lobe when compared to non-musicians. We conclude that the musicians’ superior visual rhythm discrimination is not related to cross-modal recruitment of the auditory cortex; instead, it could be related to activation in higher-level, multimodal areas in the cortex. (shrink)

This article presents a fresh perspective on the interplay among visual, verbal, and auditory modalities, positing that these modalities, as semogenic resources, compete to express dynamic meanings. The theoretical paradigm emphasizes that whether a modality or an element within a modality gets or loses semantic status, it will elicit an additional layer of social meaning to depict a comprehensive picture of a story together with an explicit semiotic meaning. The article adopts a qualitative method to analyze the (...) data, which are drawn from The Good Wife and My Roommate is a Gumiho and annotated in ELAN 6.3. It was found that modal competition can shed light on the dynamic meaning-making processes in semiotic and societal orientations. Modal competition may distort space and time of different stories, and reconstruct a different discursive spatio-temporal dimension in the TV world. It can diversify the dynamic orientations from New to Given in visual, verbal, and auditory texts of multimodal discourses to tell stories. Modal competition provides a lens to understand the multidimensional reality and to appreciate the aesthetics of a modern TV series. (shrink)

ABSTRACTBoth auditory and visual emotional memories can be made less emotional by loading working memory during memory recall. Taxing WM during recall can be modality specific (giving an audit...

The distinction between true belief and knowledge is one of the most fundamental in philosophy, and a remarkable effort has been dedicated to formulating the conditions on which true belief constitutes knowledge. For decades, much of this epistemological undertaking has been dominated by a single strategy, referred to here as the modal approach. Shared by many of the most widely influential constraints on knowledge, including the sensitivity, safety, and anti-luck/risk conditions, this approach rests on a key underlying assumption — the (...) modal profiles available to known and unknown beliefs are in some way asymmetrical. The first aim of this paper is to deconstruct this assumption, identifying its plausibility with the way in which epistemologists frequently conceptualize human perceptual systems as excluding certain varieties of close error under conditions conducive to knowledge acquisition. The second aim of this paper is to then argue that a neural phase phenomenon indicates that this conceptualization is quite likely mistaken. This argument builds on the previous introduction of this neural phase to the context of epistemology, expanding the use of neural phase cases beyond relatively narrow questions about epistemic luck to a much more expansive critique of the modal approach as a whole. (shrink)

In a recent paper, Brogaard presents counter-arguments to the conclusions of an experiment with blindsight subject GR. She argues that contrary to the apparent findings that GR’s preserved visual abilities relate to degraded visual experiences, she is in fact fully unconscious of the stimuli she correctly identifies. In this paper, we present arguments and evidence why Brogaard’s argument does not succeed in its purpose. We suggest that not only is relevant empirical evidence in opposition to Brogaard’s argument, her (...) argument misconstrues necessary criteria to decide whether a conscious experience is visual or not visual. (shrink)

This research delves into the potential of implementing artificial intelligence in architecture. It specifically provides a critical assessment of AI-enabled workflows, encompassing creative ideation, representation, materiality, and critical thinking, facilitated by prompt-based generative processes. In this context, the paper provides an examination of the concept of hybrid human–machine intelligence. In an era characterized by pervasive data bias and engineered injustices, the concept of hybrid intelligence emerges as a critical tool, enabling the transcendence of preconceived stereotypes, clichés, and linguistic prejudices. This (...) paper not only explores the applied and generative capacities of AI-enabled workflows but also suggests fundamental approaches that can enhance the creative process and confront the embedded biases and injustices within data-driven systems. (shrink)

Clinical and laboratory studies have demonstrated that executing a demanding dual-task while recollecting emotional memories weakens the emotional intensity and vividness of these memories. While t...

Restrictivists hold that visual experience only represents low-level properties such as shape, spatial location, motion, color, etc. Expansionists contend that visual experience also represents high-level properties such as being a pine tree. I outline a new approach to support expansionism called the conflict cross-modal argument. What I call the conflict cross-modal effects occur when at least two perceptual systems disagree about some property belonging to a common stimulus, and this disagreement causes a change in the representational and phenomenal (...) content of the perceptual experience associated with one, or both, modalities. The conflict cross-modal argument works by accepting that if a property figures in a conflict cross-modal effect, then prima facie that property is a strong common sensible between the two modalities, and vision and another modality disagree about a property that is high-level for vision. After outlining this argument, and showing how it overcomes two obstacles that face traditional methods employed to defend expansionism, I turn to the well known Mcgurk effect as a case where vision and audition disagree about phoneme properties in order to employ my argument for the conclusion that phoneme properties are represented in visual experience. The upshot is that since phoneme properties are high-level for vision, we have an argument supporting expansionism. (shrink)

Most everyday tasks involve multiple modalities, which raises the question of how the processing of these modalities is coordinated by the cognitive system. In this paper, we focus on the coordination of visual attention and linguistic processing during speaking. Previous research has shown that objects in a visual scene are fixated before they are mentioned, leading us to hypothesize that the scan pattern of a participant can be used to predict what he or she will say. We test (...) this hypothesis using a data set of cued scene descriptions of photo-realistic scenes. We demonstrate that similar scan patterns are correlated with similar sentences, within and between visual scenes; and that this correlation holds for three phases of the language production process (target identification, sentence planning, and speaking). We also present a simple algorithm that uses scan patterns to accurately predict associated sentences by utilizing similarity-based retrieval. (shrink)

The nature of attention is one of the oldest and most central problems in psychology. A huge amount of research has been produced on this subject in the last half century, especially on attention in the visual modality, but a general explanation has remained elusive. Many still view attention research as a field that is fundamentally fragmented. This book takes a different perspective and presents a unified theory of visual attention: the TVA model. The TVA model explains (...) the many aspects of visual attention by just two mechanisms for selection of information: filtering and pigeonholing. These mechanisms are described in a set of simple equations, which allow TVA to mathematically model a large number of classical results in the attention literature. The theory explains psychological and neuroscientific findings by the same equations; TVA is a complete theory of visual attention, linking mind and brain. Aimed at advanced students and professional researchers, Principles of Visual Attention contains a detailed review of the most important research done on attention in vision, spanning cognitive psychology, brain imaging, patient studies, and recordings from single cells in the visual cortex. The book explains the TVA model and shows how it accounts for attentional effects observed across all the research areas described. Principles of Visual Attention offers a uniquely integrated view on a central topic in cognitive neuroscience. (shrink)

Perceptual modalities have been traditionally considered the product of dedicated biological systems producing information for higher cognitive processing. Psychological and neuropsychological evidence is offered which undermines this point of view and an alternative account of modality from the enactive approach to understanding cognition is suggested. Under this view, a perceptual modality is a stable form of perception which is structured not just by the biological sensitivities of the agent, but by their goals and the set of skills or (...) expertise which they are deploying at a given time. Such a view suggests that there is no such thing as an experience that is purely visual, auditory, or otherwise modal and that our attempts to understand consciousness and the mind must be conducted within a framework that provides an account of embodied, goal-directed adaptive coping with the world. (shrink)

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62–83 years, with an auditory task-switching paradigm (...) over four sessions, to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks or not trained allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level. (shrink)

This article approaches sensory modalities as semiotically active factors and organizing principles in meaning-making. The focus will be on the special case where modalities mismatch in film – i.e., the soundtrack and visuals present contradictory meanings. The conflict can be characterized by the concept of synthesis that emerges in theories of Eisenstein, Barthes, Jakobson, Lotman, and cognitivists. The artistic functions of such synthesis will be discussed with the help of examples from selected feature films. In the first place, conflicting modalities (...) are inspected in the light of Juri Lotmanʼs theory of two incompatible, but still complementary languages that make up a mechanism for generating new information. In addition, the prospects of evaluating modality conflicts will be touched upon, dismissing synchrony and redundancy as the scale parameters, but acknowledging Lotmanʼs model of space as a primary modeling system that is capable of representing semantic conflicts. (shrink)

The importance of learning and categorizing social objects and events has become widely acknowledged over a couple of decades. Although findings from field studies have suggested that non-human animals have sophisticated abilities to recognize social objects, there is relatively little experimental evidence on this issue. Some studies have revealed animals’ excellent skills for discriminating visual and auditory social stimuli. However, because of perceptual resemblances among stimuli, it is still not clear that they recognize these objects with conceptual mechanisms that (...) are independent of the perceptual characteristics of the stimuli. At the same time, whether their concepts have an aspect of transferring information from one modality to another has not received much attention. This paper advocates approaches to a cross-modal aspect of concepts as a new framework to solve these problems, and introduces the latest studies on cross-modal representations of social objects in non-humans. (shrink)

Visual imagery is a form of sensory imagination, involving subjective experiences typically described as similar to perception, but which occur in the absence of corresponding external stimuli. We used the Activation Likelihood Estimation algorithm (ALE) to identify regions consistently activated by visual imagery across 40 neuroimaging studies, the first such meta-analysis. We also employed a recently developed multi-modal parcellation of the human brain to attribute stereotactic co-ordinates to one of 180 anatomical regions, the first time this approach has (...) been combined with the ALE algorithm. We identified a total 634 foci, based on measurements from 464 participants. Our overall comparison identified activation in the superior parietal lobule, particularly in the left hemisphere, consistent with the proposed ‘top-down’ role for this brain region in imagery. Inferior premotor areas and the inferior frontal sulcus were reliably activated, a finding consistent with the prominent semantic demands made by many visual imagery tasks. We observed bilateral activation in several areas associated with the integration of eye movements and visual information, including the supplementary and cingulate eye fields (SCEFs) and the frontal eye fields (FEFs), suggesting that enactive processes are important in visual imagery. V1 was typically activated during visual imagery, even when participants have their eyes closed, consistent with influential depictive theories of visual imagery. Temporal lobe activation was restricted to area PH and regions of the fusiform gyrus, adjacent to the fusiform face complex (FFC). These results provide a secure foundation for future work to characterise in greater detail the functional contributions of specific areas to visual imagery. (shrink)

In this paper we offer a theory of cross-modal objects. To begin, we discuss two kinds of linkages between vision and audition. The first is a duality. The the visual system detects and identifies surfaces ; the auditory system detects and identifies sources . Surfaces are illuminated by sources of light; sound is reflected off surfaces. However, the visual system discounts sources and the auditory system discounts surfaces. These and similar considerations lead to the Theory of Indispensable Attributes (...) that states the conditions for the formation of gestalts in the two modalities. The second linkage involves the formation of audiovisual objects, integrated cross-modal experiences. We describe research that reveals the role of cross-modal causality in the formation of such objects. These experiments use the canonical example of a causal link between vision and audition: a visible impact that causes a percussive sound. (shrink)