What kinds of mental states can be based on epistemic reasons? The standard answer is only beliefs. I argue that perceptual states can also be based on reasons, as the result of crossmodal interactions. A perceptual state from one modality can provide a reason on which an experience in another modality is based. My argument identifies key markers of the basing relation and locates them in the crossmodal Marimba Illusion (Schutz & Kubovy 2009). The subject’s auditory experience of (...) musical tone duration is based on the reason provided by her visual representation of the length of the musician’s gesture and other stored perceptual principles. (shrink)

In crossmodal identification, a subject token identifies an item perceived in one sensory modality with an item perceived in another sensory modality. Does crossmodal identification always occur in cognition, or does crossmodal identification sometimes take place in perception? This paper argues that crossmodal identification occurs in cognition, and not in perception. Nevertheless, multisensory perception is not unalive to crossmodal identity. Experimental evidence demonstrates that perception is differentially sensitive to the identity of individuals presented to distinct (...) senses. Such sensitivity enhances recognition and improves action. This approach relies on distinguishing crossmodal identification from perceiving crossmodal identity. Perception registers crossmodal identity, but crossmodal identification as such belongs to thought. (shrink)

Previous research has demonstrated that ratings of the perceived pleasantness and quality of odors can be modulated by auditory stimuli presented at around the same time. Here, we extend these results by assessing whether the hedonic congruence between odor and sound stimuli can modulate the perception of odor intensity, pleasantness, and quality in untrained participants. Unexpectedly, our results reveal that broadband white noise, which was rated as unpleasant in a follow-up experiment, actually had a more pronounced effect on participants’ odor (...) ratings than either the consonant or dissonant musical selections. In particular, participants rated the six smells used as being less pleasant and less sweet when they happened to be listening to white noise, as compared to any one of the other music conditions. What is more, these results also add evidence to the existence a close relationship between an odor’s hedonic character and the perception of odor quality. So, for example, independent of the sound condition, pleasant odors were rated as sweeter, less dry, and brighter than the unpleasant odors. These results are discussed in terms of their implications for the understanding of crossmodal correspondences between olfactory and auditory stimuli. (shrink)

The relationship between music and dance can sometimes be a ‘match’, a remarkable fit between the audible manifestation that music is and the visual or kinaesthetic manifestation that dance is. A match between two things seems to require a common measure with respect to which the match obtains. What can this be for two so different phenomena as music and dance? I argue that the most promising answer is: movement. This answer will not be satisfactory unless the movement of music (...) and the movement of dance are the same on some level. I suggest that they are: there are qualities of movement that guide both dancers and musicians when producing dance and music as perceptible phenomena. But the match in qualities is elusive, since the qualities are instantiated in different ways in music and dance, respectively. (shrink)

Odour-sound correspondences provide some of the most fascinating and intriguing examples of crossmodal associations, in part, because it is unclear from where exactly they originate. Although frequently used as similes, or figures of speech, in both literature and poetry, such smell-sound correspondences have recently started to attract the attention of experimental researchers too. To date, the findings clearly demonstrate that the majority of non-synaesthetic individuals associate orthonasally-presented odours with various different sound properties, e.g., pitch, instrument type, and timbre, in (...) a non-random manner. What is more, these auditory-olfactory associations exhibit a number of features that are common to other crossmodal associations, such as their consistency over time, and their bidirectionality. However, the psychological mechanism underpinning these associations in the general population remain unclear, with a number of distinct hypotheses having been put forward over the years. In this chapter, we consider auditory-olfactory associations in art and science, focusing on poetry, music composition, and performance. First, we provide examples of the use of auditory-olfactory synaesthetic metaphors in poetry, from William Shakespeare through to Romanticism, illustrating how crossmodal associations have appeared in literature for centuries. Then, we move on to focus on music composition and performance, describing a number of examples where auditory stimuli have been purposely matched with crossmodally corresponding olfactory and/or visual stimuli. Considering the scientific study of smell-sound correspondences, we review the key psychophysical studies demonstrating that, beyond the artistic context, robust and non-random crossmodal associations are also triggered in the majority of the general population. We discuss a number of hypotheses that have been put forward over the years to account for such associations, focusing on the idea that certain correspondences between olfactory and auditory stimuli are mediated by the emotional character of the component stimuli. (shrink)

The ability to resist distracting stimuli whilst voluntarily focusing on a task is fundamental to our everyday cognitive functioning. Here, we investigated how this ability develops, and thereafter declines, across the lifespan using a single task/experiment. Young children (5–7 years), older children (10–11 years), young adults (20–27 years), and older adults (62–86 years) were presented with complex visual scenes. Endogenous (voluntary) attention was engaged by having the participants search for a visual target presented on either the left or right side (...) of the display. The onset of the visual scenes was preceded – at stimulus onset asynchronies (SOAs) of 50, 200, or 500 ms – by a task-irrelevant sound (an exogenous crossmodal spatial distractor) delivered either on the same or opposite side as the visual target, or simultaneously on both sides (cued, uncued, or neutral trials, respectively). Age-related differences were revealed, especially in the extreme age-groups, which showed a greater impact of crossmodal spatial distractors. Young children were highly susceptible to exogenous spatial distraction at the shortest SOA (50 ms), whereas older adults were distracted at all SOAs, showing significant exogenous capture effects during the visual search task. By contrast, older children and young adults' search performance was not significantly affected by crossmodal spatial distraction. Overall, these findings present a detailed picture of the developmental trajectory of endogenous resistance to crossmodal spatial distraction from childhood to old age and demonstrate a different efficiency in coping with distraction across the four age-groups studied. (shrink)

Many organisms possess multiple sensory systems, such as vision, hearing, touch, smell, and taste. The possession of multiple ways of sensing the world offers many benefits. However, combining information from different senses also poses many challenges for the nervous system. In recent years there has been dramatic progress in understanding how information from the different senses gets integrated in order to construct useful representations of external space. This volume brings together the leading researchers from a broad range of scientific approaches (...) to present the first overview of this central topic in cognitive neuroscience. (shrink)

Despite the robust evidence that congruent background music in the physical store environment positively affects consumer reactions, less is known about its effects in an online context. The present study aims to examine whether congruency via multiple elicited crossmodal correspondences between background music and the online store environment leads to more positive affective, evaluative, and behavioral consumer reactions and to investigate the moderating role of shopping goals on this crossmodal congruency effect. Previous research showed that low task-relevant atmospheric (...) cues like music can have a negative effect on consumers when they visit a website with a purchase goal in mind. An online experiment was conducted with 239 respondents randomly assigned to a shopping goal and a music condition. Our results show that crossmodally incongruent background music leads to more positive consumer reactions for experiential browsers and more negative consumer reactions for goal-directed searchers. Conversely, crossmodally congruent background music has a positive effect on experiential browsers and no adverse effect on goal-directed searchers. Additionally, the presence of crossmodally congruent background music leads to more positive consumer reactions than the presence of crossmodally incongruent background music, independent of the shopping goal. We extend previous research on multisensory congruency effects by showing the added value of establishing congruency between music and the store environment via multiple elicited crossmodal correspondences in the online environment, countering previously found negative effects of low-task relevant atmospheric cues for goal-directed searchers. (shrink)

We can discern two opposing viewpoints regarding synesthesia. According to the first, it is an oddity, an outlier, or a disordered condition. According to the second, synesthesia is pervasive, driving creativity, metaphor, or language itself. Which is it? Ultimately, I favor the first perspective, according to which cross-sensory synesthesia is an outlying condition. But the second perspective is not wholly misguided. My discussion has three lessons. First, synesthesia is just one of a variety of effects in which one sense modality (...) causally impacts and reshapes experience associated with another. These effects are utterly common. However, due to their unfamiliarity, and to their conflict with a widespread conception of the role of the senses in perception and perceptual experience, until recently they have been surprising. Second, synesthesia nevertheless must be distinguished from other inter-modal effects that lead to misperception, such as crossmodal illusions. Third, synesthesia also may be distinguished from the potentially much broader class of synesthetic effects, which could be common across the population and within individuals. (shrink)

Perceptual similarity is one of the most fiercely debated topics in the philosophy and psychology of perception. The documented history of the issue spans all the way from Plato – who regarded similarity as a key factor for human perceptual experience and cognition – through to contemporary psychologists – who have tried to determine whether, and if so, how similarity relationships can be established between stimuli both within and across the senses. Recent research on cross-sensory associations, otherwise known as (...) class='Hi'>crossmodal correspondences – that is, the existence of observable consensual associations, or mappings, between stimuli across different senses – represents an especially interesting field in which to study perceptual similarity. In fact, most accounts of crossmodal association that have been put forward in the literature to date evoke perceptual similarity as a key explanatory factor mediating the underlying association. At the same time, however, these various accounts raise several important theoretical questions concerning the very nature of similarity, with, for example, the sensory, affective, or cognitive underpinnings of similarity judgements remaining unclear. We attempt to shed light on these questions by examining the various accounts of crossmodal associations that have been put forward in the literature. Our suggestion is that perceptual similarity varies from being phenomenologically-based to conceptually-based. In particular, we propose that the nature of the associations underlying similarity judgements – whether these associations are phenomenologically-, structurally-, emotionally-, or conceptually-based – may be represented in a two-dimensional space with associative strength on one axis, and cognitive penetrability on the other. (shrink)

The last couple of years have seen a rapid growth of interest in the study of crossmodal correspondences – the tendency for our brains to preferentially associate certain features or dimensions of stimuli across the senses. By now, robust empirical evidence supports the existence of numerous crossmodal correspondences, affecting people’s performance across a wide range of psychological tasks – in everything from the redundant target effect paradigm through to studies of the Implicit Association Test, and from speeded discrimination/classification (...) tasks through to unspeeded spatial localisation and temporal order judgment tasks. However, one question that has yet to receive a satisfactory answer is whether crossmodal correspondences automatically affect people’s performance , as opposed to reflecting more of a strategic, or top-down, phenomenon. Here, we review the latest research on the topic of crossmodal correspondences to have addressed this issue. We argue that answering the question will require researchers to be more precise in terms of defining what exactly automaticity entails. Furthermore, one’s answer to the automaticity question may also hinge on the answer to a second question: Namely, whether crossmodal correspondences are all ‘of a kind’, or whether instead there may be several different kinds of crossmodal mapping . Different answers to the automaticity question may then be revealed depending on the type of correspondence under consideration. We make a number of suggestions for future research that might help to determine just how automatic crossmodal correspondences really are. (shrink)

There is a task-specificity in the visual-tactile interaction for perception: The polymodal posterior parietal cortex is related to the comparison of the shapes coded by different sensory modalities, whereas the lateral occipital complex is the part of the network for multimodal shape identification. These interactions may be mediated by some latent pathways potentiated by sensory deprivation or learning.

Social interactions in daily life necessitate the integration of social signals from different sensory modalities. In the aging literature, it is well established that the recognition of emotion in facial expressions declines with advancing age, and this also occurs with vocal expressions. By contrast, crossmodal integration processing in healthy aging individuals is less documented. Here, we investigated the age-related effects on emotion recognition when faces and voices were presented alone or simultaneously, allowing for crossmodal integration. In this study, (...) 31 young adults (M = 25.8 years) and 31 older adults (M = 67.2 years) were instructed to identify several basic emotions (happiness, sadness, anger, fear, disgust) and a neutral expression, which were displayed as visual (facial expressions), auditory (nonverbal affective vocalizations) or crossmodal (simultaneous, congruent facial and vocal affective expressions) stimuli. The results showed that older adults performed slower and worse than younger adults at recognizing negative emotions from isolated faces and voices. In the crossmodal condition, although slower, older adults were as accurate as younger except for anger. Importantly, additional analyses using the ‘race model’ demonstrate that older adults benefited to the same extent as younger adults from the combination of facial and vocal emotional stimuli. These results help explain some conflicting results in the literature and may clarify emotional abilities related to daily life that are partially spared among older adults. (shrink)

Linguistic synesthesia (or synesthetic/intrafield/crossmodal metaphor) refers to crossmodal instances in which expressions in different sensory modalities are combined as in the case of sweet (taste) melody (hearing). Ullmann was among the first to show that synesthetic transfers seem to follow a potentially universal hierarchy that goes from the so-called “lower” (i.e., touch, taste and smell) to “higher” senses (i.e., hearing and sight). Several studies across languages, cultures, domains and text types seem to support the hierarchy in linguistic synesthesia (...) despite some crosslinguistic differences and varying explanations . To extend results to an underrepresented language and thus, to test the universality of the crossmodal hierarchy, 5,693 token cases of linguistic synesthesia in written and spoken Turkish were investigated using a general-purpose, large corpus. Token, type, and hapax legomena frequencies showed that although three backward and thus, hierarchy-inconsistent transfers (i.e., from taste to touch, from sight to smell and from sight to hearing) were more frequent than their hierarchy-consistent counterparts, forward transfers in the canonical direction were more frequent overall than the backward transfers in Turkish. We conclude that Turkish linguistic synesthesia complies with the hierarchy as a descriptive generalization. Results are discussed in comparison to the crossmodal use of sensory words in other languages. (shrink)

Three experiments investigated the nature of visuo-auditory crossmodal cueing in a triadic setting: participants had to detect an auditory signal while observing another agent’s head facing one of the two laterally positioned auditory sources. Experiment 1 showed that when the agent’s eyes were open, sounds originating on the side of the agent’s gaze were detected faster than sounds originating on the side of the agent’s visible ear; when the agent’s eyes were closed this pat-tern of responses was reversed. Two (...) additional experiments showed that the results were sensitive to whether participants could infer a hearing function on the part of the agent. When no ear was depicted on the agent, only a gaze-side advantage was observed , but when the agent’s ear was covered , an ear side advantage was observed only when hearing could still be inferred but not when hearing was inferred to be diminished . The findings are discussed in the context of inferential and simulation processes and joint attention mechanisms. (shrink)