This report highlights and explores five questions that arose from the multisensoryintegration workshop at the University of Toronto on May 9th and 10th, 2014: 1. What Is MultisensoryIntegration? 2. Do Multisensory Percepts Involve Emergent Features? 3. What Can Multisensory Processing Tell Us about Multisensory Awareness? 4. Is Language Processing a Special Kind of MultisensoryIntegration? 5. What Is the Purpose of MultisensoryIntegration?
The Bayesian model of multisensory cue integration proposed by Ernst and Banks provides an attractive model for understanding a way that our sensory systems may interact. Moreover, it has been suggested that the process of multisensoryintegration that it models underpins conscious experiences with multisensory representational contents merged across modalities. Should we therefore take empirical support for the Bayesian model as evidence of the multimodality of perception? Focusing on evidence of integration across vision and (...) touch, I argue that apparent support for the model does not warrant the rejection of the view that each of our conscious perceptual experiences is associated with one and only one sense modality. (shrink)
This is an excerpt from a report on the workshop on multisensoryintegration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is multisensoryintegration?
This is an excerpt from a report on the workshop on multisensoryintegration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Do multisensory percepts involve emergent features?
This is an excerpt from a report on the workshop on multisensoryintegration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What can multisensory processing tell us about multisensory awareness?
This is an excerpt from a report on the workshop on multisensoryintegration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is the purpose of multisensoryintegration?
This is an excerpt from a report on the workshop on multisensoryintegration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Is language processing a special kind of multisensoryintegration?
In the social world, multiple sensory channels are used concurrently to facilitate communication. Among human and nonhuman pri- mates, faces and voices are the primary means of transmitting social signals (Adolphs, 2003; Ghazanfar and Santos, 2004). Primates recognize the correspondence between species-specific facial and vocal expressions (Massaro, 1998; Ghazanfar and Logothetis, 2003; Izumi and Kojima, 2004), and these visual and auditory channels can be integrated into unified percepts to enhance detection and discrimination. Where and how such communication signals are integrated (...) at the neural level are poorly understood. In particular, it is unclear what role “unimodal” sensory areas, such as the auditory cortex, may play. We recorded local field potential activity, the signal that best correlates with human imaging and event-related potential signals, in both the core and lateral belt regions of the auditory cortex in awake behaving rhesus monkeys while they viewed vocalizing conspecifics. We demonstrate unequivocally that the primate auditory cortex integrates facial and vocal signals through enhancement and suppression of field potentials in both the core and lateral belt regions. The majority of these multisensory responses were specific to face/voice integration, and the lateral belt region shows a greater frequency of multisensoryintegration than the core region. These multisensory processes in the auditory cortex likely occur via recip- rocal interactions with the superior temporal sulcus. (shrink)
The integration of multisensory information is an essential mechanism in perception and action control. Research in multisensoryintegration is concerned with how the information from the different sensory modalities, such as the senses of vision, hearing, smell, taste, touch, and proprioception, are integrated to a coherent representation of objects (for an overview, see e.g., Calvert, Spence and Stein, 2004). The combination of information from the different senses is central for action control. For instance, when you grasp (...) for a rubber duck, you can see its size, feel its compliance and hear the sound it produces. Moreover, identical physical properties of an object can be provided by different senses. You can both see and feel the size of the rubber duck. Even when you grasp for the rubber duck with a tool (e.g., with tongs), the information from the proximal hand, from the effective part of the distal tool and from the eyes are integrated in a manner to act successfully (for limitations of this integration see Sutter et al., 2013).Over the recent decade a surge of interest in multisensoryintegration and action control has been witnessed, especially in connection with the idea of a statistically optimized integration of multiple sensory sources. The human information processing system is assumed to adjust moment-by-moment the relative contribution of each sense’s estimate to a multisensory task. The sense’s contribution depends on its variance, so that the total varia... (shrink)
As most work on flower foraging focuses on bees, studying Lepidoptera can offer fresh perspectives on how sensory capabilities shape the interaction between flowers and insects. Through a combination of innate preferences and learning, many Lepidoptera persistently visit particular flower species. Butterflies tend to rely on their highly developed sense of colour to locate rewarding flowers, while moths have evolved sophisticated olfactory systems towards the same end. However, these modalities can interact in complex ways; for instance, butterflies’ colour preference can (...) shift depending on olfactory context. The mechanisms by which such cross‐modal interaction occurs are poorly understood, but the mushroom bodies appear to play a central role. Because of the diversity seen within Lepidoptera in terms of their sensory capabilities and the nature of their relationships with flowers, they represent a fruitful avenue for comparative studies to shed light on the co‐evolution of flowers and flower‐visiting insects. (shrink)
This paper concerns epistemic developments in the field of sensory perception. I argue that Uexküll’s concept of the Umwelträume and certain principles of multisensoryintegration explain and describe in similar terms the manner in which different sensory modalities interact. Indeed, they both concern knowledge, describing in spatial terms how the mind makes itself up, makes up its objects, and how the objects, in turn, make up the mind. My intention is to set side by side these two trends (...) of thought in order to mutually explain them. I suggest that there is some sort of dialectic at work between the Umwelträume and multisensoryintegration: in the first case interaction between the senses results in the creation of high definition subjects, whereas in the latter the interaction of the senses results in high definition objects. (shrink)
It is common to distinguish between “holist” and “reductionist” views of brain function, where the former envisions the brain as functioning as an indivisible unit and the latter as a collection of distinct units that serve different functions. Opposing reductionism, a number of researchers have pointed out that cortical network architecture does not respect functional boundaries, and the neuroanatomist V. Braitenberg proposed to understand the cerebral cortex as a “great mixing machine” of neuronal activity from sensory inputs, motor commands, and (...) intrinsically generated processes. In this paper, we offer a contextualization of Braitenberg’s point, and we review evidence for the interactions of neuronal activity from multiple sensory inputs and intrinsic neuronal processes in the cerebral cortex. We focus on new insights from studies on audiovisual interactions and on the influence of respiration on brain functions, which do not seem to align well with “reductionist” views of areal functional boundaries. Instead, they indicate that functional boundaries are fuzzy and context dependent. In addition, we discuss the relevance of the influence of sensory, proprioceptive, and interoceptive signals on cortical activity for understanding brain-body interactions, highlight some of the consequences of these new insights for debates on embodied cognition, and offer some suggestions for future studies. (shrink)
Evidence concerning the relationship between attention and multisensoryintegration has long been thought to lead us into a paradox. The paradox has its roots in evidence that seems to show that attention exerts an influence on integration, and that integration also exerts an influence on attention. This creates an appearance of paradox only if it is understood to imply that particular instances of the integration process must occur both before and after particular instances of the (...) attention process. But this appearance of paradox can be removed if we can find a way to resist the idea that there must be fixed temporal relations between the instances of these processes. That idea can seem hard to resist if both are understood to be processes of the sort that are brought to a halt by their own completion. Reflection on a metaphysical distinction between different sorts of process shows this understanding can be rejected. The appearance of paradox is thereby removed. (shrink)
Multisensoryintegration can alter information processing, and previous research has shown that such processes are modulated by sensory switch costs and prior experience. Here we report an incidental finding demonstrating, for the first time, the interplay between these processes and experimental factors, specifically the presence of the experimenter in the testing room. Experiment 1 demonstrates that multisensory motor facilitation in response to audiovisual stimuli is higher in those trials in which the sensory modality switches than when it (...) repeats. Those participants who completed the study while alone exhibited increased RT variability. Experiment 2 replicated these findings using the letters “b” and “d” presented as unisensory stimuli or congruent and incongruent multisensory stimuli. Multisensory enhancements were inflated following a sensory switch; that is, congruent and incongruent multisensory stimuli resulted in significant gains following a sensory switch in the monitored condition. However, when the participants were left alone, multisensory enhancements were only observed for repeating incongruent multisensory stimuli. These incidental findings therefore suggest that the effects of letter congruence and sensory switching on multisensoryintegration are partly modulated by the presence of an experimenter. (shrink)
A recent report in Consciousness and Cognition provided evidence from a study of the rubber hand illusion that supports the multisensory principle of inverse effectiveness . I describe two methods of assessing the principle of inverse effectiveness , and discuss how the post-hoc method is affected by the statistical artefact of ‘regression towards the mean’. I identify several cases where this artefact may have affected particular conclusions about the PoIE, and relate these to the historical origins of ‘regression towards (...) the mean’. Although the conclusions of the recent report may not have been grossly affected, some of the inferential statistics were almost certainly biased by the methods used. I conclude that, unless such artefacts are fully dealt with in the future, and unless the statistical methods for assessing the PoIE evolve, strong evidence in support of the PoIE will remain lacking. (shrink)
The present study investigated how multisensoryintegration in peripersonal space is modulated by limb posture and limb congruency . This was done separately for the upper limbs and the lower limbs . The crossmodal congruency task was used to measure peripersonal space integration for the hands and the feet. It was found that the peripersonal space representation for the hands but not for the feet is dynamically updated based on both limb posture and limb congruency. Together these (...) findings show how dynamic cues from vision, proprioception, and touch are integrated in peripersonal limb space and highlight fundamental differences in the way in which peripersonal space is represented for the upper and lower extremity. (shrink)
Jaswal & Akhtar in their target article convincingly argue that subjects with autism do not have diminished social motivation. However, they still recognize that autistic people behave socially in an unusual way. Why? Here we suggest that these behaviours are the results of a multisensoryintegration deficit. Viewed from this perspective, the assumption that autistic people's unusual behaviours indicate diminished social motivation has to be replaced by the one that they have diminished social prediction skills.
Sensory processing dysfunction is characterized by a behaviorally observed difference in the response to sensory information from the environment. While the cerebellum is involved in normal sensory processing, it has not yet been examined in SPD. Diffusion tensor imaging scans of children with SPD and typically developing controls were compared for fractional anisotropy, mean diffusivity, radial diffusivity, and axial diffusivity across the following cerebellar tracts: the middle cerebellar peduncles, superior cerebellar peduncles, and cerebral peduncles. Compared to TDC, children with SPD (...) show reduced microstructural integrity of the SCP and MCP, characterized by reduced FA and increased MD and RD, which correlates with abnormal auditory behavior, multisensoryintegration, and attention, but not tactile behavior or direct measures of auditory discrimination. In contradistinction, decreased CP microstructural integrity in SPD correlates with abnormal tactile and auditory behavior and direct measures of auditory discrimination, but not multisensoryintegration or attention. Hence, altered cerebellar white matter organization is associated with complex sensory behavior and attention in SPD, which prompts further consideration of diagnostic measures and treatments to better serve affected individuals. (shrink)
Multisensory processing encompasses all of the various ways in which the presence of information in one sensory modality can adaptively influence the processing of information in a different modality. In Part I of this survey article, I begin by presenting a cartography of some of the more extensively investigated forms of multisensory processing, with a special focus on two distinct types of multisensoryintegration. I briefly discuss the conditions under which these different forms of multisensory (...) processing occur as well as their important perceptual consequences and interrelations. In Part II, I then turn to examining of some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical attempts to individuate the senses. (shrink)