This report highlights and explores five questions that arose from the multisensory integration workshop at the University of Toronto on May 9th and 10th, 2014: 1. What Is Multisensory Integration? 2. Do Multisensory Percepts Involve Emergent Features? 3. What Can Multisensory Processing Tell Us about Multisensory Awareness? 4. Is Language Processing a Special Kind of Multisensory Integration? 5. What Is the Purpose of Multisensory Integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Do multisensory percepts involve emergent features?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What can multisensory processing tell us about multisensory awareness?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: What is the purpose of multisensory integration?

This is an excerpt from a report on the workshop on multisensory integration at the University of Toronto, on May 9th and 10th, 2014, written by Kevin Connolly, Aaron Henry, Zoe Jenkin, and Andrew MacGregor, and available at: http://networksensoryresearch.utoronto.ca/Events_%26_Discussion.html This excerpt explores the question: Is language processing a special kind of multisensory integration?

The Bayesian model of multisensory cue integration proposed by Ernst and Banks provides an attractive model for understanding a way that our sensory systems may interact. Moreover, it has been suggested that the process of multisensory integration that it models underpins conscious experiences with multisensory representational contents merged across modalities. Should we therefore take empirical support for the Bayesian model as evidence of the multimodality of perception? Focusing on evidence of integration across vision and (...) touch, I argue that apparent support for the model does not warrant the rejection of the view that each of our conscious perceptual experiences is associated with one and only one sense modality. (shrink)

The integration of multisensory information is an essential mechanism in perception and action control. Research in multisensory integration is concerned with how the information from the different sensory modalities, such as the senses of vision, hearing, smell, taste, touch, and proprioception, are integrated to a coherent representation of objects (for an overview, see e.g., Calvert, Spence and Stein, 2004). The combination of information from the different senses is central for action control. For instance, when you grasp (...) for a rubber duck, you can see its size, feel its compliance and hear the sound it produces. Moreover, identical physical properties of an object can be provided by different senses. You can both see and feel the size of the rubber duck. Even when you grasp for the rubber duck with a tool (e.g., with tongs), the information from the proximal hand, from the effective part of the distal tool and from the eyes are integrated in a manner to act successfully (for limitations of this integration see Sutter et al., 2013).Over the recent decade a surge of interest in multisensory integration and action control has been witnessed, especially in connection with the idea of a statistically optimized integration of multiple sensory sources. The human information processing system is assumed to adjust moment-by-moment the relative contribution of each sense’s estimate to a multisensory task. The sense’s contribution depends on its variance, so that the total varia... (shrink)

In the social world, multiple sensory channels are used concurrently to facilitate communication. Among human and nonhuman pri- mates, faces and voices are the primary means of transmitting social signals (Adolphs, 2003; Ghazanfar and Santos, 2004). Primates recognize the correspondence between species-specific facial and vocal expressions (Massaro, 1998; Ghazanfar and Logothetis, 2003; Izumi and Kojima, 2004), and these visual and auditory channels can be integrated into unified percepts to enhance detection and discrimination. Where and how such communication signals are integrated (...) at the neural level are poorly understood. In particular, it is unclear what role “unimodal” sensory areas, such as the auditory cortex, may play. We recorded local field potential activity, the signal that best correlates with human imaging and event-related potential signals, in both the core and lateral belt regions of the auditory cortex in awake behaving rhesus monkeys while they viewed vocalizing conspecifics. We demonstrate unequivocally that the primate auditory cortex integrates facial and vocal signals through enhancement and suppression of field potentials in both the core and lateral belt regions. The majority of these multisensory responses were specific to face/voice integration, and the lateral belt region shows a greater frequency of multisensory integration than the core region. These multisensory processes in the auditory cortex likely occur via recip- rocal interactions with the superior temporal sulcus. (shrink)

As most work on flower foraging focuses on bees, studying Lepidoptera can offer fresh perspectives on how sensory capabilities shape the interaction between flowers and insects. Through a combination of innate preferences and learning, many Lepidoptera persistently visit particular flower species. Butterflies tend to rely on their highly developed sense of colour to locate rewarding flowers, while moths have evolved sophisticated olfactory systems towards the same end. However, these modalities can interact in complex ways; for instance, butterflies’ colour preference can (...) shift depending on olfactory context. The mechanisms by which such cross‐modal interaction occurs are poorly understood, but the mushroom bodies appear to play a central role. Because of the diversity seen within Lepidoptera in terms of their sensory capabilities and the nature of their relationships with flowers, they represent a fruitful avenue for comparative studies to shed light on the co‐evolution of flowers and flower‐visiting insects. (shrink)

This paper concerns epistemic developments in the field of sensory perception. I argue that Uexküll’s concept of the Umwelträume and certain principles of multisensory integration explain and describe in similar terms the manner in which different sensory modalities interact. Indeed, they both concern knowledge, describing in spatial terms how the mind makes itself up, makes up its objects, and how the objects, in turn, make up the mind. My intention is to set side by side these two trends (...) of thought in order to mutually explain them. I suggest that there is some sort of dialectic at work between the Umwelträume and multisensory integration: in the first case interaction between the senses results in the creation of high definition subjects, whereas in the latter the interaction of the senses results in high definition objects. (shrink)

Evidence concerning the relationship between attention and multisensory integration has long been thought to lead us into a paradox. The paradox has its roots in evidence that seems to show that attention exerts an influence on integration, and that integration also exerts an influence on attention. This creates an appearance of paradox only if it is understood to imply that particular instances of the integration process must occur both before and after particular instances of the (...) attention process. But this appearance of paradox can be removed if we can find a way to resist the idea that there must be fixed temporal relations between the instances of these processes. That idea can seem hard to resist if both are understood to be processes of the sort that are brought to a halt by their own completion. Reflection on a metaphysical distinction between different sorts of process shows this understanding can be rejected. The appearance of paradox is thereby removed. (shrink)

It is common to distinguish between “holist” and “reductionist” views of brain function, where the former envisions the brain as functioning as an indivisible unit and the latter as a collection of distinct units that serve different functions. Opposing reductionism, a number of researchers have pointed out that cortical network architecture does not respect functional boundaries, and the neuroanatomist V. Braitenberg proposed to understand the cerebral cortex as a “great mixing machine” of neuronal activity from sensory inputs, motor commands, and (...) intrinsically generated processes. In this paper, we offer a contextualization of Braitenberg’s point, and we review evidence for the interactions of neuronal activity from multiple sensory inputs and intrinsic neuronal processes in the cerebral cortex. We focus on new insights from studies on audiovisual interactions and on the influence of respiration on brain functions, which do not seem to align well with “reductionist” views of areal functional boundaries. Instead, they indicate that functional boundaries are fuzzy and context dependent. In addition, we discuss the relevance of the influence of sensory, proprioceptive, and interoceptive signals on cortical activity for understanding brain-body interactions, highlight some of the consequences of these new insights for debates on embodied cognition, and offer some suggestions for future studies. (shrink)

A recent report in Consciousness and Cognition provided evidence from a study of the rubber hand illusion that supports the multisensory principle of inverse effectiveness . I describe two methods of assessing the principle of inverse effectiveness , and discuss how the post-hoc method is affected by the statistical artefact of ‘regression towards the mean’. I identify several cases where this artefact may have affected particular conclusions about the PoIE, and relate these to the historical origins of ‘regression towards (...) the mean’. Although the conclusions of the recent report may not have been grossly affected, some of the inferential statistics were almost certainly biased by the methods used. I conclude that, unless such artefacts are fully dealt with in the future, and unless the statistical methods for assessing the PoIE evolve, strong evidence in support of the PoIE will remain lacking. (shrink)

Multisensory integration can alter information processing, and previous research has shown that such processes are modulated by sensory switch costs and prior experience. Here we report an incidental finding demonstrating, for the first time, the interplay between these processes and experimental factors, specifically the presence of the experimenter in the testing room. Experiment 1 demonstrates that multisensory motor facilitation in response to audiovisual stimuli is higher in those trials in which the sensory modality switches than when it (...) repeats. Those participants who completed the study while alone exhibited increased RT variability. Experiment 2 replicated these findings using the letters “b” and “d” presented as unisensory stimuli or congruent and incongruent multisensory stimuli. Multisensory enhancements were inflated following a sensory switch; that is, congruent and incongruent multisensory stimuli resulted in significant gains following a sensory switch in the monitored condition. However, when the participants were left alone, multisensory enhancements were only observed for repeating incongruent multisensory stimuli. These incidental findings therefore suggest that the effects of letter congruence and sensory switching on multisensory integration are partly modulated by the presence of an experimenter. (shrink)

The present study investigated how multisensory integration in peripersonal space is modulated by limb posture and limb congruency . This was done separately for the upper limbs and the lower limbs . The crossmodal congruency task was used to measure peripersonal space integration for the hands and the feet. It was found that the peripersonal space representation for the hands but not for the feet is dynamically updated based on both limb posture and limb congruency. Together these (...) findings show how dynamic cues from vision, proprioception, and touch are integrated in peripersonal limb space and highlight fundamental differences in the way in which peripersonal space is represented for the upper and lower extremity. (shrink)

Jaswal & Akhtar in their target article convincingly argue that subjects with autism do not have diminished social motivation. However, they still recognize that autistic people behave socially in an unusual way. Why? Here we suggest that these behaviours are the results of a multisensory integration deficit. Viewed from this perspective, the assumption that autistic people's unusual behaviours indicate diminished social motivation has to be replaced by the one that they have diminished social prediction skills.

Sensory processing dysfunction is characterized by a behaviorally observed difference in the response to sensory information from the environment. While the cerebellum is involved in normal sensory processing, it has not yet been examined in SPD. Diffusion tensor imaging scans of children with SPD and typically developing controls were compared for fractional anisotropy, mean diffusivity, radial diffusivity, and axial diffusivity across the following cerebellar tracts: the middle cerebellar peduncles, superior cerebellar peduncles, and cerebral peduncles. Compared to TDC, children with SPD (...) show reduced microstructural integrity of the SCP and MCP, characterized by reduced FA and increased MD and RD, which correlates with abnormal auditory behavior, multisensory integration, and attention, but not tactile behavior or direct measures of auditory discrimination. In contradistinction, decreased CP microstructural integrity in SPD correlates with abnormal tactile and auditory behavior and direct measures of auditory discrimination, but not multisensory integration or attention. Hence, altered cerebellar white matter organization is associated with complex sensory behavior and attention in SPD, which prompts further consideration of diagnostic measures and treatments to better serve affected individuals. (shrink)

Chemosensation and mechanosensation cover an enormous spectrum of processes by which animals use information from the environment to adapt their behavior. For pragmatic reasons, these sensory modalities are commonly investigated independently. Recent advances, however, have revealed numerous situations in which they function together to control animals’ actions. Highlighting examples from diverse vertebrates and invertebrates, we first discuss sensory receptors and neurons that have dual roles in the detection of chemical and mechanical stimuli. Next we present cases where peripheral chemosensory and (...) mechanosensory pathways are discrete but intimately packaged to permit coordinated reception of external cues. Finally, we consider how chemical and mechanical signals converge in central neural circuitry to enable multisensory integration. These insights demonstrate how investigation of the interplay between different sensory modalities is key to a more holistic and realistic understanding of sensory‐guided behaviors. (shrink)