This study investigated gesture–speech integration among adolescents who are deaf or hard of hearing and those with typical hearing. Thirty-eight adolescents performed a Stroop-like task in which they watched 120 short video clips of gestures and actions twice at random. Participants were asked to press one button if the visual content of the speaker’s movements was related to a written word and to press another button if it was unrelated to a written word while accuracy rates and response times were (...) recorded. We found stronger GSI effects among DHH participants than hearing participants. The semantic congruency effect was significantly larger in DHH participants than in hearing participants, and results of our experiments indicated a significantly larger gender congruency effect in DHH participants as compared to hearing participants. Results of this study shed light on GSI among DHH individuals and suggest future avenues for research examining the impact of gesture on language processing and communication in this population. (shrink)

Is the flavor of mint reducible to the minty smell, the taste, and the menthol-like coolness on the roof of one’s mouth, or does it include something over and above these—something not properly associated with any one of the contributing senses? More generally, are there features of perceptual experiences—so-called novel features—that are not associated with any of our senses taken singly? This question has received a lot of attention of late. Yet surprisingly little attention has been paid to the question (...) of what it means to say that a feature is associated with a sensory modality in the first place. Indeed, there is only one fully developed proposal in the literature, due to Casey O’Callaghan. I argue that this proposal is too permissive to inform the debate over novel features. I go on to argue that all attempts to formulate a better proposal along these lines fail. The corollary of my arguments is that the question of the existence of novel features is poorly formed. Furthermore, the problem generalizes, with the result that we should not rely on our pre-theoretical notions of the senses as the basis of theorizing about the features of perceptual experiences. (shrink)

The last decade has seen great progress in the study of the nature of crossmodal links in exogenous and endogenous spatial attention . Exogenous spatial cuing studies of human crossmodal attention and multisensory integration. In C. Spence, & J. Driver , Crossmodal space and crossmodal attention . Oxford, UK: Oxford University Press.], for a recent review). A growing body of research now highlights the existence of robust crossmodal links between auditory, visual, and tactile spatial attention. However, until recently, studies of (...) exogenous and endogenous attention have proceeded relatively independently. In daily life, however, these two forms of attentional orienting continuously compete for the control of our attentional resources, and ultimately, our awareness. It is therefore critical to try and understand how exogenous and endogenous attention interact in both the unimodal context of the laboratory and the multisensory contexts that are more representative of everyday life. To date, progress in understanding the interaction between these two forms of orienting has primarily come from unimodal studies of visual attention. We therefore start by summarizing what has been learned from this large body of empirical research, before going on to review more recent studies that have started to investigate the interaction between endogenous and exogenous orienting in a multisensory setting. We also discuss the evidence suggesting that exogenous spatial orienting is not truly automatic, at least when assessed in a crossmodal context. Several possible models describing the interaction between endogenous and exogenous orienting are outlined and then evaluated in terms of the extant data. (shrink)

The social brain hypothesis implies that humans and other primates evolved “modules“ for representing social knowledge. Alternatively, no such cognitive specializations are needed because social knowledge is already present in the world — we can simply monitor the dynamics of social interactions. Given the latter idea, what mechanism could account for coalition formation? We propose that statistical learning can provide a mechanism for fast and implicit learning of social signals. Using human participants, we compared learning of social signals with arbitrary (...) signals. We found that learning of social signals was no better than learning of arbitrary signals. While coupling faces and voices led to parallel learning, the same was true for arbitrary shapes and sounds. However, coupling versus uncoupling social signals with arbitrary signals revealed that faces and voices are treated with perceptual priority. Overall, our data suggest that statistical learning is a viable domain-general mechanism for learning social group structure. Keywords: social brain; embodied cognition; distributed cognition; situated cognition; multisensory; audiovisual speech; crossmodal; multimodal. (shrink)

In experimental psychology and neuroscience, technological advances and multisensory research have contributed to gradually dismiss a version of reductionism. Empirical results no longer support a brain model in which distinct “modules” perform discrete functions, but rather, a brain of partially overlapping networks. A similarly changed brain model is extending to psychopathology and clinical psychology, and partly accounts for the problems of reductionism.

One of the most promising trends both in the neuroscience of pain and in psychiatric treatments of chronic pain is the focus on mental imagery. My aim is to argue that if we take these findings seriously, we can draw very important and radical philosophical conclusions. I argue that what we pretheoretically take to be pain is partly constituted by sensory stimulation-driven pain processing and partly constituted by mental imagery. This general picture can explain some problematic cases of pain perception, (...) for example, phantom-limb pain, and it also has important consequences for some recent philosophical debates about the nature and content of pain. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) and (...) auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

In a seminal 1977 article, Rumelhart argued that perception required the simultaneous use of multiple sources of information, allowing perceivers to optimally interpret sensory information at many levels of representation in real time as information arrives. Building on Rumelhart's arguments, we present the Interactive Activation hypothesis—the idea that the mechanism used in perception and comprehension to achieve these feats exploits an interactive activation process implemented through the bidirectional propagation of activation among simple processing units. We then examine the interactive activation (...) model of letter and word perception and the TRACE model of speech perception, as early attempts to explore this hypothesis, and review the experimental evidence relevant to their assumptions and predictions. We consider how well these models address the computational challenge posed by the problem of perception, and we consider how consistent they are with evidence from behavioral experiments. We examine empirical and theoretical controversies surrounding the idea of interactive processing, including a controversy that swirls around the relationship between interactive computation and optimal Bayesian inference. Some of the implementation details of early versions of interactive activation models caused deviation from optimality and from aspects of human performance data. More recent versions of these models, however, overcome these deficiencies. Among these is a model called the multinomial interactive activation model, which explicitly links interactive activation and Bayesian computations. We also review evidence from neurophysiological and neuroimaging studies supporting the view that interactive processing is a characteristic of the perceptual processing machinery in the brain. In sum, we argue that a computational analysis, as well as behavioral and neuroscience evidence, all support the Interactive Activation hypothesis. The evidence suggests that contemporary versions of models based on the idea of interactive activation continue to provide a basis for efforts to achieve a fuller understanding of the process of perception. (shrink)

Sensory substitution devices make use of one substituting modality to get access to environmental information normally accessed through another modality . Based on behavioural and neuroimaging data, some authors have claimed that using a vision-substituting device results in visual perception. Reviewing these data, we contend that this claim is untenable. We argue that the kind of information processed by a SSD is metamodal, so that it can be accessed through any sensory modality and that the phenomenology associated with the use (...) of a SSD is best described in terms of spatial phenomenology, only. (shrink)

This paper provides a categorization of cross-modal experiences. There are myriad forms. Doing so allows us to think clearly about the nature of different cross-modal experiences and allows us to clearly formulate competing hypotheses about the kind of experiences involved in different cross-modal phenomena.

Traditional views separate cognitive processes from sensory–motor processes, seeing cognition as amodal, propositional, and compositional, and thus fundamentally different from the processes that underlie perceiving and acting. These were the ideas on which cognitive science was founded 30 years ago. However, advancing discoveries in neuroscience, cognitive neuroscience, and psychology suggests that cognition may be inseparable from processes of perceiving and acting. From this perspective, this study considers the future of cognitive science with respect to the study of cognitive development.

Dove and Machery both argue that recent findings about the nature of numerical representation present problems for Concept Empiricism. I shall argue that, whilst this evidence does challenge certain versions of CE, such as Prinz, it needn’t be seen as problematic to the general CE approach. Recent research can arguably be seen to support a CE account of number concepts. Neurological and behavioral evidence suggests that systems involved in the perception of numerical properties are also implicated in numerical cognition. Furthermore, (...) the discovery of associations between spatial and numerical representations also lends independent support to a CE approach. Although these findings support CE in general, certain versions of the theory may need revising in order to accommodate them. In particular, it may be necessary to either jettison Prinz's Modal Specificity Hypothesis or to revise one’s method for individuating modal representational formats. (shrink)

“Hardwired” is a term commonly used to describe the properties of certain behaviors or brain regions. As its usage has increased exponentially in the past 50 years, both in popular media and the scholarly literature, the concept appears to have gained a cloak of respectability in scientific discourse. However, its specific meaning is difficult to pinpoint. In this paper, I examine how “hardwired” has been used in the psychological and neuroscientific literature. The analysis reveals two major themes: one centers on (...) certain purported characteristics of behaviors or brain regions, such as fixedness; the other places these and other characteristics within an evolutionary framework. Overall, the analysis reveals a degree of overlap between “hardwiring” and the folk biology concept of innateness. Various complications arise from such overlap, casting doubts on the usefulness and legitimacy of “hardwired” in scientific discourse. (shrink)

Cognitive scientists have long known that the modalities interact during perceptual processing. Cross-modal illusions like the ventriloquism effect show that the course of processing in one modality can alter the course of processing in another. But how do the modalities interact in the specific domain of object perception? This paper distinguishes and analyzes two kinds of multisensory interaction in object perception. First, the modalities may bind features to a single object or event. Second, the modalities may cooperate when differentiating an (...) object or event from its surroundings. I critically evaluate evidence for various forms of multisensory binding. I then consider the case for multisensory differentiation. I argue that existing evidence for multisensory differentiation is inconclusive. I highlight ways that the issue might be empirically resolved. (shrink)

Haptic touch is an inherently active and exploratory form of perception, involving both coordinated movements and an array of distinct sensory receptors in the skin. For this reason, some have claimed that haptic touch is not a single sense, but rather a multisensory collection of distinct sensory systems. Though this claim is often made, it relies on what I regard as a confused conception of multisensory interaction. In its place, I develop a nuanced hierarchy of multisensory involvement. According to this (...) hierarchy, touch turns out to be a single modality in that its various receptors assign their features to the same tangible objects. When we grasp an object a range of distinct properties?shape, warmth, heft, texture, etc.?are all felt to belong to the object, just as different visual properties are associated with a visual object. Paradigm multisensory experiences, on the other hand, involve associations between distinct perceptual experiences, as when the way something looks affects the way something sounds. Thus despite its functional and physiological diversity, haptic touch can be regarded as a single sense. (shrink)

I argue for sensory pluralism. This is the view that there are many forms of sensory interaction and unity, and no single category that classifies them all. In other words, sensory interactions do not form a single natural kind. This view suggests that how we classify sensory systems (and the experiences they generate) partly depends on our explanatory purposes. I begin with a detailed discussion of the issue as it arises for our understanding of thermal perception, followed by a general (...) account and defense of sensory pluralism. (shrink)

The claims that “The brain processes information” or “Cognition is information processing” are accepted as truisms in cognitive science. However, it is unclear how to evaluate such claims absent a specification of “information” as it is used by neurocognitive theories. The aim of this article is, thus, to identify the key features of information that information-based neurocognitive theories posit. A systematic identification of these features can reveal the explanatory role that information plays in specific neurocognitive theories, and can, therefore, be (...) both theoretically and practically important. These features can be used, in turn, as desiderata against which candidate theories of information may be evaluated. After discussing some characteristics of explanation in cognitive science and their implications for “information”, three notions are briefly introduced: natural, sensory, and endogenous information. Subsequently, six desiderata are identified and defended based on cognitive scientific practices. The global workspace theory of consciousness is then used as a specific case study that arguably posits either five or six corresponding features of information. (shrink)

We discuss two problems for a general scientific understanding of language, sequences and synergies: how language is an intricately sequenced behavior and how language is manifested as a multidimensionally structured behavior. Though both are central in our understanding, we observe that the former tends to be studied more than the latter. We consider very general conditions that hold in human brain evolution and its computational implications, and identify multimodal and multiscale organization as two key characteristics of emerging cognitive function in (...) our species. This suggests that human brains, and cognitive function specifically, became more adept at integrating diverse information sources and operating at multiple levels for linguistic performance. We argue that framing language evolution, learning, and use in terms of synergies suggests new research questions, and it may be a fruitful direction for new developments in theory and modeling of language as an integrated system. (shrink)

A key question in studying consciousness is how neural operations in the brain can identify streams of sensory input as belonging to distinct modalities, which contributes to the representation of qualitatively different experiences. The basis for identification of modalities is proposed to be constituted by self-organized comparative operations across a network of unimodal and multimodal sensory areas. However, such network interactions alone cannot answer the question how sensory feature detectors collectively account for an integrated, yet phenomenally differentiated experiential content. This (...) problem turns out to be different from, although related to, the binding problem. It is proposed that the neural correlate of an enriched, multimodal experience is constituted by the attractor state of a dynamic associative network. Within this network, unimodal and multimodal sensory maps continuously interact to influence each other’s attractor state, so that a feature change in one modality results in a fast re-coding of feature information in another modality. In this scheme, feature detection is coded by firing-rate, whereas firing phase codes relational aspects. (shrink)

The first part of this survey article presented a cartography of some of the more extensively studied forms of multisensory processing. In this second part, I turn to examining some of the different possible ways in which the structure of conscious perceptual experience might also be characterized as multisensory. In addition, I discuss the significance of research on multisensory processing and multisensory consciousness for philosophical debates concerning the modularity of perception, cognitive penetration, and the individuation of the senses.

People’s awareness of tactile stimuli has been investigated in far less detail than their awareness of stimuli in other sensory modalities. In an attempt to fill this gap, we provide an overview of studies that are pertinent to the topic of tactile consciousness. We discuss the results of research that has investigated phenomena such as “change blindness”, phantom limb sensations, and numerosity judgments in tactile perception, together with the results obtained from the study of patients affected by deficits that can (...) adversely affect tactile perception such as neglect, extinction, and numbsense. The similarities as well as some of the important differences that have emerged when visual and tactile conscious information processing have been compared using similar experimental procedures are highlighted. We suggest that conscious information processing in the tactile modality cannot be separated completely from the more general processing of spatial information in the brain. Finally, the importance of considering tactile consciousness within the larger framework of multisensory information processing is also discussed. (shrink)

Some philosophers have argued that we do not hear sounds as located in the environment. Others have objected that this straightforwardly contradicts the phenomenology of auditory experience. And from this they draw metaphysical conclusions about the nature of sounds—that they are events or properties of vibrating surfaces rather than waves or sensations. I argue that there is a minimal, but recognizable, notion of audition to which this phenomenal objection does not apply. While this notion doesn’t correspond to our ordinary notion (...) of auditory experience, it does—in conjunction with our lack of an uncontroversial individuation of the senses and recent interest in distinctively multisensory features of perceptual experiences—raise the possibility of more expansive notions of audition, including some that do plausibly count as corresponding to our everyday notion of audition, that lack the spatial phenomenology cited in the objection. Until this possibility is ruled out, the phenomenal objection and metaphysical conclusions drawn from it remain inconclusive. (shrink)

Western science claims to provide unique, objective information about the world. This is supported by the observation that peoples across cultures will agree upon a common description of the physical world. Further, the use of scientific instruments and mathematics is claimed to enable the objectification of science. In this work, carried out by reviewing the scientific literature, the above claims are disputed systematically by evaluating the definition of physical reality and the scientific method, showing that empiricism relies ultimately upon the (...) human senses for the evaluation of scientific theories and that measuring instruments cannot replace the human sensory system. Nativist and constructivist theories of human sensory development are reviewed, and it is shown that nativist claims of core conceptual knowledge cannot be supported by the findings in the literature, which shows that perception does not simply arise from a process of maturation. Instead, sensory function requires a long process of learning through interactions with the environment. To more rigorously define physical reality and systematically evaluate the stability of perception, and thus the basis of empiricism, the development of the method of dimension analysis is reviewed. It is shown that this methodology, relied upon for the mathematical analysis of physical quantities, is itself based upon empiricism, and that all of physical reality can be described in terms of the three fundamental dimensions of mass, length and time. Hereafter the sensory modalities that inform us about these three dimensions are systematically evaluated. The following careful analysis of neuronal plasticity in these modalities shows that all the relevant senses acquire from the environment the capacity to apprehend physical reality. It is concluded that physical reality is acquired rather than given innately, and leads to the position that science cannot provide unique results. Rather, those it can provide are sufficient for a particular environmental setting. (shrink)

Long-standing neglect of the chemical senses in the philosophy of perception is due, mostly, to their being regarded as ‘lower’ senses. Smell, taste, and chemically irritated touch are thought to produce mere bodily sensations. However, empirically informed theories of perception can show how these senses lead to perception of objective properties, and why they cannot be treated as special cases of perception modelled on vision. The senses of taste, touch, and smell also combine to create unified perceptions of flavour. The (...) nature of these multimodal experiences and the character of our awareness of them puts pressure on the traditional idea that each episode of perception goes one or other of the five senses. Thus, the chemical senses, far from being peripheral to the concerns of the philosophy of perception, may hold important clues to the multisensory nature of perception in general. (shrink)