Many philosophers use findings about sensory substitution devices in the grand debate about how we should individuate the senses. The big question is this: Is “vision” assisted by (tactile) sensory substitution really vision? Or is it tactile perception? Or some sui generis novel form of perception? My claim is that sensory substitution assisted “vision” is neither vision nor tactile perception, because it is not perception at all. It is mental imagery: visual mental imagery triggered (...) by tactile sensory stimulation. But it is a special form of mental imagery that is triggered by corresponding sensory stimulation in a different sense modality, which I call “multimodal mental imagery.”. (shrink)

Sensory substitution devices make use of one substituting modality to get access to environmental information normally accessed through another modality . Based on behavioural and neuroimaging data, some authors have claimed that using a vision-substituting device results in visual perception. Reviewing these data, we contend that this claim is untenable. We argue that the kind of information processed by a SSD is metamodal, so that it can be accessed through any sensory modality and that the phenomenology associated (...) with the use of a SSD is best described in terms of spatial phenomenology, only. (shrink)

Is the brain the biological substrate of consciousness? Most naturalistic philosophers of mind have supposed that the answer must obviously be «yes » to this question. However, a growing number of philosophers working in 4e (embodied, embedded, extended, enactive) cognitive science have begun to challenge this assumption, arguing instead that consciousness supervenes on the whole embodied animal in dynamic interaction with the environment. We call views that share this claim dynamic sensorimotor theories of consciousness (DSM). Clark (2009) a founder and (...) leading proponent of the hypothesis of the extended mind, demurs, arguing that as matter of fact the biology of consciousness doesn’t allow for a brain, body and world boundary crossing architecture. We begin by looking at one of the arguments for DSM, the variable neural correlates argument. We then outline two criticisms that Clark has made of this argument and endorse his criticisms. However we finish up by using the case of sensory substitution to argue that something of this argument for DSM nevertheless survives. We suggest that Clark ought to concede sensory substitution as a case in which the conscious mind extends. (shrink)

When a user integrates a sensory substitution device into her life, the process involves perceptual learning, that is, ‘relatively long-lasting changes to an organism’s perceptual system that improve its ability to respond to its environment’ (Goldstone 1998: 585). In this paper, I explore ways in which the extensive literature on perceptual learning can be applied to help improve sensory substitution devices. I then use these findings to answer a philosophical question. Much of the philosophical debate surrounding (...) sensory substitution devices concerns what happens after perceptual learning occurs. In particular, should the resultant perceptual experience be classified in the substituted modality (as vision), in the substituting modality (as auditory or tactile), or in a new sense modality? I propose a novel empirical test to help resolve this philosophical debate. (shrink)

It is hoped that modern sensory substitution and augmentation devices will be able to replace or expand our senses. But to what extent has this been achieved to date? To what extent are the experiences created by sensory substitution devices like the sensory experiences that we are trying to replace? To what extent can we augment people’s senses providing them with new information and new experiences? The first aim of this introduction is to delve deeply (...) into this question to discover the usefulness of these devices, to outline the different sorts of experience that might be created, and what the evidence tells us about these experiences. While there are some reasons to be hopeful about the powers of sensory substitution devices, there are also reasons to wonder whether they will ever really have the practical applications that we hope they might have. The second aim is to look to see whether the study of modern sensory substitution and augmentation devices can shed light on the nature of our senses and perception in general. Much of the philosophical, psychological, and neuroscientific work that takes place concerning sensory substitution and augmentation is keenly aware of the possibility that it might help our understanding and it seeks to comprehend many different aspects of perception. (shrink)

One of the central limitations of sensory substitution devices (SSDs) is their inability to reproduce the non-sensory feelings that are normally associated with visual experiences, especially hedonic and aesthetic responses. This limitation is sometimes reported to cause SSD users frustration. To make matters worse, it is unclear that improvements in acuity, bandwidth, or training will resolve the issue. Yet, if SSDs are to actually reproduce visual experience in its fullness, it seems that the reproduction of non-sensory (...) feelings will be of some importance. We offer a novel solution. Researchers can produce hedonic and aesthetic responses by eliciting these feelings artificially, pairing distal objects that should be pleasurable to pleasurable stimulus outputs from the SSD. We outline two strategies for accomplishing this: first, by means of a prefixed, hardwired, association of pleasant distal objects to pleasant stimulus outputs from the SSD; second, by means of a flexible, feedback-based association which creates associations based on a subject-directed matching of distal objects to patterns of stimuli from the SSD which the subject takes to have the corresponding hedonic properties. We evaluate some problems with both strategies, and we argue that the feedback-based strategy is more promising. Researchers can use this strategy to help the blind, allowing them to take pleasure in the objects they perceive using SSDs. (shrink)

This report highlights and explores five questions that arose from the workshop on sensory substitution and augmentation at the British Academy, March 26th through 28th, 2013.

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: Does sensory substitution generate perceptual or cognitive states?

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: Can normal non-sensory feelings be generated through sensory substitution?

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: What can sensory substitution tell us about perceptual learning?

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: How does sensory substitution interact with the brain’s architecture?

This is an excerpt from a report on the Sensory Substitution and Augmentation Conference at the British Academy in March of 2013. This portion of the report explores the question: What are the limitations of sensory substitution.

Sensory substitution and augmentation devices are used to replace or enhance one sense by using another. Fiona Macpherson brings together neuroscientists, psychologists and philosophers to focus on the nature of the perceptual experiences, the sensory interactions, and the changes that occur in the mind and brain while using these technologies.

Sensory substitution devices (SSDs) are most familiar from their use with subjects who are deficient in a target modality (e.g. congenitally blind subjects), but there is no doubt that the use and potential value of SSDs extend to persons without such deficits. Recent work by Amedi and his team (in particular Levy-Tzedek et al. 2012) has begun to explore this. Their idea is that SSDs may facilitate behavioural transference (BT) across sense modalities. In this case, a motor skill (...) learned through visual perception might be subsequently employed in response to auditory perception, using an SSD as a mediator. They infer from the existence of such BT that the learned skill is amodally represented. After a brief overview I identify ways to more fully test for BT within this experimental paradigm and argue that their conclusion about amodal representation is premature. Additionally, I argue that their preferred SSD (Eyemusic) is of limited value for the project. While my remarks are critical, my intention is to be constructive, particularly in light of the fact that Levy-Tzedek et al. (2012), is, I believe, the first output from Amedi's lab concerning this line of research. (shrink)

A refined characterisation of sensory substitution has, as a consequence, that the substituting sense plus sensory substitution device is not always appropriately classified as the substituted sense. As a result, I argue, acclimatisation to a sensory substitution device is plausibly thought of as providing presentations of properties. Externalist accounts of experience together with objectivist characterisations of such properties have the upshot that properties putatively proprietary to a sense modality can be presented in another modality (...) in cases of substitution. I consider three objections to this argument. I close by explaining how reflection on the phenomena of sensory substitution and, in particular, acclimatisation is important for the development of any kind of representationalist or relationist theory of phenomenal properties or, at the very least, suggests we need to refine the idea of certain properties - rather than particular ways in which their presentation is bundled together - being proprietary to the particular senses. (shrink)

Block (Trends Cogn Sci 7:285–286, 2003) and Prinz (PSYCHE 12:1–19, 2006) have defended the idea that SSD perception remains in the substituting modality (auditory or tactile). Hurley and Noë (Biol Philos 18:131–168, 2003) instead argued that after substantial training with the device, the perceptual experience that the SSD user enjoys undergoes a change, switching from tactile/auditory to visual. This debate has unfolded in something like a stalemate where, I will argue, it has become difficult to determine whether the perception acquired (...) through the coupling with an SSD remains in the substituting or the substituted modality. Within this puzzling deadlock two new approaches have been recently suggested. Ward and Meijer (Conscious Cogn 19:492–500, 2010) describe SSD perception as visual-like but characterize it as a kind of artificially induced synaesthesia. Auvray et al. (Perception 36:416–430, 2007) and Auvray and Myin (Cogn Sci 33:1036–1058, 2009) suggest that SSDs let their users experience a new kind of perception. Deroy and Auvray (forthcoming) refine this position, and argue that this new kind of perception depends on pre-existing senses without entirely aligning with any of them. So, they have talked about perceptual experience in SSDs as going "beyond vision". In a similar vein, MacPherson (Oxford University Press, New York, 2011a) claims that “if the subjects (SSD users) have experiences with both vision-like and touch-like representational characteristics then perhaps they have a sense that ordinary humans do not” (MacPherson in Oxford University Press, New York, 2011a, p. 139). (shrink)

In current philosophy of mind, there is lively debate over whether emotions, moods, and other affects can extend to comprise elements beyond one’s organismic boundaries. At the same time, there has been growing interest in the nature and significance of so-called existential feelings, which, as the term suggests, are feelings of one’s overall being in the world. In this article, I bring these two strands of investigation together to ask: Can the material underpinnings of existential feelings extend beyond one’s skull (...) and skin? To begin, I introduce and adopt a componential-systemic view of extended affectivity. In doing so, I specify the vehicle externalist criteria for extension employed in my analysis. I then define what existential feelings are and pinpoint several key issues in their potential extension. More specifically, I identify sensorimotor ‘know-how’ as a possibly extending component of existential feeling and posit it as the fulcrum of my argument. Finally, I move on to consider the extension of existential feeling via sensory substitution, especially through so-called Tactile Visual Sensory Substitution (TVSS) devices. Informed by both philosophical and empirical studies, I argue that, under certain conditions, (a) the agent’s implicit sensorimotor processing and a TVSS device can become coupled into a new systemic whole, which in turn (b) reconfigures the material underpinnings of the agent’s pre-intentional world-experience. This, I conclude, counts as a cogent case for extended existential feeling. (shrink)

Visual information can be provided to blind users through sensory substitution devices that convert images into sound. Through extensive use to develop expertise, some blind users have reported visual experiences when using such a device. These blind expert users have also reported visual phenomenology to other sounds even when not using the device. The blind users acquired synthetic synaesthesia, with visual experience evoked by sounds only after gaining such expertise. Sensorimotor learning may facilitate and perhaps even be required (...) to develop expertise in the use of multimodal information. Furthermore, other areas where expertise is acquired in dividing attention amongst cross-modal information or integrating such information might also give rise to synthetic synaesthesia. (shrink)

Sensory substitution devices provide through an unusual sensory modality (the substituting modality, e.g., audition) access to features of the world that are normally accessed through another sensory modality (the substituted modality, e.g., vision). In this article, we address the question of which sensory modality the acquired perception belongs to. We have recourse to the four traditional criteria that have been used to define sensory modalities: sensory organ, stimuli, properties, and qualitative experience (Grice, 1962), (...) to which we have added the criteria of behavioral equivalence (Morgan, 1977), dedication (Keeley, 2002), and sensorimotor equivalence (O’Regan & Noe¨, 2001). We discuss which of them are fulfilled by perception through sensory substitution devices and whether this favors the view that perception belongs to the substituting or to the substituted modality. Though the application of a number of criteria might be taken to point to the conclusion that perception with a sensory substitution device belongs to the substituted modality, we argue that the evidence leads to an alternative view on sensory substitution. According to this view, the experience after sensory substitution is a transformation, extension, or augmentation of our perceptual capacities, rather than being something equivalent or reducible to an already existing sensory modality. We develop this view by comparing sensory substitution devices to other ‘‘mind-enhancing tools’’ such as pen and paper, sketchpads, or calculators. An analysis of sensory substitution in terms of mind-enhancing tools unveils it as a thoroughly transforming perceptual experience and as giving rise to a novel form of perceptual interaction with the environment. (shrink)

Sensory substitution devices (SSDs) aim to compensate for the loss of a sensory modality, typically vision, by converting information from the lost modality into stimuli in a remaining modality. “The vOICe” is a visual-to-auditory SSD which encodes images taken by a camera worn by the user into “soundscapes” such that experienced users can extract information about their surroundings. Here we investigated how much detail was resolvable during the early induction stages by testing the acuity of blindfolded sighted, (...) naïve vOICe users. Initial performance was well above chance. Participants who took the test twice as a form of minimal training showed a marked improvement on the second test. Acuity was slightly but not significantly impaired when participants wore a camera and judged letter orientations “live”. A positive correlation was found between participants' musical training and their acuity. The relationship between auditory expertise via musical training and the lack of a relationship with visual imagery, suggests that early use of a SSD draws primarily on the mechanisms of the sensory modality being used rather than the one being substituted. If vision is lost, audition represents the sensory channel of highest bandwidth of those remaining. The level of acuity found here, and the fact it was achieved with very little experience in sensory substitution by naïve users is promising. (shrink)

In this report, the phenomenology of two blind users of a sensory substitution device – “The vOICe” – that converts visual images to auditory signals is described. The users both report detailed visual phenomenology that developed within months of immersive use and has continued to evolve over a period of years. This visual phenomenology, although triggered through use of The vOICe, is likely to depend not only on online visualization of the auditory signal but also on the users’ (...) previous (albeit distant) experience of veridical vision (e.g. knowledge of shapes and visual perspective). Once established, the sensory substitution mapping between the auditory and visual domains is not confined to when the device is worn and, thus, may constitute an example of acquired synaesthesia. (shrink)

This paper analyzes the process of perceptual recalibration (PR) in light of two cases of technologically-mediated cognition: sensory substitution and perceptual modification. We hold that PR is a very useful concept - perhaps necessary - for explaining the adaptive capacity that natural perceptive systems display as they respond to functional demands from the environment. We also survey critically related issues, such as the role of learning, training, and nervous system plasticity in the recalibrating process. Attention is given to (...) the interaction between technology and cognition, and the case of epistemic prostheses is presented as an illustration. Finally, we address the following theoretical issues: (1) the dynamic character of spatial perception; (2) the role of functional demands in perception; (3) the nature and interaction of sensory modalities. We aim to show that these issues may be addressed empirically and conceptually - hence, the usefulness of sensory-substitution and perceptual-modification studies in the analysis of perception, technologically-mediated cognition, and cognition in general. (shrink)

This paper analyzes the process of perceptual recalibration in light of two cases of technologically-mediated cognition: sensory substitution and perceptual modification. We hold that PR is a very useful concept — perhaps necessary — for explaining the adaptive capacity that natural perceptive systems display as they respond to functional demands from the environment. We also survey critically related issues, such as the role of learning, training, and nervous system plasticity in the recalibrating process. Attention is given to the (...) interaction between technology and cognition, and the case of epistemic prostheses is presented as an illustration. Finally, we address the following theoretical issues: the dynamic character of spatial perception; the role of functional demands in perception; the nature and interaction of sensory modalities. We aim to show that these issues may be addressed empirically and conceptually — hence, the usefulness of sensory-substitution and perceptual-modification studies in the analysis of perception, technologically-mediated cognition, and cognition in general. (shrink)

According to proponents of the sensorimotor contingency theory of perception (Hurley & Noë 2003, Noë 2004, O’Regan 2011), active control of camera movement is necessary for the emergence of distal attribution in tactile-visual sensory substitution (TVSS) because it enables the subject to acquire knowledge of the way stimulation in the substituting modality varies as a function of self-initiated, bodily action. This chapter, by contrast, approaches distal attribution as a solution to a causal inference problem faced by the subject’s (...) perceptual systems. Given all of the available endogenous and exogenous evidence available to those systems, what is the most probable source of stimulation in the substituting modality? From this perspective, active control over the camera’s movements matters for rather different reasons. Most importantly, it generates proprioceptive and efference-copy based information about the camera’s body-relative position necessary to make use of the spatial cues present in the stimulation that the subject receives for purposes of egocentric object localization. (shrink)

What if a blind person could 'see' with her ears? Thanks to Sensory Substitution Devices (SSDs), blind people now have access to out-of-reach objects, a privilege reserved so far for the sighted. In this paper, we show that the philosophical debates have fundamentally been mislead to think that SSDs should be fitted among the existing senses or that they constitute a new sense. Contrary to the existing assumption that they get integrated at the sensory level, we present (...) a new thesis according to which they are not sensory, and get vertically integrated on the top of existing sensory abilities, from which they should be theoretically distinguished. (shrink)

In this paper we wish to bring together two seemingly independent areas of research: synaesthesia and sensory substitution. Synaesthesia refers to a rare condition where a sensory stimulus elicits not only the sensation that stimulus evokes in its own modality, but an additional one; a synaesthete may thus hear the word “Monday”, and, in addition to hearing it, have a concurrent visual experience of a red color. Sensory substitution, in contrast, attempts to substitute a (...) class='Hi'>sensory modality that a person has lost by transforming the information it provided so that it can be accessed through another, intact sensory modality. To make visual information accessible to a blind person, for example, data taken by a camera would be transformed into tactile or auditory information. What do synaesthesia and sensory substitution have in common? Research in both of these areas contributes to our understanding both of cross-modal cooperation and of sensory sensations or qualia, asking under what circumstances these can arise in a modality that is not stimulated. Synaesthesia reveals that this “sensory cross-activation” is possible, and sensory substitution research hopes to induce it. In this article, we will review briefly the literature on synaesthesia, and discuss the issue of qualia for this domain of research. We will then address the evidence for synaesthesia and visual qualia in the blind, and the research on sensory substitution, to finally ask whether sensory substitution may induce a ‘synthetic’ form of synaesthesia by taking advantage of the nervous system’s capacity for generating visual images in the absence of retinal input. (shrink)

Sensory substitution presents the philosopher of cognitive science with a particularly interesting case. Using prosthetics to map visual stimuli onto other modalities, such as touch or audition, otherwise blind individuals may develop perceptual capacities and behaviours commonly associated with sight. Experienced users can distinguish ‘visually’ presented objects and will even jerk back from a looming surface. Whether perception with sensory substitution devices should be classed as a type of vision, some other modality, or a new sense (...) remains a matter of debate, however. In the following, I review arguments commonly used to rebut the visual interpretation and, drawing on recent experimental studies and phenomenological self-reports, construct a novel case for treating sensory substitution as a visual process. (shrink)

According to the hypothesis of extended phenomenal consciousness, although the material vehicles that realize phenomenal consciousness include neural elements, they are not restricted to such elements. There will be cases in which those material vehicles additionally include not only non-neural bodily elements, but also elements located beyond the skull and skin. In this paper, I examine two arguments for ExPC, one due to Noë and the other due to Kiverstein and Farina. Both of these arguments conclude that ExPC is true (...) on the basis of an analysis of sensory substitution, the phenomenon in which technological augmentation enables one sensory modality, for instance touch, to support the kind of environmental access and interaction ordinarily supported by a different sensory modality, for instance vision. I develop considerations which reveal that, as they stand, both of these arguments fall short. If the phenomenon of sensory substitution provides evidence for the truth of ExPC, it is not because of the specific arguments offered by these authors. (shrink)

Researchers in post-war industrial laboratories such as Bell Labs and the Smith-Kettlewell Institute pioneered solutions to compensate for sensory loss through so-called sensory substitution systems, premised on an assumption of cortical and sensory plasticity. The article tracks early discussions of plasticity in psychology literature from William James, acknowledged by Wiener, but explicitly developed by Bach-y-Rita and his collaborators. After discussing the conceptual foundations of the principles of sensory substitution, two examples are discussed. First, ‘Project (...) Felix’ was an experiment in vibrotactile communication by means of ‘hearing gloves’ for the deaf at Norbert Wiener’s laboratory at Massachusetts Institute of Technology, demonstrated to Helen Keller in 1950. Second, the tactile-visual sensory substitution system for the blind pioneered by Paul Bach-y-Rita from 1968 onwards. Cumulatively, this article underlines the crucial yet occluded history of research on sensory impairments in the discovery of underlying neurophysiological processes of plasticity and the emergent discourse of neuroplastic subjectivity. (shrink)

From the outside, our senses are spatially integrated in our body in manifestly different ways. This paper starts from the suggestion that the philosophical formulation of the problem of spatial perception, as it flows from the modern opposition of mind and world, is partly responsible for the fact that philosophers have often explicitly disregarded the spatial nature of the senses themselves. An indirect consequence is that much philosophical work focuses on how the senses can – or cannot – perceive the (...) same spatial features of objects in the same way, while disregarding the massive differences in performance among the senses. This paper explores the ways in which the spatial integration of our senses determines what each of our senses excels in, and hence what they contribute to our spatial relation to a physical environment. By juxtaposing descriptive analyses of a variety of selected cases, the paper gives priority to how various aspects of the world appear to us in order to obtain insight into the meaning of spatiality for different senses. (shrink)

From the outside, our senses are spatially integrated in our body in manifestly different ways. This paper starts from the suggestion that the philosophical formulation of the problem of spatial perception, as it flows from the modern opposition of mind and world, is partly responsible for the fact that philosophers have often explicitly disregarded the spatial nature of the senses themselves. An indirect consequence is that much philosophical work focuses on how the senses can – or cannot – perceive the (...) same spatial features of objects in the same way, while disregarding the massive differences in performance among the senses. This paper explores the ways in which the spatial integration of our senses determines what each of our senses excels in, and hence what they contribute to our spatial relation to a physical environment. By juxtaposing descriptive analyses of a variety of selected cases, the paper gives priority to how various aspects of the world appear to us in order to obtain insight into the meaning of spatiality for different senses. (shrink)

Sensory substitution devices are a type of sensory prosthesis that (typically) convert visual stimuli transduced by a camera into tactile or auditory stimulation. They are designed to be used by people with impaired vision so that they can recover some of the functions normally subserved by vision. In this chapter we will consider what philosophers might learn about the nature of the senses from the neuroscience of sensory substitution. We will show how sensory (...) class='Hi'>substitution devices work by exploiting the cross-modal plasticity of sensory cortex: the ability of sensory cortex to pick up some types of information about the external environment irrespective of the nature of the sensory inputs it is processing. We explore the implications of cross-modal plasticity for theories of the senses that attempt to make distinctions between the senses on the basis of neurobiology. (shrink)

Synaesthesia is a strange sensory blending: synaesthetes report experiences of colours or tastes associated with particular sounds or words. This volume presents new essays by scientists and philosophers exploring what such cases can tell us about the nature of perception and its boundaries with illusion and imagination.

The strategies of action employed by a human subject in order to perceive simple 2-D forms on the basis of tactile sensory feedback have been modelled by an explicit computer algorithm. The modelling process has been constrained and informed by the capacity of human subjects both to consciously describe their own strategies, and to apply explicit strategies; thus, the strategies effectively employed by the human subject have been influenced by the modelling process itself. On this basis, good qualitative and (...) semi-quantitative agreement has been achieved between the trajectories produced by a human subject, and the traces produced by a computer algorithm. The advantage of this reciprocal modelling option, besides facilitating agreement between the algorithm and the empirically observed trajectories, is that the theoretical model provides an explanation, and not just a description, of the active perception of the human subject. (shrink)

Vision substitution by electro-stimulation has been studied since the 60s beginning with P. Bach-y-Rita. Camera pictures or movies encoded in gray levels are displayed using an electro-stimulation display device on the surface of a body part, such as the skin or the tongue. Medical-technical devices have been developed on this principle to compensate for sensory-motor disabilities such as blindness or loss of balance, or to guide specific actions, such as surgery. However, the electrical signals of stationary or moving (...) slowly moving objects, displayed on a Tongue display unit , are quickly lost due to saturation of receptors undergoing electrostimulation. We propose to add random saccades or sparkle to the displayed visual scene to increase the quality of pattern recognition by the subjects. In the present experimental trimodal study , we show that the presence of a moderate sparkle level enhances the perception of the direction of lines drawn on a TDU and reduces the response time. (shrink)

I argue that John McDowell’s attempt to refute Wilfrid Sellars’s two-component analysis of perceptual experience and substitute for it a conception according to which perceptual experience is the “conceptual shaping of sensory consciousness” fails. McDowell does not recognize the subtle dialectic in Sellars’s thought between transcendental and empirical considerations in favor of a substantive conception of sense impressions, and McDowell’s own proposal seems to empty the notion of sensory consciousness of any real significance.