The purpose of sensory substitution devices is to restore functionality of a deficient sensory modality, such as vision for the blind, by using another sensory modality, such as touch or audition. These devices target a broad variety of deficits, such as loss of vision, proprioception, balance, or audition. Numerous studies have been conducted to investigate sensory substitution at the behavioral, phenomenological, and neural levels. People equipped with visual-to-tactile or visual-to-auditory conversion devices are able to navigate their surroundings, recognize objects, and categorize them, and their qualitative experience has specific multisensory characteristics. Furthermore, blind people trained with these devices show increased activation in specific areas of the visual cortex. The field of sensory substitution has implications for cognitive psychology, neurosciences, and philosophy, raising questions about the mechanisms of neuroplasticity and the relevant criteria to define the different senses.

History

Sensory substitution devices were primarily developed to restore functions in sensorially impaired people. In the late 1960s, Bach-y-Rita et al. (1969) developed the tactile–visual sensory substitution device, which converted visual pictures captured by a camera into spatiotemporal patterns of tactile stimuli that were applied to a person’s back skin. Subsequent devices were designed to deliver the stimuli to the surface of the tongue or the fingertip. To compensate for the loss of vision, visual-to-auditory technologies translate visual pictures into auditory soundscape patterns. For example, the first of these, the vOICe, converted visual brightness into auditory amplitude, the x-axis of a visual picture into the time domain, and the y-axis into the frequency domain (Meijer, 1992). Later on, devices were designed targeting other deficits, such as haptic devices to compensate for loss in proprioception, balance, or audition (see Eagleman & Perrotta, 2023, for review).

Core concepts

Learning and transfer of learning

Numerous behavioral studies have been conducted to investigate learning with sensory substitution devices. For instance, users trained with visual-to-auditory devices were shown to perform recognition, discrimination, localization, reaching, navigation, obstacle avoidance, and even face recognition tasks with further training (see Kirsch et al., 2020, for a review). In addition, recognition performance has been demonstrated to transfer to new objects that were not previously introduced during training (Kim & Zatorre, 2008), to novel orientations, and to variations in the device such as its bodily location (Arnold & Auvray, 2014). Instead of only memorizing stimulus pairs, transferable learning skills show a generalizable perceptual learning [see Perceptual Learning]. The extent to which learning is specific or general and how this varies depending on the kind of stimuli, the context, and the learning protocols is still the topic of ongoing studies.

Nature of the experience

A first vivid theoretical debate in the field targeted the question of the sensory modality in which the percept resulting from the sensory substitution device is represented. According to early proponents of sensory substitution, these devices would allow blind individuals to “see with their skin” (White et al., 1970). Likewise, according to the cortical deference view (e.g., O'Regan, 2011), perceptual experience is transferred from the intact modality to the defective modality. Instead, others advocated for cortical dominance, which holds that perception remains within the substituting modality (e.g., Block, 2003). Since then, an alternative view has emerged and started to reach a consensus in the scientific community: Experience with sensory substitution devices goes beyond assimilation to either the substituting or the substituted modality. Rather, it reflects a novel or a complex multisensory experience (e.g., Arnold et al., 2017; Proulx et al., 2014; see Figure 1 for an example of such an account). In line with this view, the corresponding phenomenology has been shown to resemble, in part, the substituting modality and, in part, vision (Pesnot-Lerousseau et al., 2021).

**Figure 1**
Three alternative perspectives on the processes involved in sensory substitution, with their specificities at the phenomenological, behavioral, and neural levels (from Arnold et al., 2017).

Neural plasticity

Neuroimaging studies showed increased activation in blind people’s visual cortex during the use of both visual-to-auditory and visual-to-tactile devices (e.g., Ptito et al., 2005) [see Neuroplasticity]. When examining smaller functional areas, additional studies revealed activation corresponding to the usual function. For instance, reading with the eyes or with a sensory substitution device activates the same area (the visual word form area), or the perception of body shapes activates the same area, whether by vision or by a sensory substitution device (the extrastriate body area; Striem-Amit & Amedi, 2014; see Maidenbaum et al., 2014, for a review). These results suggest that the visual cortex might not be purely visual and that its functional specialization might be independent of visual input.

Questions, controversies, and new developments

At the fundamental level, there are still key unanswered questions regarding the mechanisms of brain plasticity, particularly concerning development. The main actual prominent theories recognize that functional brain organization may be sensory independent. However, when it comes to development, for the cross-modal plasticity theory (e.g., Proulx et al., 2014), the plasticity is believed to result from cortical reorganization following sensory deprivation. On the other hand, the meta/supramodal representations (e.g., Kupers & Ptito, 2011) are believed to exist irrespective of developmental experience. To reconcile these views, Kupers and Ptito (2011) suggested that they are not mutually exclusive, as the cross-modal plasticity arising from sensory deprivation may be the result of an unmasking of existing supramodal organization. In addition, the crucial question of the connection between the neural mechanisms underlying sensory substitution and the phenomenology that users experience remains to be explored to clarify the processes at stake in sensory substitution.

At the applied level, although sensory substitution appears as a promising technology, the devices remain barely used by people with sensory impairments, either because their design is not optimal or because their use involves a high cognitive effort. To overcome this, current research axes involve creating ergonomically tailored devices and customizing the learning protocols as a function of each user’s individual abilities and the different types of use (Elli et al., 2014).

Broader connections

The field of perceptual augmentation uses principles similar to those of sensory substitution to provide users with additional information about the outside world. For instance, devices are created to gain access to visual information outside of the visible light (ultraviolet or infrared), to a cardinal direction (such as the north), or to the magnetic field (Eagleman & Perrotta, 2023) by translating this unusual information into tactile vibrations transmitted to the user.

The studies of the perceptual processes at stake in sensory substitution are indissociable of the study of perception with the usual sensory modalities and of the processes common to several sensory modalities [see Hearing, Visual Cognitive Neuroscience, Spatial Cognition]. In addition, as a form of visual imagery has been shown to be involved (see Pesnot-Lerousseau et al., 2021), it is also indissociable from the study of perception without a direct stimulation of the senses [see Mental Imagery]. In addition, learning a novel form of perception [see Perceptual Learning], here with sensory substitution devices, reveals functional and brain plasticity [see Neuroplasticity]. Finally, sensory substitution provides insights on the philosophical question of the “taxonomy of the senses,” suggesting additional ways to investigate the extent to which the brain could be organized in a task-dependent rather than in a sensory-dependent manner.

Sensory Substitution