Sound symbolism refers to non‐arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as “maluma” and “takete” with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity (...) analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics‐to‐noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound‐to‐shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross‐modal correspondence as a basis for language users' sensitivity to this type of sound symbolism. (shrink)

The auditory system is able to recognize auditory objects and is thought to form predictive models of them even though the acoustic information arriving at our ears is often imperfect, intermixed, or distorted. We investigated implicit regularity extraction for acoustically intact versus disrupted six-tone sound patterns via event-related potentials. In an exact-repetition condition, identical patterns were repeated; in two distorted-repetition conditions, one randomly chosen segment in each sound pattern was replaced either by white noise or by a wrong (...) pitch. In a roving-standard paradigm, sound patterns were repeated 1–12 times in a row before a new pattern occurred. The participants were not informed about the roving rule and had to detect rarely occurring loudness changes. Behavioral detectability of pattern changes was assessed in a subsequent behavioral task. Pattern changes elicited mismatch negativity and P3a, and were behaviorally detected above the chance level in all conditions, suggesting that the auditory system extracts regularities despite distortions in the acoustic input. However, MMN and P3a amplitude were decreased by distortions. At the level of MMN, both types of distortions caused similar impairments, suggesting that auditory regularity extraction is largely determined by the stimulus statistics of matching information. At the level of P3a, wrong-pitch distortions caused larger decreases than white-noise distortions. Wrong-pitch distortions likely prevented the engagement of restoration mechanisms and the segregation of disrupted from true pattern segments, causing stronger informational interference with the relevant pattern information. (shrink)

Attention is often conceived as a gateway to consciousness . Although endogenous spatial attention may be independent of conscious perception , exogenous spatial orienting seems instead to be an important modulator of CP . Here, we investigate the role of auditory alerting in CP in normal observers. We used a behavioral task in which phasic alerting tones were presented either at unpredictable or at predictable time intervals prior to the occurrence of a near-threshold visual target. We find, for the (...) first time in neurologically intact observers, that phasic alertness increases CP, both objectively and subjectively. This result is consistent with evidence showing that phasic alerting can ameliorate the spatial bias exhibited by visual neglect patients . The alerting network may increase the activity of fronto-parietal networks involved in top-down amplification required to bring a stimulus into consciousness. (shrink)

Though dividing one’s attention between two input streams typically impairs performance, detecting a behaviorally relevant stimulus can sometimes enhance the encoding of unrelated information presented at the same time. Previous research has shown that selection of this kind boosts visual cortical activity and memory for concurrent items. An important unanswered question is whether such effects are reflected in processing quality and functional connectivity in visual regions and in the hippocampus. In this fMRI study, participants were asked to memorize a (...) stream of naturalistic images and press a button only when they heard a predefined target tone. Images could be presented with a target tone, with a distractor tone, or without a tone. Auditory target detection increased activity throughout the ventral visual cortex but lowered it in the hippocampus. Enhancements in functional connectivity between the ventral visual cortex and the hippocampus were also observed following auditory targets. Multi-voxel pattern classification of image category was more accurate on target tone trials than on distractor and no tone trials in the fusiform gyrus and parahippocampal gyrus. This effect was stronger in visual cortical clusters whose activity was more correlated with the hippocampus on target tone than on distractor tone trials. In agreement with accounts suggesting that subcortical noradrenergic influences play a role in the attentional boost effect, auditory target detection also caused an increase in locus coeruleus activity and phasic pupil responses. These findings outline a network of cortical and subcortical regions that are involved in the selection and processing of information presented at behaviorally relevant moments. (shrink)

Voluntary stream segregation was investigated in cochlear implant users and normal-hearing listeners using a segregation-promoting objective approach which evaluated the role of spectral and amplitude-modulation rate separations on stream segregation and its build-up. Sequences of 9 or 3 pairs of A and B narrowband noise bursts were presented which differed in either center frequency of the noise band, the AM-rate, or both. In some sequences, the last B burst was delayed by 35 ms from their otherwise-steady temporal position. In the (...) other sequences, the last B bursts were temporally advanced from 0 to 10 ms. A single interval yes/no procedure was utilized to measure participants’ sensitivity in identifying delayed vs. no-delay sequences. A higher d′ value showed the higher ability to segregate the A and B subsequences. For NH listeners, performance improved with each spectral separation. However, for CI users, performance was only significantly better for the condition with the largest spectral separation. Additionally, performance was significantly poorer for the largest AM-rate separation than for the condition with no AM-rate separation for both groups. The significant effect of sequence duration in both groups indicated that listeners made more improvement with lengthening the duration of stimulus sequences, supporting the build-up effect. The results of this study suggest that CI users are less able than NH listeners to segregate NBN bursts into different auditory streams when they are moderately separated in the spectral domain. Contrary to our hypothesis, our results indicate that AM-rate separation may interfere with the segregation of streams of NBN. Additionally, our results add evidence to the literature that CI users build up stream segregation at a rate comparable to NH listeners, when the inter-stream spectral separations are adequately large. (shrink)

Many situations require the simultaneous processing of auditory and visual information, however, stimuli presented to one sensory modality can sometimes interfere with processing in a second sensory modality (i.e., modality dominance). The current study further investigated modality dominance by examining how task demands and bimodal presentation affect speeded auditory and visual discriminations. Participants in the current study had to quickly determine if two words, two pictures, or two word-picture pairings were the same or different, and we manipulated task (...) demands across three different conditions. In an immediate recognition task, there was only one second between the two stimuli/stimulus pairs and auditory dominance was found. Compared to the respective unimodal baselines, pairing pictures and words together slowed down visual responses and sped up auditory responses. Increasing the interstimulus interval to four seconds and blocking verbal rehearsal weakened auditory dominance effects, however, conflicting and redundant visual cues sped up auditory discriminations. Thus, simultaneously presenting pictures and words had different effects on auditory and visual processing, with bimodal presentation slowing down visual processing and speeding up auditory processing. These findings are consistent with a proposed mechanism underlying auditory dominance, which posits that auditory stimuli automatically grab attention and attenuate/delay visual processing. (shrink)

Asymmetric distribution of function between the cerebral hemispheres has been widely investigated in the auditory modality. The current approach borrows heavily from visual local–global research in an attempt to determine whether, as in vision, local–global auditory processing is lateralised. In vision, lateralised local–global processing likely relies on spatial frequency information. Drawing analogies between visual spatial frequency and auditory dimensions, two sets of auditory stimuli were developed. In the high–low stimulus set we manipulate frequency information, and (...) in the fast–slow stimulus set we manipulate temporal information. The fast–slow stimuli additionally mimic visual hierarchical stimulus structure, in which the arrangement of local patterns determines the global pattern. Unlike previous auditory stimuli, the current stimulus sets contain the experimental flexibility of visual local–global hierarchical stimuli allowing independent manipulation of structural levels. Previous findings of frequency and temporal range priming were replicated. Additionally, by presenting stimuli monaurally, we found that priming of frequency ranges (but not temporal ranges) was found to vary by ear, supporting the contention that the hemispheres asymmetrically retain traces of prior frequency processing. These results contribute to the extensive literature revealing cerebral asymmetries for the processing of frequency information, and extend those results to the realm of priming. (shrink)

Mismatch negativity is the electroencephalographic waveform obtained by subtracting event-related potential responses evoked by unexpected deviant stimuli from responses evoked by expected standard stimuli. While the MMN is thought to reflect an unexpected change in an ongoing, predictable stimulus, it is unknown whether MMN responses evoked by changes in different stimulus features have different magnitudes, latencies, and topographies. The present study aimed to investigate whether MMN responses differ depending on whether sudden stimulus change occur in pitch, duration, (...) location or vowel identity, respectively. To calculate ERPs to standard and deviant stimuli, EEG signals were recorded in normal-hearing participants who listened to roving oddball sequences of artificial syllables. In the roving paradigm, any given stimulus is repeated several times to form a standard, and then suddenly replaced with a deviant stimulus which differs from the standard. Here, deviants differed from preceding standards along one of four features. The feature levels were individually chosen to match behavioral discrimination performance. We identified neural activity evoked by unexpected violations along all four acoustic dimensions. Evoked responses to deviant stimuli increased in amplitude relative to the responses to standard stimuli. A univariate analysis yielded no significant differences between MMN responses following violations of different features. However, in a multivariate analysis, acoustic features could be decoded from the topography of mismatch responses, although at later latencies than those typical for MMN. These results support the notion that deviant feature detection may be subserved by a different process than general mismatch detection. (shrink)

Background. Attending to goal-relevant information can leave us metaphorically ‘blind’ or ‘deaf’ to the next relevant information while searching among distracters. This temporal cost lasting for about a half a second on the human selective attention has been long explored using the attentional blink paradigm. Although there is evidence that certain visual stimuli relating to one’s area of expertise can be less susceptible to attentional blink effects, it remains unexplored whether the dynamics of temporal selective attention vary with expertise and (...) objects types in the auditory modality. Methods. Using the auditory version of the attentional blink paradigm, the present study investigates whether certain auditory objects relating to musical and perceptual expertise could have an impact on the transient costs of selective attention. In this study, expert cellists and novice participants were asked to first identify a target sound, and then to detect instrumental timbres of cello or organ, or human voice as a second target in a rapid auditory stream. Results. The results showed moderate evidence against the attentional blink effect for voices independent of participants’ musical expertise. Experts outperformed novices in their overall accuracy levels of target identification and detection, reflecting a clear benefit of musical expertise. Importanly, the musicianship advantage disappeared when the human voices served as the second target in the stream. Discussion. The results are discussed in terms of stimulus salience, the advantage of voice processing, as well as perceptual and musical expertise in relation to attention and working memory performances. (shrink)

A period of exposure to trains of simultaneous but spatially offset auditory and visual stimuli can induce a temporary shift in the perception of sound location. This phenomenon, known as the 'ventriloquist aftereffect', reflects a realignment of auditory and visual spatial representations such that they approach perceptual alignment despite their physical spatial discordance. Such dynamic changes to sensory representations are likely to underlie the brain's ability to accommodate inter-sensory discordance produced by sensory errors (particularly in sound localization) and (...) variability in sensory transduction. It is currently unknown, however, whether these plastic changes induced by adaptation to spatially disparate inputs occurs automatically or whether they are dependent on selectively attending to the visual or auditory stimuli. Here, we demonstrate that robust auditory spatial aftereffects can be induced even in the presence of a competing visual stimulus. Importantly, we found that when attention is directed to the competing stimuli, the pattern of aftereffects is altered. These results indicate that attention can modulate the ventriloquist aftereffect. © 2010 Elsevier B.V. (shrink)

People can discriminate the synchrony between audio-visual scenes. However, the sensitivity of audio-visual synchrony perception can be affected by many factors. Using a simultaneity judgment task, the present study investigated whether the synchrony perception of complex audio-visual stimuli was affected by audio-visual causality and stimulus reliability. In Experiment 1, the results showed that audio-visual causality could increase one's sensitivity to audio-visual onset asynchrony (AVOA) of both action stimuli and speech stimuli. Moreover, participants were more tolerant of AVOA of speech (...) stimuli than that of action stimuli in the high causality condition, whereas no significant difference between these two kinds of stimuli was found in the low causality condition. In Experiment 2, the speech stimuli were manipulated with either high or low stimulus reliability. The results revealed a significant interaction between audio-visual causality and stimulus reliability. Under the low causality condition, the percentage of “synchronous” responses of audio-visual intact stimuli was significantly higher than that of visual_intact/auditory_blurred stimuli and audio-visual blurred stimuli. In contrast, no significant difference among all levels of stimulus reliability was observed under the high causality condition. Our study supported the synergistic effect of top-down processing and bottom-up processing in audio-visual synchrony perception. (shrink)

Four experiments examined how accurately participants can report the times of their own decisions. Within an auditory reaction time task, participants reported the time at which the tone was presented, they decided on the response, or the response key was pressed. Decision time reports were checked for plausibility against the actual RTs, and we compared the effects of experimental manipulations on these two measures to see whether the reported decision times showed appropriate effects. In addition, we estimated the amount (...) of error associated with individual decision time reports by checking how often participants’ decision time reports were implausibly early or late , and by using several quantitative models. Overall, the results suggest that decision time reports are not very accurate but they may be usable for some purposes. (shrink)

People are ingesting various information from different sense organs all the time to complete different cognitive tasks. The brain integrates and regulates this information. The two significant sensory channels for receiving external information are sight and hearing that have received extensive attention. This paper mainly studies the effect of music and visual-auditory stimulation on electroencephalogram of happy emotion recognition based on a complex system. In the experiment, the presentation was used to prepare the experimental stimulation program, and the cognitive (...) neuroscience experimental paradigm of EEG evoked by happy emotion pictures was established. Using 93 videos as natural stimuli, fMRI data were collected. Finally, the collected EEG signals were removed with the eye artifact and baseline drift, and the t-test was used to analyze the significant differences of different lead EEG data. Experimental data shows that, by adjusting the parameters of the convolutional neural network, the highest accuracy of the two-classification algorithm can reach 98.8%, and the average accuracy can reach 83.45%. The results show that the brain source under the combined visual and auditory stimulus is not a simple superposition of the brain source of the single visual and auditory stimulation, but a new interactive source is generated. (shrink)

Priming is a useful tool for ascertaining the circumstances under which previous experiences influence behavior. Previously, using hierarchical stimuli, we demonstrated (Justus & List, 2005) that selectively attending to one temporal scale of an auditory stimulus improved subsequent attention to a repeated (vs. changed) temporal scale; that is, we demonstrated intertrial auditory temporal level priming. Here, we have extended those results to address whether level priming relied on absolute or relative temporal information. Both relative and absolute temporal (...) information are important in auditory perception: Speech and music can be recognized over various temporal scales but become uninterpretable to a listener when presented too quickly or slowly. We first confirmed that temporal level priming generalized over new temporal scales. Second, in the context of multiple temporal scales, we found that temporal level priming operates predominantly on the basis of relative, rather than absolute, temporal information. These findings are discussed in the context of expectancies and relational invariance in audition. (shrink)