The purpose of this study is to discuss the effects of audiovisual input on second language acquisition and the factors that influence the difficulty of audiovisual learning through a systematic literature review. Prior to this systematic review, in this paper, we searched papers on related topics for the past 10 years from 2012 to 2022, and found 46 journal papers that met the research criteria. They can basically represent the scholarly work in this field. The 46 studies were published (...) in journals indexed in Google Scholar, Eric, Scopus, and Wiley Library. Databases were selected according to a set of inclusion and exclusion criteria. The following conclusions are drawn from the literature review: Audiovisual input can provide more authentic language input and more adequate and richer multimodal cultural and situational contexts, which can better promote learners' understanding of the content and stimulate learners' interest in participating in listening comprehension tasks. The influencing factors of multimodal input on listening difficulty include subtitles, video type, and the relationship between the audio and visual input. (shrink)

Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in noise [J. Acoust. Soc. Am. 109 (2001) 2272; J. Acoust. Soc. Am. 108 (2000) 1197]. However, detection is different from comprehension, and it remains (...) to be seen whether improved sensitivity also results in an intelligibility gain in audio-visual speech perception. In this work, we use an original paradigm to show that seeing the speaker's lips enables the listener to hear better and hence to understand better. The audio-visual stimuli used here could not be differentiated by lip reading per se since they contained exactly the same lip gesture matched with different compatible speech sounds. Nevertheless, the noise-masked stimuli were more intelligible in the audio-visual condition than in the audio-only condition due to the contribution of visual information to the extraction of acoustic cues. Replacing the lip gesture by a non-speech visual input with exactly the same time course, providing the same temporal cues for extraction, removed the intelligibility benefit. This early contribution to audio-visual speech identification is discussed in relationships with recent neurophysiological data on audio-visual perception. (shrink)

Infant and adult learners are able to identify word boundaries in fluent speech using statistical information. Similarly, learners are able to use statistical information to identify word–object associations. Successful language learning requires both feats. In this series of experiments, we presented adults and infants with audio–visual input from which it was possible to identify both word boundaries and word–object relations. Adult learners were able to identify both kinds of statistical relations from the same input. Moreover, their learning (...) was actually facilitated by the presence of two simultaneously present relations. Eight-month-old infants, however, do not appear to benefit from the presence of regular relations between words and objects. Adults, like 8-month-olds, did not benefit from regular audio–visual correspondences when they were tested with tones, rather than linguistic input. These differences in learning outcomes across age and input suggest that both developmental and stimulus-based constraints affect statistical learning. (shrink)

Recent work asked if visual cues facilitate word segmentation in adults and infants. While adults showed better word segmentation when presented with a regular visual cue, infants did not. This difference was attributed to infants' lack of understanding that objects have labels. Alternatively, infants’ performance could reflect their difficulty with tracking and integrating multiple multimodal cues. We contrasted these two accounts by looking at the effect of visual cues on word segmentation in adults and across childhood. We found that older (...) children benefitted from the regular visual cues, but younger children, who already knew that objects have labels, did not. Knowing that objects have labels was not enough to use visual cues as an aid for segmentation. These findings show that the ability to integrate multimodal cues develops during childhood, and it is not yet adult-like in children. (shrink)

Cerebral Visual Impairment is an umbrella term which includes abnormalities in visual acuity, or contrast sensitivity or colour; ocular motility; visual field and the conscious and unconscious filtering or processing of visual input. Children with CVI have specific needs and problems relating to their development from infancy to adulthood which can impact on their wellbeing. Recent research indicates the complexities of living with CVI but there remains limited information of the full impact of CVI on families’ everyday lives. The (...) qualitative interviews reported here explored families’ experiences to discover the impact of CVI on all aspects of everyday life. Parents and children were invited to participate in semi-structured interviews, either face to face, by phone or video call between January 2018 and February 2019. Topics covered everyday practicalities of living with CVI, focusing on challenges and what worked well at school and home. Interviews were audio-recorded and subject to thematic analysis to look for patterns across the data. Twenty families took part in interviews, with eight children/young people within those families contributing interviews of their own. Four themes were developed from the interviews: Assessment and understanding implications of CVI, Education, Family life, Psychological wellbeing and quality of life. The interviews provide valuable insights into the impact of living with CVI and highlight the need for more awareness of the condition among professionals in both health and education settings. (shrink)

In a century where humans and machines—powered by artificial intelligence or not—increasingly work together, it is of interest to understand human processing of multi-sensory stimuli in relation to attention and working memory. This paper explores whether and when supporting visual information with rhythmic auditory stimuli can optimize multi-sensory information processing. In turn, this can make the interaction between humans or between machines and humans more engaging, rewarding and activating. For this purpose a novel working memory paradigm was developed where participants (...) are presented with a series of five target digits randomly interchanged with five distractor digits. Their goal is to remember the target digits and recall them orally. Depending on the condition support is provided by audio and/or rhythm. It is expected that the sound will lead to a better performance. It is also expected that this effect of sound is different in case of rhythmic and non-rhythmic sound. Last but not least, some variability is expected across participants. To make correct conclusions, the data of the experiment was statistically analyzed in a classic way, but also predictive models were developed in order to predict outcomes based on a range of input variables related to the experiment and the participant. The effect of auditory support could be confirmed, but no difference was observed between rhythmic and non-rhythmic sounds. Overall performance was indeed affected by individual differences, such as visual dominance or perceived task difficulty. Surprisingly a music education did not significantly affect the performance and even tended toward a negative effect. To better understand the underlying processes of attention, also brain activation data, e.g., by means of electroencephalography, should be recorded. This approach can be subject to a future work. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. /gi/) (...) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

We examined whether facial expressions of performers influence the emotional connotations of sung materials, and whether attention is implicated in audio-visual integration of affective cues. In Experiment 1, participants judged the emotional valence of audio-visual presentations of sung intervals. Performances were edited such that auditory and visual information conveyed congruent or incongruent affective connotations. In the single-task condition, participants judged the emotional connotation of sung intervals. In the dual-task condition, participants judged the emotional connotation of intervals while performing a secondary (...) task. Judgements were influenced by melodic cues and facial expressions and the effects were undiminished by the secondary task. Experiment 2 involved identical conditions but participants were instructed to base judgements on auditory information alone. Again, facial expressions influenced judgements and the effect was undiminished by the secondary task. The results suggest that visual aspects of music performance are automatically and preattentively registered and integrated with auditory cues. (shrink)

People can discriminate the synchrony between audio-visual scenes. However, the sensitivity of audio-visual synchrony perception can be affected by many factors. Using a simultaneity judgment task, the present study investigated whether the synchrony perception of complex audio-visual stimuli was affected by audio-visual causality and stimulus reliability. In Experiment 1, the results showed that audio-visual causality could increase one's sensitivity to audio-visual onset asynchrony (AVOA) of both action stimuli and speech stimuli. Moreover, participants were more tolerant of AVOA of speech stimuli (...) than that of action stimuli in the high causality condition, whereas no significant difference between these two kinds of stimuli was found in the low causality condition. In Experiment 2, the speech stimuli were manipulated with either high or low stimulus reliability. The results revealed a significant interaction between audio-visual causality and stimulus reliability. Under the low causality condition, the percentage of “synchronous” responses of audio-visual intact stimuli was significantly higher than that of visual_intact/auditory_blurred stimuli and audio-visual blurred stimuli. In contrast, no significant difference among all levels of stimulus reliability was observed under the high causality condition. Our study supported the synergistic effect of top-down processing and bottom-up processing in audio-visual synchrony perception. (shrink)

Multimodal integration is the formation of a coherent percept from different sensory inputs such as vision, audition, and somatosensation. Most research on multimodal integration in speech perception has focused on audio-visual integration. In recent years, audio-tactile integration has also been investigated, and it has been established that puffs of air applied to the skin and timed with listening tasks shift the perception of voicing by naive listeners. The current study has replicated and extended these findings by testing the effect of (...) air puffs on gradations of voice onset time along a continuum rather than the voiced and voiceless endpoints of the original work. Three continua were tested: bilabial (“pa/ba”), velar (“ka/ga”), and a vowel continuum (“head/hid”) used as a control. The presence of air puffs was found to significantly increase the likelihood of choosing voiceless responses for the two VOT continua but had no effect on choices for the vowel continuum. Analysis of response times revealed that the presence of air puffs lengthened responses for intermediate (ambiguous) stimuli and shortened them for endpoint (non-ambiguous) stimuli. The slowest response times were observed for the intermediate steps for all three continua, but for the bilabial continuum this effect interacted with the presence of air puffs: responses were slower in the presence of air puffs, and faster in their absence. This suggests that during integration auditory and aero-tactile inputs are weighted differently by the perceptual system, with the latter exerting greater influence in those cases where the auditory cues for voicing are ambiguous. (shrink)

While influential works since the 1970s have widely assumed that imitation is an innate skill in both human and non-human primate neonates, recent empirical studies and meta-analyses have challenged this view, indicating other forms of reward-based learning as relevant factors in the development of social behavior. The visual input translation into matching motor output that underlies imitation abilities instead seems to develop along with social interactions and sensorimotor experience during infancy and childhood. Recently, a new visual stream has been (...) identified in both human and non-human primate brains, updating the dual visual stream model. This third pathway is thought to be specialized for dynamics aspects of social perceptions such as eye-gaze, facial expression and crucially for audio-visual integration of speech. Here, we review empirical studies addressing an understudied but crucial aspect of speech and communication, namely the processing of visual orofacial cues and its integration with vocal auditory cues. Along this review, we offer new insights from our understanding of speech as the product of evolution and development of a rhythmic and multimodal organization of sensorimotor brain networks, supporting volitional motor control of the upper vocal tract and audio-visual voices-faces integration. (shrink)

The McGurk illusion occurs when listeners hear an illusory percept, resulting from mismatched pairings of audiovisual speech stimuli. Hearing a third percept—distinct from both the auditory and visual input—has been used as evidence of AV fusion. We examined whether the McGurk illusion is instead driven by visual dominance, whereby the third percept, e.g., “da,” represents a default percept for visemes with an ambiguous place of articulation, like/ga/. Participants watched videos of a talker uttering various consonant vowels with and without (...) audios of/ba/. Individuals transcribed the CV they saw or heard. In the V-only condition, individuals predominantly saw “da”/“ta” when viewing CVs with indiscernible POAs. Likewise, in the AV condition, upon perceiving an illusion, they predominantly heard “da”/“ta” for CVs with indiscernible POAs. The illusion was stronger in individuals who exhibited weak/ba/auditory encoding. In Experiment2, we attempted to replicate these findings using stimuli recorded from a different talker. The V-only results were not replicated, but again individuals predominately heard “da”/“ta”/“tha” as an illusory percept for various AV combinations, and the illusion was stronger in individuals who exhibited weak/ba/auditory encoding. These results demonstrate that when visual CVs with indiscernible POAs are paired with a weakly encoded auditory/ba/, listeners default to hearing “da”/“ta”/“tha”—thus, tempering the AV fusion account, and favoring a default mechanism triggered when both AV stimuli are ambiguous. (shrink)

In this paper we offer a theory of cross-modal objects. To begin, we discuss two kinds of linkages between vision and audition. The first is a duality. The the visual system detects and identifies surfaces ; the auditory system detects and identifies sources . Surfaces are illuminated by sources of light; sound is reflected off surfaces. However, the visual system discounts sources and the auditory system discounts surfaces. These and similar considerations lead to the Theory of Indispensable Attributes that states (...) the conditions for the formation of gestalts in the two modalities. The second linkage involves the formation of audiovisual objects, integrated cross-modal experiences. We describe research that reveals the role of cross-modal causality in the formation of such objects. These experiments use the canonical example of a causal link between vision and audition: a visible impact that causes a percussive sound. (shrink)

It is commonly believed that human perceptual experiences can be, and usually are, multimodal. What is more, a stronger thesis is often proposed that some perceptual multimodal characters cannot be described simply as a conjunction of unimodal phenomenal elements. If it is the case, then a question arises: what is the additional mode of combination that is required to adequately describe the phenomenal structure of multimodal experiences? The paper investigates what types of audio–visual experiences have phenomenal character that cannot (...) be analysed as a mere conjunction of visual and auditory elements; and how can we properly characterise the required, additional mode of perceptual combination. Three main modes of combination are considered: instantiation, parthood, and grouping. It is argued that some phenomena involving intermodal relations, like spatial and temporal ventriloquism, can be analysed in terms of audio–visual, perceptual grouping. On the other hand, cases of intermodal binding need a different treatment. Experiences involving audio–visual binding should be analysed as experiences presenting objects or events which instantiate, or which have a proper part instantiating, both visually and auditorily determined properties. (shrink)

Experimental research has shown that pairs of stimuli which are congruent and assumed to 'go together' are recalled more effectively than an item presented in isolation. Will this multisensory memory benefit occur when stimuli are richer and longer, in an ecological setting? In the present study, we focused on an everyday situation of audio-visual learning and manipulated the relationship between audio guide tracks and viewed portraits in the galleries of the Tate Britain. By varying the gender and narrative style of (...) the voice-over, we examined how the perceived congruency and assumed unity of the audio guide track with painted portraits affected subsequent recall. We show that tracks perceived as best matching the viewed portraits led to greater recall of both sensory and linguistic content. We provide the first evidence that manipulating crossmodal congruence and unity assumptions can effectively impact memory in a multisensory ecological setting, even in the absence of precise temporal alignment between sensory cues. (shrink)

Utilizing a pre-experimental pre-test post-test design, this study investigated the effect of an authentic audio-visual American sitcom on the intercultural sensitivity and intercultural effectiveness of a sample of male and female upper-intermediate English students. To this aim, 34 Iranian EFL students were selected through convenient non-random sampling. In order to assure the participants' homogeneity in English proficiency, the selected students were given the Oxford Quick Placement Test prior to the intervention. Over a 10-week period, the participants were presented with nearly (...) 30-min-long episodes of Friends in a flipped context, two episodes every week uploaded to the accompanying website. An online 10-item quiz on the day of watching and a next-day meeting were held. During the online discussion meetings, the participants discussed cultural questions tailored to the aims of the study. The required data were collected through the administration of the intercultural sensitivity scale and intercultural effectiveness scale in pre- and post-test assessments. The analyses of the data indicated that both ICS and ICE improved significantly during the intervention. However, ICS indicated more improvement. In addition, a semi-structured interview was administered to the participants to examine their perception of the flipped classroom experience. Having analyzed the data using MAXQDA 2020, some significant themes emerged which are reported. On the whole, the participants portrayed positive attitudes toward using technology in flipped classes. The educational and practical implications are discussed. (shrink)

This paper has already been published in Sociological Research Online, 21/3, August 2016. All rights reserved. © SOCIOLOGICAL RESEARCH ONLINE. It is available from Kent Academic Repository. We thank Dawn Lyon for the permission to republish it here.: This article documents, shows and analyses the everyday rhythms of Billingsgate, London's wholesale fish market. It takes the form of a short film based an audiovisual montage of time-lapse photography and sound recordings, and a - Sociologie – Nouvel article.

We investigate whether smile mimicry and emotional contagion are evident in non-text-based computer-mediated communication (CMC). Via an ostensibly real-time audio-visual CMC platform, participants interacted with a confederate who either smiled radiantly or displayed a neutral expression throughout the interaction. Automatic analyses of expressions displayed by participants indicated that smile mimicry was at play: A higher level of activation of the facial muscle that characterises genuine smiles was observed among participants who interacted with the smiling confederate than among participants who interacted (...) with the unexpressive confederate. However, there was no difference in the self-reported level of joviality between participants in the two conditions. Our findings demonstrate that people mimic smiles in audio-visual CMC, but that even though the diffusion of emotions has been documented in text-based CMC in previous studies, we find no convincing support for the phenomenon of emotional contagion in non-text-based CMC. (shrink)

This article investigates the hypothesis of creating non-linear audio-visual narratives, through an unanticipated use of traditional print-based games, enriched with videos, via augmented reality (AR) possibilities. A ludic system has been created and presented. Based on a traditional card game, a non-linear cinematic narrative occurs. We attempt to examine the following questions: in which way can we bring together different forms of visual communication, such as graphic design and video? Can the above forms create a complex narrative whole and what (...) kind of rules will be needed for this? How can we enrich traditional forms of gaming with the potentials of AR? Gaming itself demands a set of rules. Can these rules play the role of algorithms in the combined universe that we have designed and created? In which way can the designer on the one hand and the user on the other influence the overall output of the system? What will the user experience be like? The printed card system chosen for this is Tarot and more precisely the Great Arcana, which makes use of the 22 fundamental Tarot figures. (shrink)

This article provides an ethical analysis of the moral conflict over access to premium audio-visual online content, which is currently regulated by geo-blocking in the domain of digital film distri...

This paper revisits some early applications of audio-visual imaging technologies used in physiology in a dialogue with reflections on Henri Bergson’s philosophy. It focuses on the aspects of time and memory in relation to spatial representations of movement measurements and critically discusses them from the perspective of the observing participant and the public exhibitions of scientific films. Departing from an audio-visual example, this paper is informed by a thick description of the philosophical implications and contemporary discourses surrounding the scientific inventions, (...) technologies and theories of moving image technologies, representations of time, and the measurement of bodies in motion such as those by the French physiologist Étienne-Jules Marey. The application of Bergson’s thinking will show how the scientific graphing and representations of the body through technologies inevitably stand in tension with the limits and potentials of the ‘human apparatus’ and its abilities, filters and internalised processes of perception, memory and consciousness. It proposes an ontological approach to scientific imaging technologies and a critical engagement with the ‘realism’ debate in the discourse of audio-visual media. (shrink)

Previous research with speech and non-speech stimuli suggested that in audiovisual perception, visual information starting prior to the onset of corresponding sound can provide visual cues, and form a prediction about the upcoming auditory sound. This prediction leads to audiovisual interaction. Auditory and visual perception interact and induce suppression and speeding up of the early auditory event-related potentials such as N1 and P2. To investigate AV interaction, previous research examined N1 and P2 amplitudes and latencies in response to audio only, (...) video only, audiovisual, and control stimuli, and compared AV with auditory perception based on four AV interaction models. The current study addresses how different models of AV interaction express N1 and P2 suppression in music perception. Furthermore, the current study took one step further and examined whether previous musical experience, which can potentially lead to higher N1 and P2 amplitudes in auditory perception, influenced AV interaction in different models. Musicians and non-musicians were presented the recordings of a keyboard /C4/ key being played, as well as CO stimuli. Results showed that AV interaction models differ in their expression of N1 and P2 amplitude and latency suppression. The calculation of model and has consequences for the resulting N1 and P2 difference waves. Furthermore, while musicians, compared to non-musicians, showed higher N1 amplitude in auditory perception, suppression of amplitudes and latencies for N1 and P2 was similar for the two groups across the AV models. Collectively, these results suggest that when visual cues from finger and hand movements predict the upcoming sound in AV music perception, suppression of early ERPs is similar for musicians and non-musicians. Notably, the calculation differences across models do not lead to the same pattern of results for N1 and P2, demonstrating that the four models are not interchangeable and are not directly comparable. (shrink)

The sense of agency is a subjective feeling that one's own actions drive action outcomes. Previous studies have focused primarily on the temporal contingency between actions and sensory inputs as a possible mechanism for the sense of agency. However, the contribution of the integrity of visual inputs has not been systematically addressed. In the current study, we developed a psychophysical task to examine the role of visual inputs as well as temporal contingencies toward the sense of agency. Specifically, participants were (...) required to track a target on a sinusoidal curve on a computer screen. Visual integrity of sensory inputs was manipulated by gradually occluding a computer cursor, and participants were asked to report the sense of agency on a nine-point Likert scale. Temporal contingency was manipulated by varying the delay between finger movements on a touchpad and cursor movements. The results showed that the sense of agency was influenced by both visual integrity and temporal contingency. These results are discussed in the context of current models that have proposed that the sense of agency emerges from the comparison of visual inputs with motor commands. (shrink)

During the COVID-19 pandemic, questions have been raised about the impact of face masks on communication in classroom settings. However, it is unclear to what extent visual obstruction of the speaker’s mouth or changes to the acoustic signal lead to speech processing difficulties, and whether these effects can be mitigated by semantic predictability, i.e., the availability of contextual information. The present study investigated the acoustic and visual effects of face masks on speech intelligibility and processing speed under varying semantic predictability. (...) Twenty-six children and twenty-six adults performed an internet-based cued shadowing task, in which they had to repeat aloud the last word of sentences presented in audio-visual format. The results showed that children and adults made more mistakes and responded more slowly when listening to face mask speech compared to speech produced without a face mask. Adults were only significantly affected by face mask speech when both the acoustic and the visual signal were degraded. While acoustic mask effects were similar for children, removal of visual speech cues through the face mask affected children to a lesser degree. However, high semantic predictability reduced audio-visual mask effects, leading to full compensation of the acoustically degraded mask speech in the adult group. Even though children did not fully compensate for face mask speech with high semantic predictability, overall, they still profited from semantic cues in all conditions. Therefore, in classroom settings, strategies that increase contextual information such as building on students’ prior knowledge, using keywords, and providing visual aids, are likely to help overcome any adverse face mask effects. (shrink)

In 2015 the first mass migration and the first geopolitical and transnational crisis of the digital age began. This brought along the awakening of a new phase in slavery trade and trafficking of human beings (THB) at the European borders. In this article I introduce an ongoing field study I began in January 2016, in collaboration with experts from Spanish and European institutions, that so far has taken me to hotspots and borders, and to the accumulation of a vast multimedia (...) archive. This led to the creation of zonaautonomamilitarizada.eu, an immersive/360°, interactive and reactive audio-visual performative system conceptually rooted in military technologies, second-order cybernetics and mass media developments. (shrink)

Pretendemos reflexionar sobre la manera en la que los discursos audiovisuales están hilvanando las relaciones entre crisis medioambiental y quiebra de los marcos comunitarios sociales. Partiendo de una metodología híbrida basada tanto en la filosofía del arte (y de la ecología) de Heidegger como del análisis narratológico, proponemos pensar el campo mediante dos modelos: las representaciones mainstream que se basan en la repetición de estructuras narrativas clásicas donde la amenaza climática es un conflicto solucionable mediante una intervención heroica y aquellos (...) textos complejos en los que se exige el replanteamiento de una eco(teo)logía que ponga en su foco la necesidad de reactivar la comunidad. (shrink)