Highly mobile computing devices promise to improve quality of life, productivity, and performance. Increased situation awareness and reduced mental workload are two potential means by which this can be accomplished. However, it is difficult to measure these concepts in the ‘wild’. We employed an ultra-portable battery operated and wireless functional near infrared spectroscopy (fNIRS) to non-invasively measure hemodynamic changes in the brain’s prefrontal cortex. Measurements were taken during navigation of a college campus with either a hand-held display, or an augmented (...) reality wearable display. Hemodynamic measures were also paired with secondary tasks of visual perception and auditory working memory to provide behavioral assessment of situation awareness and mental workload. Navigating with an augmented reality wearable display produced the least workload during both secondary tasks. The hemodynamics associated with errors were also different between the two devices. Errors with an augmented reality wearable display were associated with increased prefrontal activity and the opposite was observed for the hand-held display. This suggests that the cognitive mechanisms underlying errors between the two devices differ. These findings show fNIRS is a valuable tool for assessing new technology in ecologically valid settings and that head-mounted displays offer benefits with regards to mental workload while navigating, and potentially superior situation awareness with improved display design. (shrink)

We investigate the dynamics of sensory integration for perceiving musical performance, a complex natural behavior. Thirty musically trained participants saw, heard, or both saw and heard, performances by two clarinetists. All participants used a sliding potentiometer to make continuous judgments of tension (a measure correlated with emotional response) and continuous judgments of phrasing (a measure correlated with perceived musical structure) as performances were presented. The data analysis sought to reveal relations between the sensory modalities (vision and audition) and to quantify (...) the effect of seeing the performances on participants' overall subjective experience of the music. In addition to traditional statistics, functional data analysis techniques were employed to analyze time-varying aspects of the data. The auditory and visual channels were found to convey similar experiences of phrasing but different experiences of tension through much of the performances. We found that visual information served both to augment and to reduce the experience of tension at different points in the musical piece (as revealed by functional linear modeling and functional significance testing). In addition, the musicians' movements served to extend the sense of phrasing, to cue the beginning of new phrases, to indicate musical interpretation, and to anticipate changes in emotional content. Evidence for an interaction effect suggests that there may exist an emergent quality when musical performances are both seen and heard. The investigation augments knowledge of human communicative processes spanning language and music, and involving multiple modalities of emotion and information transfer. (shrink)

Despite the power of VR in immersing viewers in an experience, it generally only targets viewers via visual and auditory cues. Human beings use more senses to gather information, so expectedly, the full potential of this medium is currently not yet tapped. This study contributes in answering two research questions: How can conventional VR ads be enriched by also addressing the forgotten sense of smell?; and Does doing so indeed instill more engaging experiences? A 2 × 3 between-subjects study (...) is conducted, whereby an existing branded VR commercial is augmented with “sound” and “scents.” The power of these sensory augmentations is evaluated by inspecting emotional, cognitive and conative dimensions of customer engagement. The results identify product-scent congruence as a deal-maker, albeit product-scent incongruence is not necessarily a deal-breaker. The article concludes with further research avenues and a translation into managerial implications. (shrink)

When a user integrates a sensory substitution device into her life, the process involves perceptual learning, that is, ‘relatively long-lasting changes to an organism’s perceptual system that improve its ability to respond to its environment’ (Goldstone 1998: 585). In this paper, I explore ways in which the extensive literature on perceptual learning can be applied to help improve sensory substitution devices. I then use these findings to answer a philosophical question. Much of the philosophical debate surrounding sensory substitution devices concerns (...) what happens after perceptual learning occurs. In particular, should the resultant perceptual experience be classified in the substituted modality (as vision), in the substituting modality (as auditory or tactile), or in a new sense modality? I propose a novel empirical test to help resolve this philosophical debate. (shrink)

Sensory substitution devices (SSDs) are most familiar from their use with subjects who are deficient in a target modality (e.g. congenitally blind subjects), but there is no doubt that the use and potential value of SSDs extend to persons without such deficits. Recent work by Amedi and his team (in particular Levy-Tzedek et al. 2012) has begun to explore this. Their idea is that SSDs may facilitate behavioural transference (BT) across sense modalities. In this case, a motor skill learned through (...) visual perception might be subsequently employed in response to auditory perception, using an SSD as a mediator. They infer from the existence of such BT that the learned skill is amodally represented. After a brief overview I identify ways to more fully test for BT within this experimental paradigm and argue that their conclusion about amodal representation is premature. Additionally, I argue that their preferred SSD (Eyemusic) is of limited value for the project. While my remarks are critical, my intention is to be constructive, particularly in light of the fact that Levy-Tzedek et al. (2012), is, I believe, the first output from Amedi's lab concerning this line of research. (shrink)

Augmented reality (AR) technologies function to ‘augment’ normal perception by superimposing virtual objects onto an agent’s visual field. The philosophy of augmented reality is a small but growing subfield within the philosophy of technology. Existing work in this subfield includes research on the phenomenology of augmented experiences, the metaphysics of virtual objects, and different ethical issues associated with AR systems, including (but not limited to) issues of privacy, property rights, ownership, trust, and informed consent. This paper addresses some epistemological issues (...) posed by AR systems. I focus on a near-future version of AR technology called the Real-World Web, which promises to radically transform the nature of our relationship to digital information by mixing the virtual with the physical. I argue that the Real-World Web (RWW) threatens to exacerbate three existing epistemic problems in the digital age: the problem of digital distraction, the problem of digital deception, and the problem of digital divergence. The RWW is poised to present new versions of these problems in the form of what I call the augmented attention economy, augmented skepticism, and the problem of other augmented minds. The paper draws on a range of empirical research on AR and offers a phenomenological analysis of virtual objects as perceptual affordances to help ground and guide the speculative nature of the discussion. It also considers a few policy-based and designed-based proposals to mitigate the epistemic threats posed by AR technology. (shrink)

The study sought to empirically examine the opposing encounter of the social media and suggests regulatory initiatives that will arrest the dilemmas. Eighteen academics and professionals were thoroughly interviewed. Thematic analysis was employed for the data analysis. It was found out that although there are great roles played by the online social media in sectors including social and community relations, business, education and politics, there are also several forms of abuses of the social media. The unprofessional behaviours of some users (...) of the online social media and poor media ethics need urgent management. Implications for moral and ethical, educational and business policy have also been discussed. The paper suggests ways to regulate the abuses of the social media, including censorship of information and controlled measures against those who abuse the use, and also exposes users to the benefits associated with the social media. (shrink)

It is widely accepted that there is a visual field, but the analogous notion of an auditory field is rejected by many philosophers on the grounds that the metaphysics or phenomenology of audition lack the necessary spatial or phenomenological structure. In this paper, I argue that many of the common objections to the existence of an auditory field are misguided and that, contrary to a tradition of philosophical scepticism about the spatiality of auditory experience, it is as (...) richly spatial as visual experience—and in some ways even more so. By carefully considering the spatiality and boundedness of audition, along with how sounds or their sources are experienced as occurring within the surrounding acoustic environment, we can gain a better understanding of (i) our auditory experience of space and (ii) the conditions for the existence of spatial sensory fields in general in a way that does not privilege vision over the other senses. (shrink)

Previous studies of auditory expectation have focused on the expectedness perceived by listeners retrospectively in response to events. In contrast, this research examines predictive uncertainty —a property of listeners' prospective state of expectation prior to the onset of an event. We examine the information-theoretic concept of Shannon entropy as a model of predictive uncertainty in music cognition. This is motivated by the Statistical Learning Hypothesis, which proposes that schematic expectations reflect probabilistic relationships between sensory events learned implicitly through exposure. (...) Using probability estimates from an unsupervised, variable-order Markov model, 12 melodic contexts high in entropy and 12 melodic contexts low in entropy were selected from two musical repertoires differing in structural complexity (simple and complex). Musicians and non-musicians listened to the stimuli and provided explicit judgments of perceived uncertainty (explicit uncertainty). We also examined an indirect measure of uncertainty computed as the entropy of expectedness distributions obtained using a classical probe-tone paradigm where listeners rated the perceived expectedness of the final note in a melodic sequence (inferred uncertainty). Finally, we simulate listeners' perception of expectedness and uncertainty using computational models of auditory expectation. A detailed model comparison indicates which model parameters maximize fit to the data and how they compare to existing models in the literature. The results show that listeners experience greater uncertainty in high-entropy musical contexts than low-entropy contexts. This effect is particularly apparent for inferred uncertainty and is stronger in musicians than non-musicians. Consistent with the Statistical Learning Hypothesis, the results suggest that increased domain-relevant training is associated with an increasingly accurate cognitive model of probabilistic structure in music. (shrink)

Purpose– Investments in the technologies of borders and their securitisation continue to be a focal point for many governments across the globe. This paper is concerned with a particular example of such technologies, namely, “Big Data” analytics. In the past two years, the technology of Big Data has gained a remarkable popularity within a variety of sectors, ranging from business and government to scientific and research fields. While Big Data techniques are often extolled as the next frontier for innovation and (...) productivity, they are also raising many ethical and political issues. The aim of this paper is to consider some of these issues and provide a critical reflection on the implications of using Big Data for the governance of borders.Design/methodology/approach– The author draws on the example of the new Big Data solution recently developed by IBM for the Australian Customs and Border Protection Service. The system, which relies on data collected from Passenger Name Records, aims to facilitate and automate mechanisms of profiling enable the identification of “high-risk” travellers. It is argued that the use of such Big Data techniques risks augmenting the function and intensity of borders.Findings– The main concerns addressed here revolve around three key elements, namely, the problem of categorisation, the projective and predictive nature of Big Data techniques and their approach to the future and the implications of Big Data on understandings and practices of identity.Originality/value– By exploring these issues, the paper aims to contribute to the debates on the impact of information and communications technology-based surveillance in border management. (shrink)

Fractal structure is a ubiquitous property found in nature and biology, and has been observed in processes at different levels of organization, including rhythmic behavior and musical structure. A temporal process is characterized as fractal when serial long-term correlations and statistical self-similarity (scaling) are present. Previous studies of sensorimotor synchronization using isochronous (non-fractal) stimuli show that participants' errors exhibit persistent structure (positive long-term correlations), while their inter-tap intervals (ITIs) exhibit anti-persistent structure (negative long-term correlations). Auditory-motor synchronization has not been (...) investigated with anti-persistent stimuli. In the current study, we systematically investigated whether the fractal structure of auditory rhythms was reflected in the responses of participants who were asked to coordinate their taps with each event. We asked musicians and non-musicians to tap with 12 different rhythms that ranged from anti-persistent to persistent. The scaling exponents of the ITIs were strongly correlated with the scaling exponents of the stimuli, showing that the long-term structure of the participants' taps scaled with the long-term structure of the stimuli. Surprisingly, the performance of the musicians was not significantly better than that of the non-musicians. Our results imply that humans are able to readily adapt (rather than simply react) to the overall statistical structure of temporally fluctuating stimuli, regardless of musical skill. (shrink)

The sense of agency is the perception of willfully causing something to happen. Wegner and Wheatley proposed three prerequisites for SA: temporal contiguity between an action and its effect, congruence between predicted and observed effects, and exclusivity . We investigated how temporal contiguity, congruence, and the order of two human agents’ actions influenced SA on a task where participants rated feelings of self-agency for producing a tone. SA decreased when tone onsets were delayed, supporting contiguity as important, but the order (...) of the agents’ actions also mattered. Relative contiguity was the main determinant of SA, as delayed tones were usually attributed to the most recent action. This was unaffected by contingencies between the two actors’ actions , showing that contiguity has a powerful influence on SA, even during joint action in the presence of other cues. (shrink)

This paper addresses the question whether we can conceive of music cognition in ecosemiotic terms. It claims that music knowledge must be generated as a tool for adaptation to the sonic world and calls forth a shift from a structural description of music as an artifact to a process-like approach to dealing with music. As listeners, we are observers who construct and organize our knowledge and bring with us our observational tools. What matters is not merely the sonic world in (...) its objective qualities, but the world as perceived. In order to make these claims operational we can rely on the ecological concept of coping with the sonic world and the cybernetic concepts of artificial and adaptive devices. Listeners, on this view, are able to change their semantic relations with the sonic world through functional adaptations at the level of sensing, acting and coordinating between action and perception. This allows us to understand music in functional terms of what it affords to us and not merely in terms of its acoustic qualities. There are, however, degrees of freedom and constraints which shape the semiotization of the sonic world. As such we must consider the role of event perception and cognitive economy: listeners do not perceive the acoustical environment in terms of phenomenological descriptions but as ecological events. (shrink)

Background. Attending to goal-relevant information can leave us metaphorically ‘blind’ or ‘deaf’ to the next relevant information while searching among distracters. This temporal cost lasting for about a half a second on the human selective attention has been long explored using the attentional blink paradigm. Although there is evidence that certain visual stimuli relating to one’s area of expertise can be less susceptible to attentional blink effects, it remains unexplored whether the dynamics of temporal selective attention vary with expertise and (...) objects types in the auditory modality. Methods. Using the auditory version of the attentional blink paradigm, the present study investigates whether certain auditory objects relating to musical and perceptual expertise could have an impact on the transient costs of selective attention. In this study, expert cellists and novice participants were asked to first identify a target sound, and then to detect instrumental timbres of cello or organ, or human voice as a second target in a rapid auditory stream. Results. The results showed moderate evidence against the attentional blink effect for voices independent of participants’ musical expertise. Experts outperformed novices in their overall accuracy levels of target identification and detection, reflecting a clear benefit of musical expertise. Importanly, the musicianship advantage disappeared when the human voices served as the second target in the stream. Discussion. The results are discussed in terms of stimulus salience, the advantage of voice processing, as well as perceptual and musical expertise in relation to attention and working memory performances. (shrink)

Infant and adult learners are able to identify word boundaries in fluent speech using statistical information. Similarly, learners are able to use statistical information to identify word–object associations. Successful language learning requires both feats. In this series of experiments, we presented adults and infants with audio–visual input from which it was possible to identify both word boundaries and word–object relations. Adult learners were able to identify both kinds of statistical relations from the same input. Moreover, their learning was actually facilitated (...) by the presence of two simultaneously present relations. Eight-month-old infants, however, do not appear to benefit from the presence of regular relations between words and objects. Adults, like 8-month-olds, did not benefit from regular audio–visual correspondences when they were tested with tones, rather than linguistic input. These differences in learning outcomes across age and input suggest that both developmental and stimulus-based constraints affect statistical learning. (shrink)

Multisensory integration allows us to link sensory cues from multiple sources and plays a crucial role in speech development. However, it is not clear whether humans have an innate ability or whether repeated sensory input while the brain is maturing leads to efficient integration of sensory information in speech. We investigated the integration of auditory and somatosensory information in speech processing in a bimodal perceptual task in 15 young adults (age 19 to 30) and 14 children (age 5 to (...) 6). The participants were asked to identify if the perceived target was the sound /e/ or /ø/. Half of the stimuli were presented under a unimodal condition with only auditory input. The other stimuli were presented under a bimodal condition with both auditory input and somatosensory input consisting of facial skin stretches provided by a robotic device, which mimics the articulation of the vowel /e/. The results indicate that the effect of somatosensory information on sound categorization was larger in adults than in children. This suggests that integration of auditory and somatosensory information evolves throughout the course of development. (shrink)

While much has been written about whether visual perception is cognitively penetrable, the analogous question with respect to auditory perception has received very little attention. Here we argue that instances of top-down modulation of auditory processing, although extensive, do not constitute cases of cognitive penetration of auditory perception since the changes in the phenomenology of auditory perception caused by top-down influences cannot plausibly be attributed to the listeners’ discursive thoughts.

It is a commonly held view that auditory perception functions to tell us about sounds and their properties. In this paper I argue that this common view is mistaken and that auditory perception functions to tell us about the objects that are the sources of sounds. In doing so, I provide a general theory of auditory perception and use it to give an account of the content of auditory experience and of the nature of sounds.