Natural fluctuations in sustained attention can lead to attentional failures in everyday tasks and even dangerous incidences. These fluctuations depend on personal factors, as well as task characteristics. So far, our understanding of sustained attention is partly due to the common usage of laboratory setups and tasks, and the complex interplay between behavior and brain activity. The focus of the current study was thus to test the feasibility of applying a single-channel wireless EEG to monitor patterns of sustained attention during (...) a set of ecological tasks. An EEG marker of attention (BEI—Brain Engagement Index) was continuously recorded from 42 healthy volunteers during auditory and visual tasks from the Test of Everyday Attention (TEA) and Trail Making Test (TMT). We found a descending pattern of both performance and BEI in the auditory tasks as task complexity increases, while the increase in performance and decrease in BEI on the visual task. In addition, patterns of BEI in the complex tasks were used to detect outliers and the optimal range of attention through exploratory models. The current study supports the feasibility of combined electrophysiological and neurocognitive investigation of sustained attention in ecological tasks yielding unique insights on patterns of sustained attention as a function of task modality and task complexity. (shrink)

RESUMO O objetivo do artigo é aperfeiçoar heurísticas para o discurso das notícias por meio do desenvolvimento de um modelo cognitivo de abstração, em contextos de mídia social. Para tal, discute-se a notícia como gênero, estrutura e metalinguagem, partindo da noção formal de modo semiótico. A anotação sob essa visada é uma tecnologia bem-sucedida para controlar os efeitos das restrições de gênero, revelar relações e inquirir sobre dados e documentos. Procede-se à anotação da notícia na interface do Twitter, caracterizada em (...) termos de suas regularidades formais e materiais empregadas recursivamente. Demonstra-se a formalização de uma semântica para o discurso da notícia nas chamadas formas lógicas, adaptada de ferramentas desenvolvidas pela Segmented Discourse Representation Theory (SDRT), a fim de construir novas hipóteses sobre coerência em diferentes níveis de detalhe. Por fim, discute-se a implicação desta abordagem como disciplina na produção de conhecimento transversal com vistas à literacia digital, em tempos urgentes1. (shrink)

This article, taking a social semiotic approach, analyses two pieces of music written, shared and exalted by two pre-1945 European fascist movements – the German NSDAP and the British Union of Fascists. These movements, both political and cultural, employed mythologies of unity, common identity and purpose in order to elide the realities of social distinction and political–economic inequalities between bourgeois and proletarian groups in capitalist societies. Visually and inter-personally, the fascist cultural project communicated a machine-like certainty about a vision for (...) a new society based on discipline, conformity and the might of the nation. In this article, we are interested in the ways that these very same discourses are also communicated through sound and music in two songs: The Horst Wessel Lied and the BUF marching song, two songs that used the same melody. We analyse the discourses communicated by the semiotic choices made in melody, arrangements, sound qualities, rhythms as well as in lyrics. The article first identifies some of the underlying semiotic resources for meaning making in sound and then shows how these are used in order to communicate specific ideas, values and attitudes. (shrink)

Open peer commentary on the article “A Cybernetic Approach to Contextual Teaching and Learning” by Philip Baron. Upshot: Universities in South Africa and elsewhere are being challenged to prepare graduates for the world of work and to recognize the multitude of resources that students bring with them to the higher-education experience. A multimodal approach to pedagogy may assist in both of these regards.

Millions of memes are created and shared every day on social media platforms. Memes are a great tool to spread humour. However, some people use it to target an individual or a group generating offensive content in a polite and sarcastic way. Lack of moderation of such memes spreads hatred and can lead to depression like psychological conditions. Many successful studies related to analysis of language such as sentiment analysis and analysis of images such as image classification have been performed. (...) However, most of these studies rely only upon either one of these components. As classifying meme is one problem which cannot be solved by relying upon only any one of these aspects, the present work identifies, addresses, and ensembles both the aspects for analyzing such data. In this research, we propose a solution to the problems in which the classification depends on more than one model. This paper proposes two different approaches to solve the problem of identifying hate memes. The first approach uses sentiment analysis based on image captioning and text written on the meme. The second approach is to combine features from different modalities. These approaches utilize a combination of glove, encoder-decoder, and OCR with Adamax optimizer deep learning algorithms. Facebook Challenge Hateful Meme Dataset is utilized which contains approximately 8500 meme images. Both the approaches are implemented on the live challenge competition by Facebook and predicted quite acceptable results. Both approaches are tested on the validation dataset, and results are found to be promising for both models. (shrink)

Defining technology : technology as apparatus -- Multimodal critical discourse analysis -- Analysing multimodal discourse : a toolkit approach -- Discourses of technology as progress -- Discourses of technological determinism -- Discourses of technological fetishism : (over)valuing technologies -- Discourses of technological (dis)satisfaction : consuming technologies.

Frame Semantics includes context as a central aspect of the theory. Frames themselves can be regarded as a representation of the immediate context against which meaning is to be construed. Moreover, the notion of frame invocation includes context as one possible source of information comprehenders use to construe meaning. As the original implementation of Frame Semantics, Berkeley FrameNet is capable of providing computational representations of some aspects of context, but not all of them. In this article, we present FrameNet Brasil: (...) a framenet enriched with qualia relations and capable of taking other semiotic modes as input data, namely pictures and videos. We claim that such an enriched model is capable of addressing other types of contextual information in a framenet, namely sentence-level cotext and commonsense knowledge. We demonstrate how the FrameNet Brasil software infrastructure addresses contextual information in both database construction and corpora annotation. We present the guidelines for the construction of two multimodal datasets whose annotations represent contextual information and also report on two experiments: the identification of frame-evoking lexical units in sentences and a methodology for domain adaptation in Neural Machine Translation that leverages frames and qualia for representing sentence-level context. Experimental results emphasize the importance of computationally representing contextual information in a principled structured fashion as opposed to trying to derive it from the manipulation of linguistic form alone. (shrink)

Interviewing offers the biographer unique opportunities for gathering data. I offer three examples. The emphatic bacterial geneticist Norton Zinder confronted me with an interpretation of Barbara McClintock's science that was as surprising as it proved to be robust. The relaxed setting of the human geneticist Walter Nance's rural summer home contributed to an unusually improvisational oral history that produced insights into his experimental and thinking style. And "embedding" myself with the biochemical geneticist Charles Scriver in his home, workplace, and city (...) enabled me to experience the social networks that drive the practical events of his career, which in turn helped me explain the theoretical basis of his science. Face-to-face interaction and multisensory experience will shape each biographer's experience uniquely. Recent developments in sensory physiology suggest that the experience of integrating sense data encourages different patterns of observation and reflection. It is reasonable, then, to think that biography based on face-to-face interviews will, for a given author, have a different character than one based entirely on documents. I reflect on how interviewing shapes my own writing and I encourage the reader to do the same. (shrink)

Most native speakers of Spanish are intuitively able to construct correct structures with the marker ‘se’. On the other hand, non-native speakers, even those at advanced proficiency levels, have difficulties producing most constructions with ‘se’. This is hardly surprising as the marker ‘se’, one of the most common words in Spanish, can convey highly pragmatic nuances with a variety of functions that are still much debated among linguists. This study analyses some of the most used functions of the marker in (...) the oral production of 18 Peninsular Spanish speakers from a multimodal Cognitive Grammar approach, identifying how gestures might be employed by speakers to create or clarify meaning. Our results confirm that gestures are used differently depending on the function of the marker. Middle voices where ‘se’ marks high levels of subject involvement and/or energy co-occur with related gestures, while middle voices or impersonal structures, where the involvement and/or energy is low, co-occur more often with unrelated gestures, if any. (shrink)

Name der Zeitschrift: Semiotica - Journal of the International Association for Semiotic Studies / Revue de l'Association Internationale de Sémiotique Jahrgang: 2013 Heft: 197 Seiten: 79-100.

This paper proposes a methodological synthesis in order to study multimodal media discourse and argumentation in the context of the so-called ‘refugee crisis’ in Greece. It follows the framework of Multimodal Critical Discourse Analysis, integrating this with argumentation studies, with a particular emphasis on the analysis of inference. Our data come from the Greek newspapers Kathimerini and Ta Nea. We contend that the proposed methodological synergy enables scrutiny of (a) racist conceptualizations cultivated by the representation of migrants and (...) refugees in headlines and photographs in newspaper articles along with (b) the argumentative potential that is implicitly sustained in these multimodal representations. (shrink)

In the past few decades, intensive and extensive social science studies have attempted to determine how textbooks provide society- and social reality-related information. This research type gained...

Adult-infant early dyadic interactions have been extensively explored by developmental psychologists. Around the age of 2 months, infants already demonstrate complex, delicate and very sensitive behaviors that seem to express their ability to interact and share emotions with their caregivers. This paper presents 3 pilot studies of parent-infant dyadic interaction in various set-ups. The first two present longitudinal data collected on two infants aged between 1 and 6 months and their mothers. We analyzed the development of coordination between them, at (...) the motor and at the vocal level. The 3rd pilot study aims to explore interpersonal coordination in both vocal behavior and motor activity for one infant and his mother at 2, 4 and 6 months. These pilot studies however leave a number of questions open concerning developmental changes and infants’ progressive mastery of interaction. We identify areas worth examining and try to tease out specific issues that may help develop new methodological pathways for the study of early naturalistic social interaction. We assume that a continuous, rather than discrete, approach would better capture the changes taking place in the various communicative modalities, while also displaying each dyad’s specificity and the narrative dimension of social engagement between infants and caregivers. (shrink)

Taste is a central sense for humans and animals, and it has been largely studied either from physiological and neurological approaches or from socio-cultural ones. This paper adopts another view, focused on the activity of tasting rather than on the sense of taste, approached within the perspective of ethnomethodology and multimodal conversation analysis. This view addresses the activity of tasting as it is interactionally organized in specific social settings, observed in a naturalistic way, on the basis of video (...) recordings. Focusing on video recorded improvised tastings of cheese in gourmet shop encounters, the paper offers a systematic analysis of the way in which tasting is orderly achieved in an intersubjective way. It follows the various steps characterizing tasting, from the invitation to taste, to the grasping of a bit to taste, which is put in the mouth, chewed, and swallowed; it details how an interactional moment offering the taster a priviledged, individual, focused space in which to devote exclusive attention to the object tasted is actively tailored by all parties. By contrast, the completion of tasting is marked by a return to mutual gaze, the animation of facial expressions and nods, and the final production of a judgment of taste. By offering a systematic reconstruction of how these tasting moments are organized, the paper invites to a multimodal approach of sensoriality in social interaction. (shrink)

In contrast to symbol-manipulation approaches, Cognitive Linguistics offers a modal rather than an amodal account of meaning in language. From this perspective, the meanings attached to linguistic expressions, in the form of conceptualisations, have various properties in common with visual forms of representation. This makes Cognitive Linguistics a potentially useful framework for identifying and analysing language-image relations in multimodal texts. In this paper, we investigate language-image relations with a specific focus on intersemiotic convergence. Analogous with research on gesture, (...) we extend the notion of co-text images and argue that images and language usages which are proximal to one another in a multimodal text can be expected to exhibit the same or consistent construals of the target scene. We outline some of the dimensions of conceptualisation along which intersemiotic convergence may be enacted in texts, including event-structure, viewpoint, distribution of attention and metaphor. We take as illustrative data photographs and their captions in online news texts covering a range of topics including immigration, political protests, and inter-state conflict. Our analysis suggests the utility of Cognitive Linguistics in allowing new potential sites of intersemiotic convergence to be identified and in proffering an account of language-image relations that is based in language cognition. (shrink)

Recently, it has been observed that the usual type-theoretic restrictions are not enough to block certain paradoxes involving two or more predicates. In particular, when we have a self-referential language containing modal predicates, new paradoxes might appear even if there are type restrictions for the principles governing those predicates. In this article we consider two type-theoretic solutions to multimodal paradoxes. The first one adds types for each of the modal predicates. We argue that there are a number of problems (...) with most versions of this approach. The second one, which we favour, represents modal notions by using the truth predicate together with the corresponding modal operator. This way of doing things is not only useful because it avoids multimodal paradoxes, but also because it preserves the expressive capacity of the language. As an example of the sort of theory we have in mind, we provide a type-theoretic axiomatization that combines truth with necessity and knowledge. (shrink)

Philosophers and cognitive scientists of perception by custom have investigated individual sense modalities in relative isolation from each other. However, perceiving is, in a number of respects, multimodal. The traditional sense modalities should not be treated as explanatorily independent. Attention to the multimodal aspects of perception challenges common assumptions about the content and phenomenology of perception, and about the individuation and psychological nature of sense modalities. Multimodal perception thus presents a valuable opportunity for a case study in (...) mature interdisciplinary cognitive science. This chapter aims to raise these issues against the background of unimodal approaches in the study of perception. It presents some of the central empirical findings concerning multimodality, and it explains the philosophical implications of these findings. Foremost, it aims to encourage and open avenues for future research. (shrink)

Name der Zeitschrift: Semiotica Jahrgang: 2018 Heft: 220 Seiten: 155-172.

Magnetic resonance imaging (MRI) can provide a number of measurements relevant to sport-related concussion (SRC) symptoms; however, most studies to date have used a single MRI modality and whole-brain exploratory analyses in attempts to localize concussion injury. This has resulted in highly variable findings across studies due to wide ranging symptomology, severity and nature of injury within studies. A multimodal MRI, symptom-guided region-of-interest (ROI) approach is likely to yield more consistent results. The functions of the cerebellum and basal ganglia (...) transcend many common concussion symptoms, and thus these regions, plus the white matter tracts that connect or project from them, constitute plausible ROIs for MRI analysis. We performed diffusion tensor imaging (DTI), resting-state functional MRI, quantitative susceptibility mapping (QSM), and cerebral blood flow (CBF) imaging using arterial spin labeling (ASL), in youth aged 12-18 years following SRC, with a focus on the cerebellum, basal ganglia and white matter tracts. Compared to controls similar in age, sex and sport (N = 20), recent SRC youth (N = 29; MRI at 8 ± 3 days post injury) exhibited increased susceptibility in the cerebellum (p = 0.032), decreased functional connectivity between the caudate and each of the pallidum (p = 0.035) and thalamus (p = 0.021), and decreased diffusivity in the mid-posterior corpus callosum (p < 0.038); no changes were observed in recovered asymptomatic youth (N = 16; 41 ± 16 days post injury). For recent symptomatic-only SRC youth (N = 24), symptom severity was associated with increased susceptibility in the superior cerebellar peduncles (p = 0.011) and reduced activity in the cerebellum (p = 0.013). Fewer days between injury and MRI were associated with reduced cerebellar-parietal functional connectivity (p < 0.014), reduced activity of the pallidum (p = 0.002), increased CBF in the caudate (p = 0.005), and reduced diffusivity in the central corpus callosum (p < 0.05). Youth SRC is associated with acute cerebellar inflammation accompanied by reduced cerebellar activity and cerebellar-parietal connectivity, as well as structural changes of the middle regions of the corpus callosum accompanied by functional changes of the caudate, all of which resolve with recovery. Early MRI post-injury is important to establish objective MRI-based indicators for concussion diagnosis, recovery assessment and prediction of outcome. (shrink)

One of the main limitations of natural language‐based approaches to meaning is that they do not incorporate multimodal representations the way humans do. In this study, we evaluate how well different kinds of models account for people's representations of both concrete and abstract concepts. The models we compare include unimodal distributional linguistic models as well as multimodal models which combine linguistic with perceptual or affective information. There are two types of linguistic models: those based on text corpora (...) and those derived from word association data. We present two new studies and a reanalysis of a series of previous studies. The studies demonstrate that both visual and affective multimodal models better capture behavior that reflects human representations than unimodal linguistic models. The size of the multimodal advantage depends on the nature of semantic representations involved, and it is especially pronounced for basic‐level concepts that belong to the same superordinate category. Additional visual and affective features improve the accuracy of linguistic models based on text corpora more than those based on word associations; this suggests systematic qualitative differences between what information is encoded in natural language versus what information is reflected in word associations. Altogether, our work presents new evidence that multimodal information is important for capturing both abstract and concrete words and that fully representing word meaning requires more than purely linguistic information. Implications for both embodied and distributional views of semantic representation are discussed. (shrink)

In academic oral genres such as conference presentations, speakers resort to more than words to convey meaning. Research also suggests that persuasion, an important element of the communicative purpose of conference presentations, is frequently achieved through a combination of semiotic modes. Therefore, a skilful orchestration of these modes can be considered key to achieving effective communication in this genre. However, our understanding of persuasion has often focused on specific elements of the message considered in isolation and mainly from the linguistic (...) perspective. Relatively little attention has been paid to the overall persuasive effect achieved by the complex multimodal ensemble. This study approaches the analysis of persuasive strategies in conference presentations combining multimodal discourse analysis and ethnographic methods. It focuses on a particular attention-getting technique: enactment of characters, or acting the part of a person that is being referred to. Our analysis shows how it is achieved through the orchestration of different modes such as words, intonation, gestures, head movements, gaze and facial expression. (shrink)

This book positions itself at the intersection of the key areas of the modern humanities. Different authors from a variety of countries take innovative approaches to investigating multimodal communication, adapting pedagogical design to digital environments and enhancing cognitive skills through transformations in teaching and learning practices. The eclectic forms under study require eclectic approaches and methodologies, and the authors cross disciplinary boundaries drawing on philosophy, linguistics, semiotics, computational linguistics, mathematics, cognitive studies and neuroaesthetics. Part I presents methods (...) of analysing multimodal communication in its different displays, covering promotional video in crowdfunding project presentations, multimodal public signs of prohibition and visuals as arguments. Part II explores varied teaching methodologies that have emerged as a result of and in response to modern technological changes and contains some practical hints for educators. It demonstrates the pedagogical potential of video games, virtual worlds, linguistic corpora and online dictionaries. Part III focuses on psychological and cognitive factors influencing success in the classroom, primarily, ways of developing students’ and teachers’ personalities. The volume sits at the intersection between Communication Studies, Digital Humanities, Discourse Analysis, Education Theory and Cognitive Studies and is useful to scholars and students of communication, languages, education and other areas of the humanities. This book should trigger scholarly discussions as well as stimulating practitioners’ interest in these fields. (shrink)

A growing number of children in the United States are exposed to multiple languages at home from birth. However, relatively little is known about the early process of word learning—how words are mapped to the referent in their child-centered learning experiences. The present study defined parental input operationally as the integrated and multimodal learning experiences as an infant engages with his/her parent in an interactive play session with objects. By using a head-mounted eye tracking device, we recorded visual scenes (...) from the infant’s point of view, along with the parent’s social input with respect to gaze, labeling, and actions of object handling. Fifty-one infants and toddlers from an English monolingual or a diverse bilingual household were recruited to observe the early multimodal learning experiences in an object play session. Despite that monolingual parents spoke more and labeled more frequently relative to bilingual parents, infants from both language groups benefit from a comparable amount of socially coordinated experiences where parents name the object while the object is looked at by the infant. Also, a sequential path analysis reveals multiple social coordinated pathways that facilitate infant object looking. Specifically, young children’s attention to the referent objects is directly influenced by parent’s object handling. These findings point to the new approach to early language input and how multimodal learning experiences are coordinated socially for young children growing up with monolingual and bilingual learning contexts. (shrink)

This book presents the methodological framework of combining Multimodal Conversation Analysis (MCA) with Interpretative Phenomenological Analysis (IPA) to interpretively analyse translanguaging practices in educational contexts. Beginning with an overview of the three uses of translanguaging - translanguaging as a theory of language, as a pedagogical practice and as an analytical perspective - the book goes on to critically examine the different methodological approaches for analysing translanguaging practices in multilingual classroom interactions. It explains how MCA and IPA are useful (...) methodologies for understanding how and why translanguaging practices are constructed by participants in the classroom and discusses types of data collected and data collection procedures. The author, Kevin W. H. Tai, shows how combining these approaches enables researchers to study how translanguaging practices are constructed in multilingual classrooms and how teachers make sense of their own translanguaging practices at particular moments of classroom interaction. This detailed and concise guide is indispensable for students, practitioners, policymakers, and researchers from across the globe, particularly those working in the fields of applied linguistics and language education. (shrink)

This book envisions a relational and multimodal conceptualisation of higher education, which advocates for knowledge to be analyzed under three relational dimensions: social, digital and environmental. The volume draws on interdisciplinary approaches grounded in Peirceian semiotics, in exploring an integration of these dimensions in higher education theory and practice. The book situates learning in an awareness of the environment, grounded in principles of interconnectedness and flattened hierarchies. The volume features practical case studies through dialogues with higher education teachers, (...) presenting different ways of practicing multimodal and relational higher education which prioritize and embrace expression and creation through more than one dominant modality of communication. The book highlights how a relational multimodal paradigm can serve as a way forward for universities to address global challenges concerning social, technological, and environmental futures through a social justice lens. This innovative book will be of interest to students and scholars in semiotics, multimodality, and higher education, as well as stakeholders in these fields. (shrink)

Sensory Individuals: Unimodal and Multimodal Perspectives provides an interdisciplinary, well-balanced, and comprehensive look at different aspects of unisensory and multisensory objects, using both nuanced philosophical analysis and informed empirical work. The research presented in this book represents the field's progression from treating neural sensory processes as primarily modality-specific towards its current state of the art, according to which perception, and its supporting neural processes, are multi-modal, modality-independent, meta-modal, and task-dependent. Even within such approaches sensory stimuli, properties, brain activations, (...) and corresponding perceptual phenomenology can still be characterized in a modality-specific way. The book examines the basic building blocks of human perception, and whether they are best understood as sense modality dependent units of different forms or multimodal perceptual objects. The book combines a variety of innovative and integrative angles to explore the topic and acts as a catalyst for an increasingly diverse field of research, which is in an exciting phase of growth and advancement. New questions are arising as quickly as they are being answered, and the collection Sensory Individuals provides an original and up-to-date addition to the field. (shrink)

Ledin and Machin's critique of the use of some current approaches to multimodality for the purposes of critical discourse studies raises some important methodological concerns that need to be addressed. However, both the particular position they develop as well as some of the key points they raise are themselves problematic. In this response, I argue that Ledin and Machin misconstrue some significant aspects of the body of theory they critique and, as a consequence, offer a potentially misleading view of (...) the relevant state of the art in multimodality. I suggest that pursuing closer alignments with theoretically more nuanced accounts of multimodality would allow the account that Ledin and Machin propose to be revised in ways better suited to the task of conducting empirically-driven multimodal critical discourse studies. (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are (...) better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

Although multimodal input has the potential to lead to more sound learning outcomes, it carries the risk of causing cognitive overload, making it difficult to determine the exact effects of multimodal input on the second language phrase learning. This study tests the efficacy of multimodal input on L2 phrase learning. It adopts a mixed-method approach by utilizing both quantitative and qualitative data. The experimental design is a 2 × 3 mixed model, with a group [the experimental group (...) and the control group ] as the between-subject factor and time as the within-subject factor. A total of 66 participants were divided into two groups. All materials incorporated three aspects of phrase knowledge, but the materials of the CG were unimodal in that they were offered only on paper, and of the EG were multimodal in that they included pictures, audio recordings, and video clips. After the treatment, a questionnaire and a semi-structured interview were given to the EG learners to explore their perceptions of using multimodal materials to learn L2 phrases. The results indicate that both groups had significant gains in learning phrases, but students with the multimodal input achieved significantly better results than those with the unimodal input. Moreover, the EG students had a generally positive attitude toward the use of multimodal resources. This study validates the efficacy of multimodal input on the acquisition of English phrases and shows that cognitive overload was avoided by sequencing the information. (shrink)

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are (...) better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field. (shrink)

Background: Maximum safe resection of infiltrative brain tumors in eloquent area is the primary objective in surgical neuro-oncology. This goal can be achieved with direct electrical stimulation to perform a functional mapping of the brain in patients awake intraoperatively. When awake surgery is not possible, we propose a pipeline procedure that combines advanced techniques aiming at performing a dissection that respects the anatomo-functional connectivity of the peritumoral region. This procedure can benefit from intraoperative monitoring with computerized tomography scan and brain (...) shift correction. Associated with this intraoperative monitoring, the additional value of preoperative investigation combining brain mapping by navigated transcranial magnetic stimulation with various neuroimaging modalities has not yet been reported.Case Report: A 42-year-old left-handed man had increased intracranial pressure, left hand muscle deficit, and dysarthria, related to an infiltrative tumor of the right frontal lobe with large mass effect and circumscribed contrast enhancement in motor and premotor cortical areas. Spectroscopy profile and intratumoral calcifications on CT-scan suggested an WHO grade III glioma, later confirmed by histology. The aforementioned surgical procedure was considered, since standard awake surgery was not appropriate for this patient. In preoperative time, nTMS mapping of motor function was performed, combined with magnetic resonance imaging -based tractography reconstruction of 6 neural tracts and resting-state functional MRI connectivity of sensorimotor and language networks. In intraoperative time, DES mapping was performed with motor evoked response recording and tumor resection was optimized using non-rigid image transformation of the preoperative data to iCT data. Image guidance was updated with correction for brain shift and tissue deformation using biomechanical modeling taking into account brain elastic properties. This correction was done at crucial surgical steps, i.e., when tumor bulged through the craniotomy after dura mater opening and when approaching the presumed eloquent brain regions. This procedure allowed a total resection of the tumor region with contrast enhancement as well as a complete regression of IICP and dysarthria. Hand paresis remained stable with no additional deficit. Postoperative nTMS mapping confirmed the good functional outcome.Conclusion: This case report and technical note highlights the value of preoperative functional evaluation by nTMS updated intraoperatively with correction of brain deformation by iCT. This multimodal approach may become the optimized technique of reference for patients with brain tumors in eloquent areas that are unsuitable for awake brain surgery. (shrink)

This mini-review provides a detailed outline of studies that have used multimodal approaches in non-invasive brain stimulation to investigate the pathophysiology of the three common movement disorders, namely, essential tremor, Parkinson’s disease, and dystonia. Using specific search terms and filters in the PubMed®database, we finally shortlisted 27 studies in total that were relevant to this review. While two-thirds (Brittain et al., 2013) of these studies were performed on Parkinson’s disease patients, we could find only three studies that were (...) conducted in patients with essential tremor. We clearly show that although multimodal non-invasive brain stimulation holds immense potential in unraveling the physiological mechanisms that are disrupted in movement disorders, the technical challenges and pitfalls of combining these methods may hinder their widespread application by movement disorder specialists. A multidisciplinary team with clinical and technical expertise may be crucial in reaping the fullest benefits from such novel multimodal approaches. (shrink)

This article investigates a multiparty interaction in an accounting office by applying a multimodal approach to discourse. This approach allows the incorporation of all relevant communicative modes and is based on the following three notions: 1) the notion of mediated action; 2) the notion of modal density; and 3) the notion of a foreground– background continuum of attention/awareness. The article illustrates that a social actor in a multiparty interaction simultaneously co-constructs several higher-level actions with the various participants on different (...) levels of their attention/awareness. On a theoretical level, the article argues that traditional approaches to discourse analysis, with their unconditional focus on language as the primary mode, misconstrue the multiple higher-level actions that a social actor is engaged in simultaneously as dyadic or triadic interactions in quick succession. (shrink)

Musical performance is a multimodal experience, for performers and listeners alike. This paper reports on a pilot study which constitutes the first step toward a comprehensive approach to the experience of music as performed. We aim at bridging the gap between qualitative and quantitative approaches, by combining methods for data collection. The purpose is to build a data corpus containing multimodal measures linked to high-level subjective observations. This will allow for a systematic inclusion of the knowledge of (...) music professionals in an analytic framework, which synthesizes methods across established research disciplines. We outline the methods we are currently developing for the creation of a multimodal data corpus dedicated to the analysis and exploration of instrumental music performance from the perspective of embodied music cognition. This will enable the study of the multiple facets of instrumental music performance in great detail, as well as lead to the development of music creation techniques that take advantage of the cross-modal relationships and higher-level qualities emerging from the analysis of this multi-layered, multimodal corpus. The results of the pilot project suggest that qualitative analysis through stimulated recall is an efficient method for generating higher-level understandings of musical performance. Furthermore, the results indicate several directions for further development, regarding observational movement analysis, and computational analysis of coarticulation, chunking, and movement qualities in musical performance. We argue that the development of methods for combining qualitative and quantitative data are required to fully understand expressive musical performance, especially in a broader scenario in which arts, humanities, and science are increasingly entangled. The future work in the project will therefore entail an increasingly multimodal analysis, aiming to become as holistic as is music in performance. (shrink)

In this paper, we propose a person verification scheme using a novel combination of palmprint and speech. The crucial aspect of biometric based verification lies in its use of features in verification. Thus, in this paper, we propose two novel feature extraction methods for palmprint verification. The proposed methods are based on the Gaussian mixture model followed by subspace based approaches such as ICA I and ICA II, called independent component analysis I mixture model and independent component analysis II (...) mixture model for palmprint verification. The speech verification system uses sphericity based measurement for person verification. The proposed multimodal biometric system uses a fairly large database of 150 users. The robustness of the proposed methods is validated by introducing six different types of noises on palmprint images and Gaussian white noise with two different signal-to-noise ratios on speech samples. Then, fusion of palmprint and speech are carried out at match score level using the weighted SUM rule for both clean and noisy conditions. Finally, we compare the results of combined biometric with the results of individual biometric and also the results of the proposed methods against conventional subspace methods. (shrink)

Sensory Individuals: Unimodal and Multimodal Perspectives provides an interdisciplinary, well-balanced, and comprehensive look at different aspects of unisensory and multisensory objects, using both nuanced philosophical analysis and informed empirical work. The research presented in this book represents the field's progression from treating neural sensory processes as primarily modality-specific towards its current state of the art, according to which perception, and its supporting neural processes, are multi-modal, modality-independent, meta-modal, and task-dependent. Even within such approaches sensory stimuli, properties, brain activations, (...) and corresponding perceptual phenomenology can still be characterized in a modality-specific way. The book examines the basic building blocks of human perception, and whether they are best understood as sense modality dependent units of different forms or multimodal perceptual objects. The book combines a variety of innovative and integrative angles to explore the topic and acts as a catalyst for an increasingly diverse field of research, which is in an exciting phase of growth and advancement. New questions are arising as quickly as they are being answered, and the collection Sensory Individuals provides an original and up-to-date addition to the field. (shrink)

Place-based policies tackle grand socio-economic challenges through differentiated, context-sensitive interventions. However, they often run the risk of under- or mis-performing. This work studies how grand challenges translate from policy to cross-sector partnerships through place. By focusing on the place-based policy of regional smart specialization (RIS3), I investigate how the setup of science and technology parks mediates the practices of the actors in the translation chain: a transnational policymaker (macro), a regional broker (meso), and a local partnership which served as prototype (...) for the regional policy (micro level). I document two types of practices—emplacement and space configuration—enacted at each level, and show how their interplay transformed the grand challenge from a cautious ideal at the macro level which balances risk and responsibility, to an optimistic and risk-prone approach at the partnership level. The study contributes to the policy and cross-sector partnerships literatures by documenting a two-sided effect of place-based policy and a consequent risk of ethical reversal, from an early attractor bringing partners together to a later accomplice keeping the partners together despite evident signs of failure. By adopting a strong multimodal approach, I also distinguish four types of multimodal outcomes—ideal type, prototype, virtual model, and lived artifact—which perform the two-sided effect and bring about the risk of reversal. Practical implications include a redistribution of the burden of failure from the CSPs implementing the grand challenges to the institutional fields in which these are bred. (shrink)