In this paper, I review the main criteria offered for distinguishing the modal and amodal approaches to conceptual format: the type of input to which the representations respond, the relation they bear to perceptual states, and the specific neural systems to which they belong. I evaluate different interpretations of them and argue that they all face difficulties. I further show that they lead to cross-classifications of certain types of representations, using approximate number representations as an example.

There is a widespread intuition that readers’ empathy with fictional characters can help to understand certain works of literature. In a nutshell, the argument.

Previous studies show that reading sentences about actions leads to specific motor activity associated with actually performing those actions. We investigate how sign language input may modulate motor activation, using British Sign Language sentences, some of which explicitly encode direction of motion, versus written English, where motion is only implied. We find no evidence of action simulation in BSL comprehension, but we find effects of action simulation in comprehension of written English sentences by deaf native BSL signers. These results provide (...) constraints on the nature of mental simulations involved in comprehending action sentences referring to transfer events, suggesting that the richer contextual information provided by BSL sentences versus written or spoken English may reduce the need for action simulation in comprehension, at least when the event described does not map completely onto the signer's own body. (shrink)

Facial electromyography research shows that corrugator supercilii ('frowning muscle') activity tracks the emotional valence of linguistic stimuli. Grounded or embodied accounts of language processing take such activity to reflect the simulation or ‘reenactment’ of emotion, as part of the retrieval of word meaning (e.g., of “furious”) and/or of building a situation model (e.g., for “Mark is furious”). However, the same muscle also expresses our primary emotional evaluation of things we encounter. Language-driven affective simulation can easily be at odds with the (...) reader’s affective evaluation of what language describes (e.g., when we like Mark being furious). To examine what happens in such cases, we independently manipulated simulation valence and moral evaluative valence in short narratives. Participants first read about characters behaving in a morally laudable or objectionable fashion: this immediately led to corrugator activity reflecting positive or negative affect. Next, and critically, a positive or negative event befell these same characters. Here, the corrugator response did not track the valence of the event, but reflected both simulation and moral evaluation. This highlights the importance of unpacking coarse notions of affective meaning in language processing research into components that reflect simulation and evaluation. Our results also call for a re-evaluation of the interpretation of corrugator EMG, as well as other affect-related facial muscles and other peripheral physiological measures, as unequivocal indicators of simulation. Research should explore how such measures behave in richer and more ecologically valid language processing, such as narrative; refining our understanding of simulation within a framework of grounded language comprehension. (shrink)

Do we mentally simulate olfactory information? We investigated mental simulation of odors and sounds in two experiments. Participants retained a word while they smelled an odor or heard a sound, then rated odor/sound intensity and recalled the word. Later odor/sound recognition was also tested, and pleasantness and familiarity judgments were collected. Word recall was slower when the sound and sound-word mismatched. Sound recognition was higher when sounds were paired with a match or near-match word. This indicates sound-words are mentally simulated. (...) However, using the same paradigm no memory effects were observed for odor. Instead it appears odor-words only affect lexical-semantic representations, demonstrated by higher ratings of odor intensity and pleasantness when an odor was paired with a match or near-match word. These results suggest fundamental differences in how odor and sound-words are represented. (shrink)

In this article, we examine how cognitive metaphor theories might contribute to the theory and practice of poetry translation. We focus on translations from Chinese to English by Xu Yuanchong, both...

Describing actions entails that relations between objects are discovered. A pervasively neural account of this process requires that fundamental problems are solved: the neural pointer problem, the binding problem, and the problem of generating discrete processing steps from time-continuous neural processes. We present a prototypical solution to these problems in a neural dynamic model that comprises dynamic neural fields holding representations close to sensorimotor surfaces as well as dynamic neural nodes holding discrete, language-like representations. Making the connection between these two (...) types of representations enables the model to describe actions as well as to perceptually ground movement phrases—all based on real visual input. We demonstrate how the dynamic neural processes autonomously generate the processing steps required to describe or ground object-oriented actions. By solving the fundamental problems of neural pointing, binding, and emergent discrete processing, the model may be a first but critical step toward a systematic neural processing account of higher cognition. (shrink)

The debate between the amodal and the grounded views of cognition seems to be stuck. Their only substantial disagreement is about the vehicle or format of concepts. Amodal theorists reject the grounded claim that concepts are couched in the same modality-specific format as representations in sensory systems. The problem is that there is no clear characterization of format or its neural correlate. In order to make the disagreement empirically meaningful and move forward in the discussion we need a neurocognitive criterion (...) for representational format. I argue that efficient coding models in computational neuroscience can be used to characterize modal codes: These are codes which satisfy special informational demands imposed by sensory tasks. Additionally, I examine recent studies on neural coding and argue that although they do not provide conclusive evidence for either the grounded or the amodal views, they can be used to determine what predictions these approaches can make and what experimental and theoretical developments would be required to settle the debate. (shrink)

I argue that Duffley’s sign-based semantics and embodied semantics may be mutually beneficial if we conceive them as a semantic theory and as a foundational theory, respectively. First, I describe embodied semantics as a research program that conceives the foundations of meaning in terms of embodied simulation. Afterwards, I draw attention to three points where Duffley’s semantics could find support in such a foundational theory. Finally, I suggest that two pressing challenges currently on the agenda of embodied semantics could be (...) met by Duffley’s theory. (shrink)

In an evolutionary perspective Louwerse elaborates the Symbol Interdependency Hypothesis (Louwerse, 2011), arguing that language has evolved such that it maps onto the perceptual system, allowing to bootstrap meaning also when grounding is limited. The author concludes that in principle the processing of abstract and concrete words is the same and that in both cases language users tend to rely anyway on indexical relationships that words entertain with other words.

Grammatical categories represent implicit knowledge, and it is not known if such abstract linguistic knowledge can be continuously grounded in real-life experiences, nor is it known what types of mental states can be simulated. A former study showed that attention bias in peripersonal space affects reaction times in grammatical congruency judgments of nominal classifiers, suggesting that simulated semantics may include reenactment of attention. In this study, we contrasted a Chinese nominal classifier used with nouns denoting pinch grip objects with a (...) classifier for nouns with big object referents in a pupil dilation experiment. Twenty Chinese native speakers read grammatical and ungrammatical classifier-noun combinations and made grammaticality judgment while their pupillary responses were measured. It was found that their pupils dilated significantly more to the pinch grip classifier than to the big object classifier, indicating attention simulation in PPS. Pupil dilations were also significantly larger with congruent trials on the whole than in incongruent trials, but crucially, congruency and classifier semantics were independent of each other. No such effects were found in controls. (shrink)

Embodied cognitive theories predict that linguistic conceptual representations are grounded and continually represented in real world, sensorimotor experiences. However, there is an on-going debate on whether this also holds for abstract concepts. Grammar is the archetype of abstract knowledge, and therefore constitutes a test case against embodied theories of language representation. Former studies have largely focussed on lexical-level embodied representations. In the present study we take the grounding-by-modality idea a step further by using reaction time (RT) data from the linguistic (...) processing of nominal classifiers in Chinese. We take advantage of an independent body of research, which shows that attention in hand space is biased. Specifically, objects near the hand consistently yield shorter RTs as a function of readiness for action on graspable objects within reaching space, and the same biased attention inhibits attentional disengagement. We predicted that this attention bias would equally apply to the graspable object classifier but not to the big object classifier. Chinese speakers (N = 22) judged grammatical congruency of classifier-noun combinations in two conditions: graspable object classifier and big object classifier. We found that RTs for the graspable object classifier were significantly faster in congruent combinations, and significantly slower in incongruent combinations, than the big object classifier. There was no main effect on grammatical violations, but rather an interaction effect of classifier type. Thus, we demonstrate here grammatical category- specific effects pertaining to the semantic content and by extension the visual and tactile modality of acquisition underlying the acquisition of these categories. We conclude that abstract grammatical categories are subjected to the same mechanisms as general cognitive and neurophysiological processes and may therefore be grounded. (shrink)

An emerging perspective on human cognition and performance sees it as a kind of self-organizing phenomenon involving dynamic coordination across the body, brain and environment. Measuring this coordination faces a major challenge. Time series obtained from such cognitive, behavioral, and physiological coordination are often complicated in terms of non-stationarity and non-linearity, and in terms of continuous vs. categorical scales. Researchers have proposed several analytical tools and frameworks. One method designed to overcome these complexities is recurrence quantification analysis, developed in the (...) study of non-linear dynamics. It has been applied in various domains, including linguistic data or motion data. However, most previous studies have applied recurrence methods individually to categorical or continuous data. To understand how complex coordination works, an integration of these types of behavior is needed. We aimed to integrate these methods to investigate the relationship between language and motion directly. To do so, we added temporal information to categorical data, and applied joint recurrence analysis methods to visualize and quantify speech-motion coordination coupling during a rap performance. We illustrate how new dynamic methods may capture this coordination in a small case-study design on this expert rap performance. We describe a case study suggesting this kind of dynamic analysis holds promise, and end by discussing the theoretical implications of studying complex performances of this kind as a dynamic, coordinated phenomenon. (shrink)

The focus of much sensory language research has been on vocabulary and codability, not how language is used in communication of sensory perceptions. We make a case for discourse-oriented research about sensory language as an alternative to the prevailing vocabulary orientation. To consider the language of sound in authentic textual data, we presented participants with 20 everyday sounds of unknown sources and asked them to describe the sounds in as much detail as possible, as if describing them to someone who (...) could not hear them. We explored how the participants use language to describe these sounds. Do they describe their listening experiences (stressful), sound properties (intermittent beeping), and/or the events that caused the sounds (eating an apple)? The results show that out of these three soundscape elements, events are the most frequent and most indispensable element. We let the results from the study illustrate the need for more discursive data in studies of sensory language and argue that there is no designated language of sound. Our study highlights that in order to account for sensory language use, we need an analytical framework that accommodates discursive language in a non-trivial way beyond stable couplings between individual words and meanings. (shrink)

Embodied cognition represents one of most important theoretical developments in contemporary cognitive science. Many cognitive processes appear to be influenced by body morphology, emotions, and sensorimotor systems. This perspective is supported by an ever increasing collection of empirical studies that fall into two broad classes: one consisting of experiments that implicate action, emotion, and perception systems in seemingly abstract cognitive tasks and the other consisting of experiments that demonstrate the contribution of bodily interaction with the external environment to the performance (...) of such tasks. Now that embodied cognition is fairly well established, the time seems right for assessing its further promise and potential limitations. This research topic aimed to create an interdisciplinary forum for discussing where we go from here. Given that we have good reason to think that the body influences cognition in surprisingly robust ways, the central question is no longer whether or not some cognitive processes are embodied. Other questions have come to the forefront. To what extent are cognitive processes embodied? Are there disembodied processes? Among those that are embodied, how are they embodied? Is there more than one kind of embodiment? Is embodiment a matter of degree? Extending the Research ProgramMany of the contributions to this research topic involve experiments that extend the empirical reach of embodied cognition. For instance, Soliman, Gibson, and Gl... (shrink)

Cognitive Science, Volume 46, Issue 7, July 2022.

Social robots serve people as tutors, caretakers, receptionists, companions, and other social agents. People know that the robots are mechanical artifacts, yet they interact with them as if they were actual agents. How is this possible? The proposal here is that people construe social robots not as social agents per se, but as depictions of social agents. They interpret them much as they interpret ventriloquist dummies, hand puppets, virtual assistants, and other interactive depictions of people and animals. Depictions as a (...) class consist of three physical scenes with part-by-part mappings between them: (a) a base scene (the raw physical artifact), (b) the depiction proper (the artifact construed as a depiction), and (c) the scene depicted (the scene people are to imagine). With social robots, evidence shows people form the same three scenes plus mappings: They perceive the raw machinery of a robot, construe it as a depiction of a character, and, using the depiction as a guide, engage in the pretense that they are interacting with the character depicted. With social robots, people also recognize three classes of agents – the characters depicted, the intended recipients of the depictions (those who view or interact with the robots), and the authorities responsible for the robots (the designers, makers, and owners). Construing social robots as depictions, we argue, accounts for many phenomena not covered by alternative models. (shrink)

An eye-tracking while listening study based on the blank screen paradigm was conducted to investigate the processing of literal and metaphorical verbs of motion. The study was based on two assumpti...

Summary The paper presents the result of a collective reflection inspired by the individual suggestions of 30 researchers working in different research areas. They are all familiar with the Experimental Phenomenology of Perception, and are aware of the importance that this approach might represent nowadays in their specific research field. The picture that emerges from this ‘mosaic’ stimulates us to consider the potential future developments of this approach if we accept that we need to push its borders beyond the traditional (...) aims of the study of perception (as masterfully developed by the historic Italian Maestri of this approach). If we take this broader view, the Experimental Phenomenology of Perception can extend its perimeters from an analysis of strictly perceptual aspects to an analysis of cognitive and metacognitive aspects (such as aesthetic evaluations, the perception of risk, the experience of certainty/uncertainty in a reasoning process, the perception of proximity to/distance from the solution to a problem and meaning-making in language). The cognitive and metacognitive aspects referred to are grounded in and modelled on the perceiver’s experience of a given situation. (shrink)

The meaning of most words in language depends on their context. Understanding how the human brain extracts contextualized meaning, and identifying where in the brain this takes place, remain important scientific challenges. But technological and computational advances in neuroscience and artificial intelligence now provide unprecedented opportunities to study the human brain in action as language is read and understood. Recent contextualized language models seem to be able to capture homonymic meaning variation (“bat”, in a baseball vs. a vampire context), as (...) well as more nuanced differences of meaning—for example, polysemous words such as “book”, which can be interpreted in distinct but related senses (“explain a book”, information, vs. “open a book”, object) whose differences are fine‐grained. We study these subtle differences in lexical meaning along the concrete/abstract dimension, as they are triggered by verb‐noun semantic composition. We analyze functional magnetic resonance imaging (fMRI) activations elicited by Italian verb phrases containing nouns whose interpretation is affected by the verb to different degrees. By using a contextualized language model and human concreteness ratings, we shed light on where in the brain such fine‐grained meaning variation takes place and how it is coded. Our results show that phrase concreteness judgments and the contextualized model can predict BOLD activation associated with semantic composition within the language network. Importantly, representations derived from a complex, nonlinear composition process consistently outperform simpler composition approaches. This is compatible with a holistic view of semantic composition in the brain, where semantic representations are modified by the process of composition itself. When looking at individual brain areas, we find that encoding performance is statistically significant, although with differing patterns of results, suggesting differential involvement, in the posterior superior temporal sulcus, inferior frontal gyrus and anterior temporal lobe, and in motor areas previously associated with processing of concreteness/abstractness. (shrink)

Our ability to deal with abstract concepts is one of the most intriguing faculties of human cognition. Still, we know little about how such concepts are formed, processed, and represented in mind. Current views are presented in their most recent and advanced form in this special issue, and directly compared and discussed in a lively debate, reported at the end of each chapter. The main results are reported in the editors’ introduction.

While language use in general is currently being explored as essentially situated in immediate physical environment, narrative reading is primarily regarded as a means of decoupling one’s consciousness from the environment. In order to offer a more diversified view of narrative reading, the article distinguishes between three different roles the environment can play in the reading experience. Next to the traditional notion that environmental stimuli disrupt attention, the article proposes that they can also serve as a prop for mental imagery (...) and/or a locus of pleasure more generally. The latter two perspectives presuppose a more clear-cut distinction between consciousness and attention than typically assumed in the communication literature. The article concludes with a list of implications for research and practice. (shrink)

A great deal of research has focused on the question of whether or not concepts are embodied as a rule. Supporters of embodiment have pointed to studies that implicate affective and sensorimotor systems in cognitive tasks, while critics of embodiment have offered nonembodied explanations of these results and pointed to studies that implicate amodal systems. Abstract concepts have tended to be viewed as an important test case in this polemical debate. This essay argues that we need to move beyond a (...) pretheoretical notion of abstraction. Against the background of current research and theory, abstract concepts do not pose a single, unified problem for embodied cognition but, instead, three distinct problems: the problem of generalization, the problem of flexibility, and the problem of disembodiment. Identifying these problems provides a conceptual framework for critically evaluating, and perhaps improving upon, recent theoretical proposals. (shrink)