The Visual World Paradigm (VWP) presents listeners with a challenging problem: They must integrate two disparate signals, the spoken language and the visual context, in support of action (e.g., complex movements of the eyes across a scene). We present Impulse Processing, a dynamical systems approach to incremental eye movements in the visual world that suggests a framework for integrating language, vision, and action generally. Our approach assumes that impulses driven by the language and the visual context impinge minutely on a (...) dynamical landscape of attractors corresponding to the potential eye-movement behaviors of the system. We test three unique predictions of our approach in an empirical study in the VWP, and describe an implementation in an artificial neural network. We discuss the Impulse Processing framework in relation to other models of the VWP. (shrink)

Three online mouse cursor-tracking experiments investigated predictive sentence processing at speed. Participants viewed visual arrays with objects like a bike and kite while hearing predictive sentences like, “What the man will ride, which is shown on this page, is the bike,” or non-predictive sentences like, “What the man will spot, which is shown on this page, is the bike.” Based on the selectional restrictions of “ride” (i.e., vs. “spot”), participants made mouse cursor movements to the bike before hearing the noun (...) “bike.” Compellingly, this effect was observed at speech rates of ∼3 (Experiment 1), ∼6 (Experiment 2), and ∼9 (Experiment 3) syllables/s. While prior research suggests striking limits on prediction, these results highlight temporal dynamics that may impact comprehenders’ ability to preactivate information when hearing impressively rapid speech. Implications for theories of sentence processing are discussed. (shrink)

We investigated the influence of globally ungrammatical local syntactic constraints on sentence comprehension, as well as the corresponding activation of global and local representations. In Experiment 1, participants viewed visual scenes with objects like a carousel and motorbike while hearing sentences with noun phrase (NP) or verb phrase (VP) modifiers like “The girl who likes the man (from London/very much) will ride the carousel.” In both cases, “girl” and “ride” predicted carousel as the direct object; however, the locally coherent combination (...) “the man from London will ride…” in NP cases alternatively predicted motorbike. During “ride,” local constraints, although ruled out by the global constraints, influenced prediction as strongly as global constraints: While motorbike was fixated less than carousel in VP cases, it was fixated as much as carousel in NP cases. In Experiment 2, these local constraints likewise slowed reading times. We discuss implications for theories of sentence processing. (shrink)