Abstract
Humans communicate with different modalities. We offer an account of
multi-modal meaning coordination, taking speech-gesture meaning coordination
as a prototypical case. We argue that temporal synchrony (plus prosody) does not
determine how to coordinate speech meaning and gesture meaning. Challenging
cases are asynchrony and broadcasting cases, which are illustrated with empirical
data. We propose that a process algebra account satisfies the desiderata. It models
gesture and speech as independent but concurrent processes that can communicate
flexibly with each other and exchange the same information more than once. The account utilizes the ψ-calculus, allowing for agents, input-output-channels, concurrent
processes, and data transport of typed λ-terms. A multi-modal meaning is produced
integrating speech meaning and gesture meaning into one semantic package. Two
cases of meaning coordination are handled in some detail: the asynchrony between
gesture and speech, and the broadcasting of gesture meaning across several dialogue contributions. This account can be generalized to other cases of multi-modal
meaning.