The Role of Negative Information in Distributional Semantic Learning

Cognitive Science 43 (5):e12730 (2019)
  Copy   BIBTEX


Distributional models of semantics learn word meanings from contextual co‐occurrence patterns across a large sample of natural language. Early models, such as LSA and HAL (Landauer & Dumais, 1997; Lund & Burgess, 1996), counted co‐occurrence events; later models, such as BEAGLE (Jones & Mewhort, 2007), replaced counting co‐occurrences with vector accumulation. All of these models learned from positive information only: Words that occur together within a context become related to each other. A recent class of distributional models, referred to as neural embedding models, are based on a prediction process embedded in the functioning of a neural network: Such models predict words that should surround a target word in a given context (e.g., word2vec; Mikolov, Sutskever, Chen, Corrado, & Dean, 2013). An error signal derived from the prediction is used to update each word's representation via backpropagation. However, another key difference in predictive models is their use of negative information in addition to positive information to develop a semantic representation. The models use negative examples to predict words that should not surround a word in a given context. As before, an error signal derived from the prediction prompts an update of the word's representation, a procedure referred to as negative sampling. Standard uses of word2vec recommend a greater or equal ratio of negative to positive sampling. The use of negative information in developing a representation of semantic information is often thought to be intimately associated with word2vec's prediction process. We assess the role of negative information in developing a semantic representation and show that its power does not reflect the use of a prediction mechanism. Finally, we show how negative information can be efficiently integrated into classic count‐based semantic models using parameter‐free analytical transformations.



    Upload a copy of this work     Papers currently archived: 91,349

External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

On Quantifying Semantic Information.Simon D'Alfonso - 2011 - Information 2 (1):61-101.
Pre-cognitive Semantic Information.Orlin Vakarelov - 2010 - Knowledge, Technology & Policy 23 (1-2):193-226.
Some suggested additions to the semantic cognition model.Jean M. Mandler - 2008 - Behavioral and Brain Sciences 31 (6):721-722.
Is semantic information meaningful data?Luciano Floridi - 2007 - Philosophy and Phenomenological Research 70 (2):351-370.


Added to PP

14 (#961,492)

6 months
5 (#652,053)

Historical graph of downloads
How can I increase my downloads?