Coding Psychological Constructs in Text Using Mechanical Turk: A Reliable, Accurate, and Efficient Alternative

Frontiers in Psychology 7:191179 (2016)
  Copy   BIBTEX


In this paper we evaluate how to effectively use the crowdsourcing service, Amazon’s Mechanical Turk (MTurk), to content analyze textual data for use in psychological research. MTurk is a marketplace for discrete tasks completed by workers, typically for small amounts of money. MTurk has been used to aid psychological research in general, and content analysis in particular. In the current study, MTurk workers content analyzed personally-written textual data using coding categories previously developed and validated in psychological research. These codes were evaluated for reliability, accuracy, completion time, and cost. Results indicate that MTurk workers categorized textual data with comparable reliability and accuracy to both previously published studies and expert raters. Further, the coding tasks were performed quickly and cheaply. These data suggest that crowdsourced content analysis can help advance psychological research.



    Upload a copy of this work     Papers currently archived: 92,150

External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

The Historical Work of Niqula El-Turk 1763-1828.George M. Haddad & Niqula El-Turk - 1961 - Journal of the American Oriental Society 81 (3):247-251.
Logical constructs and psychological theory.F. H. George - 1953 - Psychological Review 60 (1):1-6.
Nonconstructural aspects of psychological constructs.W. W. Meissner - 1958 - Psychological Review 65 (3):143-150.
How to Be a Reliabilist.Jonathan Kvanvig - 1986 - American Philosophical Quarterly 23 (2):189 - 198.
Abbe de Saint-pierre: European union and the turk.T. Mastnak - 1998 - History of Political Thought 19 (4):570-598.
Coding lemmata in L.George Kafkoulis - 2004 - Archive for Mathematical Logic 43 (2):193-213.


Added to PP

11 (#1,140,884)

6 months
5 (#645,438)

Historical graph of downloads
How can I increase my downloads?

Citations of this work

No citations found.

Add more citations

References found in this work

No references found.

Add more references