During the interwar years psychologists Louis Leon Thurstone and Rensis Likert produced newly standardized forms of questionnaires. Both built on developments in mental testing, including the use of restricted sets of answers and the emergence of statistical techniques, to create questionnaires that employed numerical scaling. This transformation in shape of questionnaires was intimately tied up with both psychologists’ nominal subject of investigation: attitudes. Efforts to render psychology a socially valuable and influential science spurred psychologists to create sophisticated and (...) increasingly precise means of measuring social attitudes. Reducing mental dispositions to mere numbers on a scale, these developments also initiated new relationships between psychology and the public. Rather than engage a wide spectrum of the public directly in the research process, questionnaire research was limited to those within academic circles. Even so, research with questionnaires aimed to comment on attitudes in the public more broadly. The kind of ‘thin description’ afforded by numerical scales, though used to measure individual psychological subjects, afforded psychologists the opportunity to craft their vision of an increasingly attitudinal public, one positioned as best governed with the aid of psychological expertise. (shrink)

Measuring psychological attributes, such as motivation, typically involves rating scales, assuming that an attribute can be ordered, and that ratings represent this order. Previously, only the first assumption had been tested, albeit limited. First, we checked the ordinal structure of motivation, looking at whether people can establish transitive relations between motivation levels in pairwise comparisons; and we found different ordering patterns: strict transitive, weak transitive, changing order, and intransitivity. The rate of intransitivity was similar to that found previously and was (...) somewhat higher than we obtained when we asked participants to compare definitely quantitative attributes. Second, we checked if specific ordering patterns were related to individual interpretations of the statements that deviated from expected motivation types. Indeed, about a third of participants miscategorized statements, and these deviant interpretations were related to intransitivity as well as weak transitivity. Third, we checked whether Likert ratings represent the order of motives obtained from pairwise comparisons. We found rather homomorphic representation: ratings correlated with the order, but they did not differentiate between different ordering patterns and hierarchies of motives. We conclude that the Likert rating scale provides less information about respondents than pairwise ordering. The findings question the mainstream practice of using rating scales without testing underlying assumptions. (shrink)

This brief commentary treats Polanyi’s newly found lecture, “Rules of Rightness,” as an occasion to revisit some earlier claims I made about the use of rating scales in social science research. It serves as something of an interim report on an ongoing inquiry into what an effective response to social science would look like from a Polanyian perspective.

Developing self-report Likert scales is an essential part of modern psychology. However, it is hard for psychologists to remain apprised of best practices as methodological developments accumulate. To address this, this current paper offers a selective review of advances in Likert scale development that have occurred over the past 25 years. We reviewed six major measurement journals between the years 1995–2019 and identified key advances, ultimately including 40 papers and offering written summaries of each. We supplemented this review (...) with an in-depth discussion of five particular advances: conceptions of construct validity, creating better construct definitions, readability tests for generating items, alternative measures of precision [e.g., coefficient omega and item response theory information], and ant colony optimization for creating short forms. The Supplementary Material provides further technical details on these advances and offers guidance on software implementation. This paper is intended to be a resource for psychological researchers to be informed about more recent psychometric progress in Likert scale creation. (shrink)