Abstract

Citizen science projects have started to utilize Machine Learning to sort through large datasets generated in fields like astronomy, ecology and biodiversity, biology, and neuroimaging. Human–machine systems have been created to take advantage of the complementary strengths of humans and machines and have been optimized for efficiency and speed. We conducted qualitative content analysis on meta-summaries of documents reporting the results of 12 citizen science projects that used machine learning to optimize classification tasks. We examined the distribution of tasks between citizen scientists, experts, and algorithms, and how epistemic agency was enacted in terms of whose knowledge shapes the distribution of tasks, who decides what knowledge is relevant to the classification, and who validates it. In our descriptive results, we found that experts, who include professional scientists and algorithm developers, are involved in every aspect of a project, from annotating or labelling data to giving data to algorithms to train them to make decisions from predictions. Experts also test and validate models to improve their accuracy by scoring their outputs when algorithms fail to make correct decisions. Experts are mostly the humans involved in a loop, but when algorithms encounter problems, citizens are also involved at several stages. In this paper, we present three main examples of citizens-in-the-loop: when algorithms provide incorrect suggestions; when algorithms fail to know how to perform classification; and when algorithms pose queries. We consider the implications of the emphasis on optimization on the ideal of science and the role of citizen scientists from a perspective informed by Science and Technology Studies and Information Systems. Based on our findings, we conclude that ML in CS classification projects, far from being deterministic in its nature and effects, may be open to question. There is no guarantee that these technologies can replace citizen scientists, nor any guarantee that they can provide citizens with opportunities for more interesting tasks.