In this essay we explore whether and how we should use bibliometrics in hiring, promoting, and granting in the academy. We suggest a Deweyan-Hickmanian pragmatist approach to reflecting on the technology of bibliometrics as a resource for inherently qualitative judgements in these deliberations. We begin with a literature review of current work evaluating the role and use of bibliometrics in the academy, from advocating for them to questioning their construct validity and assessing their limitations and/or dangerous consequences. In the next (...) section, we take a step back and provide a Deweyan philosophical analysis of the validity and practical use of qualitative assessment. In our conclusion we reflect on what pragmatism can teach about recent algorithmic tools used for faculty evaluation. (shrink)