What is it for a Machine Learning Model to Have a Capability?

British Journal for the Philosophy of Science (forthcoming)
  Copy   BIBTEX

Abstract

What can contemporary machine learning (ML) models do? Given the proliferation of ML models in society, answering this question matters to a variety of stakeholders, both public and private. The evaluation of models' capabilities is rapidly emerging as a key subfield of modern ML, buoyed by regulatory attention and government grants. Despite this, the notion of an ML model possessing a capability has not been interrogated: what are we saying when we say that a model is able to do something? And what sorts of evidence bear upon this question? In this paper, we aim to answer these questions, using the capabilities of large language models (LLMs) as a running example. Drawing on the large philosophical literature on abilities, we develop an account of ML models' capabilities which can be usefully applied to the nascent science of model evaluation. Our core proposal is a conditional analysis of model abilities (CAMA): crudely, a machine learning model has a capability to X just when it would reliably succeed at doing X if it 'tried'. The main contribution of the paper is making this proposal precise in the context of ML, resulting in an operationalisation of CAMA applicable to LLMs. We then put CAMA to work, showing that it can help make sense of various features of ML model evaluation practice, as well as suggest procedures for performing fair inter-model comparisons.

Links

PhilArchive

External links

  • This entry has no external links. Add one.
Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

Do ML models represent their targets?Emily Sullivan - forthcoming - Philosophy of Science.
What kind of novelties can machine learning possibly generate? The case of genomics.Emanuele Ratti - 2020 - Studies in History and Philosophy of Science Part A 83:86-96.

Analytics

Added to PP
2024-05-14

Downloads
157 (#125,307)

6 months
157 (#23,535)

Historical graph of downloads
How can I increase my downloads?

Author Profiles

Nathaniel Sharadin
University of Hong Kong
Jacqueline Harding
Stanford University

Citations of this work

No citations found.

Add more citations

References found in this work

No references found.

Add more references