In their contribution ‘Ethics of the algorithmic prediction of goal of care preferences’1 Ferrario et al elaborate a from theory to practice contribution concerning the realisation of artificial intelligence (AI)-based patient preference prediction (PPP) systems. Such systems are intended to help find the treatment that the patient would have chosen in clinical situations—especially in the intensive care or emergency units—where the patient is no longer capable of making that decision herself. The authors identify several challenges that complicate their effective development, (...) application and evaluation—and offer solutions to them. One of these issues is the question of who should ultimately use said systems. While it is undisputed that clinicians should use these AI systems for their decision-making process, there is an ongoing debate about whether next of kin should use them as well. The authors advocate that ‘’access should be provided to both clinicians and loved ones with due explanations and as desired’. We will disagree with this assessment and explain in our commentary why it is important that surrogates provide their own assessments with as little external (AI) influence as possible. Why do next of kin actually participate in the process of preference finding and treatment decision-making? A key reason is that clinicians usually …. (shrink)

The use of machine learning, or “artificial intelligence” (AI) in medicine is widespread and growing. In this paper, I focus on a specific proposed clinical application of AI: using models to predict incapacitated patients’ treatment preferences. Drawing on results from machine learning, I argue this proposal faces a special moral problem. Machine learning researchers owe us assurance on this front before experimental research can proceed. In my conclusion I connect this concern to broader issues in AI safety.

Tensions between respect for autonomy and paternalism loom large in Ferrario et al ’s discussion of artificial intelligence (AI)-based preference predictors.1 To be sure, their analysis (rightfully) brings out the moral matter of respecting patient preferences. My point here, however, is that their consideration of AI-based preference predictors in treatment of incapacitated patients opens more fundamental moral questions about the desirability of over-ruling considered patient preferences, not only if these are disclosed by surrogates, but possibly also in treating competent patients. (...) I do not advocate such an evolution—the moral desirability of that calls for a much broader debate, one in which the meaning of ‘doing good’ in medicine, and how this intersects with normative views on ‘the goal(s) of medicine’ would be central elements. While my aim in this piece is more modest, I nonetheless hope to approach it sideways, by indicating how the contribution by Ferrario et al reopens discussion about paternalism and the normativity of preferences in medicine. This follows from what these authors give as reason for endorsing this technology in care for incapacitated patients. Their argument for employing such tools in case of incapacitation is based on the premise that advice of surrogate decision-makers about care preferences is suboptimal because of their biased perception …. (shrink)

BackgroundHealthcare providers have to make ethically complex clinical decisions which may be a source of stress. Researchers have recently introduced Artificial Intelligence (AI)-based applications to assist in clinical ethical decision-making. However, the use of such tools is controversial. This review aims to provide a comprehensive overview of the reasons given in the academic literature for and against their use.MethodsPubMed, Web of Science, Philpapers.org and Google Scholar were searched for all relevant publications. The resulting set of publications was title and abstract (...) screened according to defined inclusion and exclusion criteria, resulting in 44 papers whose full texts were analysed using the Kuckartz method of qualitative text analysis.ResultsArtificial Intelligence might increase patient autonomy by improving the accuracy of predictions and allowing patients to receive their preferred treatment. It is thought to increase beneficence by providing reliable information, thereby, supporting surrogate decision-making. Some authors fear that reducing ethical decision-making to statistical correlations may limit autonomy. Others argue that AI may not be able to replicate the process of ethical deliberation because it lacks human characteristics. Concerns have been raised about issues of justice, as AI may replicate existing biases in the decision-making process.ConclusionsThe prospective benefits of using AI in clinical ethical decision-making are manifold, but its development and use should be undertaken carefully to avoid ethical pitfalls. Several issues that are central to the discussion of Clinical Decision Support Systems, such as justice, explicability or human–machine interaction, have been neglected in the debate on AI for clinical ethics so far.Trial registrationThis review is registered at Open Science Framework (https://osf.io/wvcs9). (shrink)

In response to Ferrario _et al_’s 1 work entitled ‘Ethics of the algorithmic prediction of goal of care preferences: from theory to practice’, we would like to point out an area of concern: the risk of artificial intelligence (AI) paternalism in their proposed framework. Accordingly, in this commentary, we underscore the importance of the implementation of safeguards for AI algorithms before they are deployed in clinical practice. The goal of documenting a living will and advanced directives is to convey personal (...) preferences regarding the acceptance of therapies, including life support, for future use in case of losing one’s capacity. This is standard practice in the care of incapacitated critically ill patients as it is considered to extend the individual’s autonomy. Notably, most of the documents that intensivists encounter in clinical practice are written in a generic fashion and lack context. This problem usually leads to the reliance on... (shrink)

Nietzche claimed that once we know why to live, we’ll suffer almost any how.1 Artificial intelligence (AI) is used widely for the how, but Ferrario et al now advocate using AI for the why.2 Here, I offer my doubts on practical grounds but foremost on ethical ones. Practically, individuals already vacillate over the why, wavering with time and circumstance. That AI could provide prosthetics (or orthotics) for human agency feels unrealistic here, not least because ‘answers’ would be largely unverifiable. Ethically, (...) the concern is that AI stands to frack our humanity. We form a fragile ecosystem of ethical subjects, our responsiveness to others’ suffering, enabled by our own. To deliberate together for incapacitated others is among those solemn privileges that verify our humanity. Having AI mine these delicate pain-forests risks treating our suffering as the new oil—to be extracted and exploited, but beyond our vision and at our cost. Let’s briefly develop each idea, starting with the how/why distinction. This is palpable, even for more prosaic questions like how or why to drive. The former admits of increasingly sophisticated technological fixes and nudge; the latter often remains very particular and personal. How much greater then, the difference between …. (shrink)

The JME editorial team selects its feature articles from the best papers accepted for publication based on their quality, novelty and capacity to move debate forward on a specific issue. Feature articles are made freely available and are published alongside reviewed and submitted commentaries. We do this partly to promote and acknowledge excellent work in medical ethics, but also to encourage authors to submit their best papers to the JME. JME feature articles have deepened the analysis of some central issues (...) for medical ethics. ‘Common morality’ is one way of underpinning general ethical principles and it is a concept that is central to the justification that Beauchamp and Childress give in their classic articulation of the four principles. 1 Rhodes’ critical discussion of common morality in her feature article 2 has been well cited since then, which is a good thing from the perspective of the author... (shrink)

Large language models (LLMs) have now entered the realm of medical ethics. In a recent study, Balaset alexamined the performance of GPT-4, a commercially available LLM, assessing its performance in generating responses to diverse medical ethics cases. Their findings reveal that GPT-4 demonstrates an ability to identify and articulate complex medical ethical issues, although its proficiency in encoding the depth of real-world ethical dilemmas remains an avenue for improvement. Investigating the integration of LLMs into medical ethics decision-making appears to be (...) an interesting avenue of research. However, despite the promising trajectory of LLM technology in medicine, it is crucial to exercise caution and refrain from attributing their expertise to medical ethics. Our thesis follows an examination of the nature of expertise and the epistemic limitations that affect LLM technology. As a result, we propose two more fitting applications of LLMs in medical ethics: first, as tools for mining electronic health records or scientific literature, thereby supplementing evidence for resolving medical ethics cases, and second, as educational platforms to foster ethical reflection and critical thinking skills among students and residents. The integration of LLMs in medical ethics, while promising, requires careful consideration of their epistemic limitations. Consequently, a well-considered definition of their role in ethically sensitive decision-making is crucial. (shrink)

In our recent article ‘The Ethics of the Algorithmic Prediction of Goal of Care Preferences: From Theory to Practice’1, we aimed to ignite a critical discussion on why and how to design artificial intelligence (AI) systems assisting clinicians and next-of-kin by predicting goal of care preferences for incapacitated patients. Here, we would like to thank the commentators for their valuable responses to our work. We identified three core themes in their commentaries: (1) the risks of AI paternalism, (2) worries about (...) attacks to our humanity stemming from the use of AI and (3) the possibility of designing AI systems for more relevant use cases than the one we consider in our work. We shall focus on these themes, leaving aside some other interesting suggestions, given the limited space available for our response. Diaz Milian and Bhattacharyya discuss the risks of AI paternalism highlighting how the use of an AI to predict goal of care preferences may incentivise clinicians and surrogates to shift the burden of decision making to the machine.2 In particular, ‘[i]f the AI-generated response is given priority above the other agents’2 and the AI makes decisions autonomously, then ‘AI paternalism’ would emerge.2 Against this possibility, the authors suggest four safeguards to be implemented in the AI life-cycle.2 These are procedures that may improve the trustworthiness of the AI and human control over it.3 We agree that it is important …. (shrink)

When making substituted judgments for incapacitated patients, surrogates often struggle to guess what the patient would want if they had capacity. Surrogates may also agonize over having the (sole) responsibility of making such a determination. To address such concerns, a Patient Preference Predictor (PPP) has been proposed that would use an algorithm to infer the treatment preferences of individual patients from population-level data about the known preferences of people with similar demographic characteristics. However, critics have suggested that even if such (...) a PPP were more accurate, on average, than human surrogates in identifying patient preferences, the proposed algorithm would nevertheless fail to respect the patient’s (former) autonomy since it draws on the ‘wrong’ kind of data: namely, data that are not specific to the individual patient and which therefore may not reflect their actual values, or their reasons for having the preferences they do. Taking such criticisms on board, we here propose a new approach: the Personalized Patient Preference Predictor (P4). The P4 is based on recent advances in machine learning, which allow technologies including large language models to be more cheaply and efficiently ‘fine-tuned’ on person-specific data. The P4, unlike the PPP, would be able to infer an individual patient’s preferences from material (e.g., prior treatment decisions) that is in fact specific to them. Thus, we argue, in addition to being potentially more accurate at the individual level than the previously proposed PPP, the predictions of a P4 would also more directly reflect each patient’s own reasons and values. In this article, we review recent discoveries in artificial intelligence research that suggest a P4 is technically feasible, and argue that, if it is developed and appropriately deployed, it should assuage some of the main autonomy-based concerns of critics of the original PPP. We then consider various objections to our proposal and offer some tentative replies. (shrink)