Background The use of Artificial Intelligence (AI) has the potential to provide relief in the challenging and often stressful clinical setting for physicians. So far, however, the actual changes in work for physicians remain a prediction for the future, including new demands on the social level of medical practice. Thus, the question of how the requirements for physicians will change due to the implementation of AI is addressed. Methods The question is approached through conceptual considerations based on the potentials that (...) AI already offer and the focus on central normative concepts of trust, explainability, and responsibility which play an important role when implementing AI in everyday clinical practice. Conclusion Interpersonal communication will not disappear upon implementation of AI. Instead, it is much more likely that the exchange between various actors in medical practice will become increasingly important. This adds another level of complexity to practical concepts such as Shared-Decision-Making, which must be addressed in empirical research, including the involvement of AI systems as actors in communication. (shrink)

Artificial intelligence (AI) and its introduction into clinical pathways presents an array of ethical issues that are being discussed in the JME. 1–7 The development of AI technologies that can produce text that will pass plagiarism detectors 8 and are capable of appearing to be written by a human author 9 present new issues for medical ethics. One set of worries concerns authorship and whether it will now be possible to know that an author or student in fact produced submitted (...) work. That seems likely to be a general worry for secondary and higher education, as well as for all academic journals. Thus far generative AI chatbots do not seem to be able to produce a fully referenced and well-argued ethics article, but they probably could generate a blog or student essay that would... (shrink)

A discussion concerning whether to conceive Artificial Intelligence systems as responsible moral entities, also known as “artificial moral agents”, has been going on for some time. In this regard, we argue that the notion of “moral agency” is to be attributed only to humans based on their autonomy and sentience, which AI systems lack. We analyze human responsibility in the presence of AI systems in terms of meaningful control and due diligence and argue against fully automated systems in medicine. With (...) this perspective in mind, we focus on the use of AI-based diagnostic systems and shed light on the complex networks of persons, organizations and artifacts that come to be when AI systems are designed, developed, and used in medicine. We then discuss relational criteria of judgment in support of the attribution of responsibility to humans when adverse events are caused or induced by errors in AI systems. (shrink)

The notion of “responsibility gap” with artificial intelligence (AI) was originally introduced in the philosophical debate to indicate the concern that “learning automata” may make more difficult or impossible to attribute moral culpability to persons for untoward events. Building on literature in moral and legal philosophy, and ethics of technology, the paper proposes a broader and more comprehensive analysis of the responsibility gap. The responsibility gap, it is argued, is not one problem but a set of at least four interconnected (...) problems – gaps in culpability, moral and public accountability, active responsibility—caused by different sources, some technical, other organisational, legal, ethical, and societal. Responsibility gaps may also happen with non-learning systems. The paper clarifies which aspect of AI may cause which gap in which form of responsibility, and why each of these gaps matter. It proposes a critical review of partial and non-satisfactory attempts to address the responsibility gap: those which present it as a new and intractable problem (“fatalism”), those which dismiss it as a false problem (“deflationism”), and those which reduce it to only one of its dimensions or sources and/or present it as a problem that can be solved by simply introducing new technical and/or legal tools (“solutionism”). The paper also outlines a more comprehensive approach to address the responsibility gaps with AI in their entirety, based on the idea of designing socio-technical systems for “meaningful human control", that is systems aligned with the relevant human reasons and capacities. (shrink)

BackgroundMachine learning-based clinical decision support systems (ML_CDSS) are increasingly employed in various sectors of health care aiming at supporting clinicians’ practice by matching the characteristics of individual patients with a computerised clinical knowledge base. Some studies even indicate that ML_CDSS may surpass physicians’ competencies regarding specific isolated tasks. From an ethical perspective, however, the usage of ML_CDSS in medical practice touches on a range of fundamental normative issues. This article aims to add to the ethical discussion by using professionalisation theory (...) as an analytical lens for investigating how medical action at the micro level and the physician–patient relationship might be affected by the employment of ML_CDSS.Main textProfessionalisation theory, as a distinct sociological framework, provides an elaborated account of what constitutes client-related professional action, such as medical action, at its core and why it is more than pure expertise-based action. Professionalisation theory is introduced by presenting five general structural features of professionalised medical practice: (i) the patient has a concern; (ii) the physician deals with the patient’s concern; (iii) s/he gives assistance without patronising; (iv) s/he regards the patient in a holistic manner without building up a private relationship; and (v) s/he applies her/his general expertise to the particularities of the individual case. Each of these five key aspects are then analysed regarding the usage of ML_CDSS, thereby integrating the perspectives of professionalisation theory and medical ethics.ConclusionsUsing ML_CDSS in medical practice requires the physician to pay special attention to those facts of the individual case that cannot be comprehensively considered by ML_CDSS, for example, the patient’s personality, life situation or cultural background. Moreover, the more routinized the use of ML_CDSS becomes in clinical practice, the more that physicians need to focus on the patient’s concern and strengthen patient autonomy, for instance, by adequately integrating digital decision support in shared decision-making. (shrink)

Assistive systems based on Artificial Intelligence (AI) are bound to reshape decision-making in all areas of society. One of the most intricate challenges arising from their implementation in high-stakes environments such as medicine concerns their frequently unsatisfying levels of explainability, especially in the guise of the so-called black-box problem: highly successful models based on deep learning seem to be inherently opaque, resisting comprehensive explanations. This may explain why some scholars claim that research should focus on rendering AI systems understandable, rather (...) than explainable. Yet, there is a grave lack of agreement concerning these terms in much of the literature on AI. We argue that the seminal distinction made by the philosopher and physician Karl Jaspers between different types of explaining and understanding in psychopathology can be used to promote greater conceptual clarity in the context of Machine Learning (ML). Following Jaspers, we claim that explaining and understanding constitute multi-faceted epistemic approaches that should not be seen as mutually exclusive, but rather as complementary ones as in and of themselves they are necessarily limited. Drawing on the famous example of Watson for Oncology we highlight how Jaspers’ methodology translates to the case of medical AI. Classical considerations from the philosophy of psychiatry can therefore inform a debate at the centre of current AI ethics, which in turn may be crucial for a successful implementation of ethically and legally sound AI in medicine. (shrink)

The justification and rationality of this paper is to present some fundamental principles, theories, and concepts that we believe moulds the nucleus of a good artificial intelligence (AI) society. The morally accepted significance and utilitarian concerns that stems from the inception and realisation of an AI’s structural foundation are displayed in this study. This paper scrutinises the structural foundation, fundamentals, and cardinal righteous remonstrations, as well as the gaps in mechanisms towards novel prospects and perils in determining resilient fundamentals, accountability, (...) and AI’s convoluted and responsible implications. We outline a number of salient and practical benefits, in which to place moral norms within the mise en scène of AI, to delineate the rudimentary ethical dilemmas and decorous directions within the realms of AI. (shrink)

Machine intelligence already helps medical staff with a number of tasks. Ethical decision-making, however, has not been handed over to computers. In this proof-of-concept study, we show how an algorithm based on Beauchamp and Childress’ prima-facie principles could be employed to advise on a range of moral dilemma situations that occur in medical institutions. We explain why we chose fuzzy cognitive maps to set up the advisory system and how we utilized machine learning to train it. We report on the (...) difficult task of operationalizing the principles of beneficence, non-maleficence and patient autonomy, and describe how we selected suitable input parameters that we extracted from a training dataset of clinical cases. The first performance results are promising, but an algorithmic approach to ethics also comes with several weaknesses and limitations. Should one really entrust the sensitive domain of clinical ethics to machine intelligence? (shrink)

In this paper, we first classify different types of second opinions and evaluate the ethical and epistemological implications of providing those in a clinical context. Second, we discuss the issue of how artificial intelligent could replace the human cognitive labour of providing such second opinion and find that several AI reach the levels of accuracy and efficiency needed to clarify their use an urgent ethical issue. Third, we outline the normative conditions of how AI may be used as second opinion (...) in clinical processes, weighing the benefits of its efficiency against concerns of responsibility attribution. Fourth, we provide a ‘rule of disagreement’ that fulfils these conditions while retaining some of the benefits of expanding the use of AI-based decision support systems in clinical contexts. This is because the rule of disagreement proposes to use AI as much as possible, but retain the ability to use human second opinions to resolve disagreements between AI and physician-in-charge. Fifth, we discuss some counterarguments. (shrink)

This paper explores the role and resolution of disagreements between physicians and their diagnostic AI-based decision support systems. With an ever-growing number of applications for these independently operating diagnostic tools, it becomes less and less clear what a physician ought to do in case their diagnosis is in faultless conflict with the results of the DSS. The consequences of such uncertainty can ultimately lead to effects detrimental to the intended purpose of such machines, e.g. by shifting the burden of proof (...) towards a physician. Thus, we require normative clarity for integrating these machines without affecting established, trusted, and relied upon workflows. In reconstructing different causes of conflicts between physicians and their AI-based tools—inspired by the approach of “meaningful human control” over autonomous systems and the challenges to resolve them—we will delineate normative conditions for “meaningful disagreements”. These incorporate the potential of DSS to take on more tasks and outline how the moral responsibility of a physician can be preserved in an increasingly automated clinical work environment. (shrink)

Background Big data and AI applications now play a major role in many health contexts. Much research has already been conducted on ethical and social challenges associated with these technologies. Likewise, there are already some studies that investigate empirically which values and attitudes play a role in connection with their design and implementation. What is still in its infancy, however, is the comparative investigation of the perspectives of different stakeholders. Methods To explore this issue in a multi-faceted manner, we conducted (...) semi-structured interviews as well as focus group discussions with patients and clinicians. These empirical methods were used to gather interviewee’s views on the opportunities and challenges of medical AI and other data-intensive applications. Results Different clinician and patient groups are exposed to medical AI to differing degrees. Interviewees expect and demand that the purposes of data processing accord with patient preferences, and that data are put to effective use to generate social value. One central result is the shared tendency of clinicians and patients to maintain individualistic ascriptions of responsibility for clinical outcomes. Conclusions Medical AI and the proliferation of data with import for health-related inferences shape and partially reconfigure stakeholder expectations of how these technologies relate to the decision-making of human agents. Intuitions about individual responsibility for clinical outcomes could eventually be disrupted by the increasing sophistication of data-intensive and AI-driven clinical tools. Besides individual responsibility, systemic governance will be key to promote alignment with stakeholder expectations in AI-driven and data-intensive health settings. (shrink)

Artificial intelligence is currently changing many areas of society. Especially in health, where critical decisions are made, questions of control must be renegotiated: who is in control when an automated system makes clinically relevant decisions? Increasingly, the concept of meaningful human control (MHC) is being invoked for this purpose. However, it is unclear exactly how this concept is to be understood in health. Through a systematic review, we present the current state of the concept of MHC in health. The results (...) show that there is not yet a robust MHC concept for health. We propose a broader understanding of MHC along three strands of action: enabling, exercising and evaluating control. Taking into account these strands of action and the established rules and processes in the different health sectors, the MHC concept needs to be further developed to avoid falling into two gaps, which we have described as theoretical and labelling gaps. (shrink)

Artificial intelligence (AI) is a powerful tool for several healthcare tasks. AI tools are suited to optimize predictive models in medicine. Ethical debates about AI’s extension of the predictive power of medical models suggest a need to adapt core principles of medical ethics. This article demonstrates that a popular interpretation of the principle of justice in healthcare needs amendment given the effect of AI on decision-making. The procedural approach to justice, exemplified with Norman Daniels and James Sabin’saccountability for reasonablenessconception, needs (...) amendment because, as research into algorithmic fairness shows, it is insufficiently sensitive to differential effects of seemingly just principles on different groups of people. The same line of research generates methods to quantify differential effects and make them amenable for correction. Thus, what is needed to improve the principle of justice is a combination of procedures for selecting just criteria and principles and the use of algorithmic tools to measure the real impact these criteria and principles have. In this article, the author shows that algorithmic tools do not merely raise issues of justice but can also be used in their mitigation by informing us about the real effects certain distributional principles and criteria would create. (shrink)

In recent years, there has been a surge of high-profile publications on applications of artificial intelligence (AI) systems for medical diagnosis and prognosis. While AI provides various opportunities for medical practice, there is an emerging consensus that the existing studies show considerable deficits and are unable to establish the clinical benefit of AI systems. Hence, the view that the clinical benefit of AI systems needs to be studied in clinical trials—particularly randomised controlled trials (RCTs)—is gaining ground. However, an issue that (...) has been overlooked so far in the debate is that, compared with drug RCTs, AI RCTs require methodological adjustments, which entail ethical challenges. This paper sets out to develop a systematic account of the ethics of AI RCTs by focusing on the moral principles of clinical equipoise, informed consent and fairness. This way, the objective is to animate further debate on the (research) ethics of medical AI. (shrink)

Machine learning-driven clinical decision support systems (ML-CDSSs) seem impressively promising for future routine and emergency care. However, reflection on their clinical implementation reveals a wide array of ethical challenges. The preferences, concerns and expectations of professional stakeholders remain largely unexplored. Empirical research, however, may help to clarify the conceptual debate and its aspects in terms of their relevance for clinical practice. This study explores, from an ethical point of view, future healthcare professionals’ attitudes to potential changes of responsibility and decision-making (...) authority when using ML-CDSS. Twenty-seven semistructured interviews were conducted with German medical students and nursing trainees. The data were analysed based on qualitative content analysis according to Kuckartz. Interviewees’ reflections are presented under three themes the interviewees describe as closely related: (self-)attribution of responsibility, decision-making authority and need of (professional) experience. The results illustrate the conceptual interconnectedness of professional responsibility and its structural and epistemic preconditions to be able to fulfil clinicians’ responsibility in a meaningful manner. The study also sheds light on the four relata of responsibility understood as a relational concept. The article closes with concrete suggestions for the ethically sound clinical implementation of ML-CDSS. (shrink)

Background Given that AI-driven decision support systems (AI-DSS) are intended to assist in medical decision making, it is essential that clinicians are willing to incorporate AI-DSS into their practice. This study takes as a case study the use of AI-driven cardiotography (CTG), a type of AI-DSS, in the context of intrapartum care. Focusing on the perspectives of obstetricians and midwives regarding the ethical and trust-related issues of incorporating AI-driven tools in their practice, this paper explores the conditions that AI-driven CTG (...) must fulfill for clinicians to feel justified in incorporating this assistive technology into their decision-making processes regarding interventions in labor. Methods This study is based on semi-structured interviews conducted online with eight obstetricians and five midwives based in England. Participants were asked about their current decision-making processes about when to intervene in labor, how AI-driven CTG might enhance or disrupt this process, and what it would take for them to trust this kind of technology. Interviews were transcribed verbatim and analyzed with thematic analysis. NVivo software was used to organize thematic codes that recurred in interviews to identify the issues that mattered most to participants. Topics and themes that were repeated across interviews were identified to form the basis of the analysis and conclusions of this paper. Results There were four major themes that emerged from our interviews with obstetricians and midwives regarding the conditions that AI-driven CTG must fulfill: (1) the importance of accurate and efficient risk assessments; (2) the capacity for personalization and individualized medicine; (3) the lack of significance regarding the type of institution that develops technology; and (4) the need for transparency in the development process. Conclusions Accuracy, efficiency, personalization abilities, transparency, and clear evidence that it can improve outcomes are conditions that clinicians deem necessary for AI-DSS to meet in order to be considered reliable and therefore worthy of being incorporated into the decision-making process. Importantly, healthcare professionals considered themselves as the epistemic authorities in the clinical context and the bearers of responsibility for delivering appropriate care. Therefore, what mattered to them was being able to evaluate the reliability of AI-DSS on their own terms, and have confidence in implementing them in their practice. (shrink)

Many ethical concerns have been voiced about Clinical Decision Support Systems (CDSSs). Special attention has been paid to the effect of CDSSs on autonomy, responsibility, fairness and transparency. This journal has featured a discussion between Rosalind McDougall and Ezio Di Nucci that focused on the impact of IBM’s Watson for Oncology (Watson) on autonomy. The present article elaborates on this discussion in three ways. First, using Jonathan Pugh’s account of rational autonomy we show that how Watson presents its results might (...) impact decisional autonomy, while how Watson produces knowledge might affect practical autonomy. Second, by drawing an analogy with patient decision aids we identify an empirical way of estimating Watson’s impact on autonomy (ie, value-congruence). Lastly, McDougall introduced the notion of value-flexible design as a way to account for the diverging preferences patients hold. We will clarify its relation with the established domain of value-sensitive design. In terms of the tripartite methodology of value-sensitive design, we offer aconceptualclarification using Pugh’s account of rational autonomy, anempiricaltool to evaluate Watson’s impact on autonomy and situate a group oftechnicaloptions to incorporate autonomy in Watson’s design. (shrink)

The application of machine learning in health care holds great promise for improving care. Indeed, our own team is collaborating with experts in machine learning and statistical modeling to bu...

Zusammenfassung Die rasanten Entwicklungen im Bereich der Künstlichen Intelligenz und Robotik stellen nicht nur die Ethik, sondern auch das Recht vor neue Herausforderungen, gerade im Bereich der Medizin und Pflege. Grundsätzlich hat der Einsatz von KI dabei das Potenzial, sowohl die Heilbehandlungen als auch den adäquaten Umgang im Rahmen der Pflege zu erleichtern, wenn nicht sogar zu verbessern. Verwaltungsaufgaben, die Überwachung von Vitalfunktionen und deren Parameter sowie die Untersuchung von Gewebeproben etwa könnten autonom ablaufen. In Diagnostik und Therapie können Systeme (...) die behandelnde Ärztin unterstützen. Intelligente Betten ermöglichen eine Frühmobilisierung der Patient:innen bei gleichzeitig geringerem Personalaufwand. Gleichwohl birgt der Einsatz der Systeme rechtliche Herausforderungen. So besteht das Risiko einer Verletzung der beteiligten Personen. Im Gegensatz zu herkömmlichen Technologien „leidet“ KI unter der „Black-Box-Problematik“: Die von den Systemen generierten Ergebnisse sind nicht mehr vollständig vorhersehbar und nachvollziehbar. Der Einsatz birgt unbekannte und unkalkulierbare Risiken, was sich insbesondere auf die zivilrechtliche Haftung und strafrechtliche Verantwortung auswirkt. Wem die Entscheidungen der Systeme normativ zuzurechnen sind, ist eine Kernfrage des juristischen Diskurses. Die aus praktikabilitätsgründen naheliegende Wahl, dem letztentscheidenden Menschen das Verhalten eines KI-Systems zuzurechnen, überzeugt nicht in allen Fällen, sondern degradiert ihn häufig zum symbolischen Haftungsknecht und legt ihm einseitig die Risiken der Technologien auf. Weiterhin ergeben sich im Bereich der Medizin und Pflege Fragen hinsichtlich der Zulassung von KI-Systemen, da die Maschinen während der Nutzung weiterlernen und so ihren strukturellen Aufbau kontinuierlich verändern. Es ist daher angebracht, sich frühzeitig mit dem Konfliktpotential aus ethischer und rechtlicher Sicht auseinander zu setzen, um einer potenziellen gesellschaftlichen Angst vor derartigen Systemen vorzubeugen und einen praxisgerechten Handlungsrahmen zu schaffen. (shrink)

The rapid adoption and implementation of artificial intelligence in medicine creates an ontologically distinct situation from prior care models. There are both potential advantages and disadvantages with such technology in advancing the interests of patients, with resultant ontological and epistemic concerns for physicians and patients relating to the instatiation of AI as a dependent, semi- or fully-autonomous agent in the encounter. The concept of libertarian paternalism potentially exercised by AI (and those who control it) has created challenges to conventional assessments (...) of patient and physician autonomy. The unclear legal relationship between AI and its users cannot be settled presently, an progress in AI and its implementation in patient care will necessitate an iterative discourse to preserve humanitarian concerns in future models of care. This paper proposes that physicians should neither uncritically accept nor unreasonably resist developments in AI but must actively engage and contribute to the discourse, since AI will affect their roles and the nature of their work. One’s moral imaginative capacity must be engaged in the questions of beneficence, autonomy, and justice of AI and whether its integration in healthcare has the potential to augment or interfere with the ends of medical practice. (shrink)

BackgroundThere is a growing concern about artificial intelligence (AI) applications in healthcare that can disadvantage already under-represented and marginalised groups (eg, based on gender or race).ObjectivesOur objectives are to canvas the range of strategies stakeholders endorse in attempting to mitigate algorithmic bias, and to consider the ethical question of responsibility for algorithmic bias.MethodologyThe study involves in-depth, semistructured interviews with healthcare workers, screening programme managers, consumer health representatives, regulators, data scientists and developers.ResultsFindings reveal considerable divergent views on three key issues. First, (...) views on whether bias is a problem in healthcare AI varied, with most participants agreeing bias is a problem (which we call the bias-critical view), a small number believing the opposite (the bias-denial view), and some arguing that the benefits of AI outweigh any harms or wrongs arising from the bias problem (the bias-apologist view). Second, there was a disagreement on the strategies to mitigate bias, and who is responsible for such strategies. Finally, there were divergent views on whether to include or exclude sociocultural identifiers (eg, race, ethnicity or gender-diverse identities) in the development of AI as a way to mitigate bias.Conclusion/significanceBased on the views of participants, we set out responses that stakeholders might pursue, including greater interdisciplinary collaboration, tailored stakeholder engagement activities, empirical studies to understand algorithmic bias and strategies to modify dominant approaches in AI development such as the use of participatory methods, and increased diversity and inclusion in research teams and research participant recruitment and selection. (shrink)