The ‘black box problem’ is a long-standing talking point in debates about artificial intelligence. This is a significant point of tension between ethicists, programmers, clinicians and anyone else working on developing AI for healthcare applications. However, the precise definition of these systems are often left undefined, vague, unclear or are assumed to be standardised within AI circles. This leads to situations where individuals working on AI talk over each other and has been invoked in numerous debates between opaque and explainable (...) systems. This paper proposes a coherent and clear definition for the black box problem to assist in future discussions about AI in healthcare. This is accomplished by synthesising various definitions in the literature and examining several criteria that can be extrapolated from these definitions. There are no data in this work. (shrink)

Background: The COVID-19 pandemic severely strained the already unprepared Italian healthcare system. This had repercussions on healthcare workers, stemming, in particular, from a lack of clear guidelines, adequate protective equipment, and professional preparedness. Such conditions were especially prevalent in Northern Italy.Objectives: This study aimed to examine COVID-19-related professional and psychoemotional stress among nurses and doctors in two hospitals in Northern Italy, along with the worst critical incidents affecting healthcare personnel. A parallel objective was to elicit healthcare professionals' opinions about what (...) changes are needed in the healthcare system's operations, as well as about the relational/emotional skills that are needed to better manage relationships with patients in emergency situations.Participants: Snowball sampling was used to recruit participants and yielded 17 hospital professionals: six nurses and 11 doctors. Three of these professionals worked in intensive care and the others in different wards. All had close contact with COVID-19 patients.Methods: The study employed a qualitative research design, using in-depth interviews of ~60 min each that were conducted via Skype video calls. The interviews were recorded and transcribed, then analysed. The qualitative analysis employed mixed methods to identify the most relevant and recursive themes from the interviews.Results: Four fundamental themes emerged from our analysis of the interview texts: disorganisation and psychoemotional stress; urgency and critical incidents; everything surreal; and disruptions in empathetic relationships with patients.Conclusions: Through our analysis of the interview narratives, we found that systematic and in-depth psychological training is needed to prepare professionals for altered relationships with patients in emergencies; use of exceptional medical equipment; elaboration of new bioethical models suitable for disasters and pandemics; and engagement with the themes of death and dying. (shrink)

Mental health apps bring unprecedented benefits and risks to individual and public health. A thorough evaluation of these apps involves considering two aspects that are often neglected: the algorithms they deploy and the functions they perform. We focus on mental health apps based on black box algorithms, explore their forms of opacity, discuss the implications derived from their opacity, and propose how to use their outcomes in mental healthcare, self‐care practices, and research. We argue that there is a relevant distinction (...) between functions performed by algorithms in mental health apps, and we focus on the functions of analysis and generation of advice. When performing analytic functions, such as identifying patterns and making predictions concerning people's emotions, thoughts, and behaviors, black box algorithms can be better than other algorithms to provide information to identify early signs of relapse, support diagnostic processes, and improve research by generating outcomes that lead to a better understanding of mental health. However, when carrying out the function of providing mental health advice, black box algorithms have the potential to deliver unforeseen advice that may harm users. We argue that the outcomes of these apps may be trustworthy as a complementary source of information, but express caution about black box algorithms that give advice directly to users. To reap the benefits of mental health apps based on black box algorithms and avoid unintended consequences, we critically need to know whether these algorithms are fulfilling the function of providing mental health advice. (shrink)

Developers are often the engine behind the creation and implementation of new technologies, including in the artificial intelligence surge that is currently underway. In many cases these new technologies introduce significant risk to affected stakeholders; risks that can be reduced and mitigated by such a dominant party. This is fully recognized by texts that analyze risks in the current AI transformation, which suggest voluntary adoption of ethical standards and imposing ethical standards via regulation and oversight as tools to compel developers (...) to reduce such risks. However, what these texts usually sidestep is the question of how aware developers are to the risks they are creating with these new AI technologies, and what their attitudes are towards such risks. This paper asks to rectify this gap in research, by analyzing an ongoing case study. Focusing on six Israeli AI startups in the field of radiology, I carry out a content analysis of their online material in order to examine these companies’ stances towards the potential threat their automated tools pose to patient safety and to the work-standing of healthcare professionals. Results show that these developers are aware of the risks their AI products pose, but tend to deny their own role in the technological transformation and dismiss or downplay the risks to stakeholders. I conclude by tying these findings back to current risk-reduction recommendations with regards to advanced AI technologies, and suggest which of them hold more promise in light of developers’ attitudes. (shrink)

This paper provides the first comprehensive analysis of ethical issues raised by artificial intelligence (AI) in veterinary medicine for companion animals. Veterinary medicine is a socially valued service, which, like human medicine, will likely be significantly affected by AI. Veterinary AI raises some unique ethical issues because of the nature of the client–patient–practitioner relationship, society’s relatively minimal valuation and protection of nonhuman animals and differences in opinion about responsibilities to animal patients and human clients. The paper examines how these distinctive (...) features influence the ethics of AI systems that might benefit clients, veterinarians and animal patients—but also harm them. It offers practical ethical guidance that should interest ethicists, veterinarians, clinic owners, veterinary bodies and regulators, clients, technology developers and AI researchers. (shrink)

Do we suddenly become justified in treating robots like humans by positing new notions like “artificial moral agency” and “artificial moral responsibility”? I answer no. Or, to be more precise, I argue that such notions may become philosophically acceptable only after crucial metaphysical issues have been addressed. My main claim, in sum, is that “artificial moral responsibility” betokens moral responsibility to the same degree that a “fake orgasm” betokens an orgasm.

Zusammenfassung Die rasanten Entwicklungen im Bereich der Künstlichen Intelligenz und Robotik stellen nicht nur die Ethik, sondern auch das Recht vor neue Herausforderungen, gerade im Bereich der Medizin und Pflege. Grundsätzlich hat der Einsatz von KI dabei das Potenzial, sowohl die Heilbehandlungen als auch den adäquaten Umgang im Rahmen der Pflege zu erleichtern, wenn nicht sogar zu verbessern. Verwaltungsaufgaben, die Überwachung von Vitalfunktionen und deren Parameter sowie die Untersuchung von Gewebeproben etwa könnten autonom ablaufen. In Diagnostik und Therapie können Systeme (...) die behandelnde Ärztin unterstützen. Intelligente Betten ermöglichen eine Frühmobilisierung der Patient:innen bei gleichzeitig geringerem Personalaufwand. Gleichwohl birgt der Einsatz der Systeme rechtliche Herausforderungen. So besteht das Risiko einer Verletzung der beteiligten Personen. Im Gegensatz zu herkömmlichen Technologien „leidet“ KI unter der „Black-Box-Problematik“: Die von den Systemen generierten Ergebnisse sind nicht mehr vollständig vorhersehbar und nachvollziehbar. Der Einsatz birgt unbekannte und unkalkulierbare Risiken, was sich insbesondere auf die zivilrechtliche Haftung und strafrechtliche Verantwortung auswirkt. Wem die Entscheidungen der Systeme normativ zuzurechnen sind, ist eine Kernfrage des juristischen Diskurses. Die aus praktikabilitätsgründen naheliegende Wahl, dem letztentscheidenden Menschen das Verhalten eines KI-Systems zuzurechnen, überzeugt nicht in allen Fällen, sondern degradiert ihn häufig zum symbolischen Haftungsknecht und legt ihm einseitig die Risiken der Technologien auf. Weiterhin ergeben sich im Bereich der Medizin und Pflege Fragen hinsichtlich der Zulassung von KI-Systemen, da die Maschinen während der Nutzung weiterlernen und so ihren strukturellen Aufbau kontinuierlich verändern. Es ist daher angebracht, sich frühzeitig mit dem Konfliktpotential aus ethischer und rechtlicher Sicht auseinander zu setzen, um einer potenziellen gesellschaftlichen Angst vor derartigen Systemen vorzubeugen und einen praxisgerechten Handlungsrahmen zu schaffen. (shrink)