The paper presents a framework to realise “meaningful human control” over Automated Driving Systems. The framework is based on an original synthesis of the results of the multidisciplinary research project “Meaningful Human Control over Automated Driving Systems” lead by a team of engineers, philosophers, and psychologists at Delft University of the Technology from 2017 to 2021. Meaningful human control aims at protecting safety and reducing responsibility gaps. The framework is based on the core assumption that human (...) persons and institutions, not hardware and software and their algorithms, should remain ultimately—though not necessarily directly—in control of, and thus morally responsible for, the potentially dangerous operation of driving in mixed traffic. We propose an Automated Driving System to be under meaningful human control if it behaves according to the relevant reasons of the relevant human actors (tracking), and that any potentially dangerous event can be related to a human actor (tracing). We operationalise the requirements for meaningful human control through multidisciplinary work in philosophy, behavioural psychology and traffic engineering. The tracking condition is operationalised via a proximal scale of reasons and the tracing condition via an evaluation cascade table. We review the implications and requirements for the behaviour and skills of human actors, in particular related to supervisory control and driver education. We show how the evaluation cascade table can be applied in concrete engineering use cases in combination with the definition of core components to expose deficiencies in traceability, thereby avoiding so-called responsibility gaps. Future research directions are proposed to expand the philosophical framework and use cases, supervisory control and driver education, real-world pilots and institutional embedding. (shrink)

Partially automated vehicle technology is increasingly common on-road. While this technology can provide safety benefits to drivers, it also introduces new concerns about driver attention. In particular, during partially automated driving, drivers are expected to stay vigilant so they can readily respond to important events in their environment. However, using partially automated vehicles on the highway places drivers in monotonous situations and requires them to do very little. This can place the driver in a state of (...) cognitive underload in which they experience a very small amount of cognitive demand. In this situation, drivers can exhibit vigilance decrements which impact their ability to respond to on-road threats. This is of particular concern in situations when the partially automated vehicle fails to respond to a potentially critical situation and leaves all responsibility to safely navigate to the driver. This paper reviews situations that lead to vigilance decrements and characterizes the different methodologies of measuring driver vigilance during PAD, highlighting their advantages and limitations. Based on our reading of the literature, we summarize several factors future research on vigilance decrements in PAD should consider. (shrink)

The ethics of autonomous cars and automated driving have been a subject of discussion in research for a number of years :28–58, 2016). As levels of automation progress, with partially automated driving already becoming standard in new cars from a number of manufacturers, the question of ethical and legal standards becomes virulent. For exam-ple, while automated and autonomous cars, being equipped with appropriate detection sensors, processors, and intelligent mapping material, have a chance of being much (...) safer than human-driven cars in many regards, situations will arise in which accidents cannot be completely avoided. Such situations will have to be dealt with when programming the software of these vehicles. In several instances, internationally, regulations have been passed, based on legal considerations of road safety, mostly. However, to date, there have been few, if any, cases of a broader ethics code for autonomous or automated driving preceding actual regulation and being based on a broadly composed ethics committee of independent experts. In July 2016, the German Federal Minister of Transport and Digital Infrastructure, Alexander Dobrindt, appointed a national ethics committee for automated and connected driving, which began its work in September 2016. In June 2017, this committee presented a code of ethics which was published in German and in English. It consists of 20 ethical guidelines. Having been a member of this committee, I will present the main ethical topics of these guidelines and the discussions that lay behind them. (shrink)

In this paper, we discuss the ethics of automated driving. More specifically, we discuss responsible human-robot coordination within mixed traffic: i.e. traffic involving both automated cars and conventional human-driven cars. We do three main things. First, we explain key differences in robotic and human agency and expectation-forming mechanisms that are likely to give rise to compatibility-problems in mixed traffic, which may lead to crashes and accidents. Second, we identify three possible solution-strategies for achieving better human-robot coordination within (...) mixed traffic. Third, we identify important ethical challenges raised by each of these three possible strategies for achieving optimized human-robot cordination in this domain. Among other things, we argue that we should not just explore ways of making robotic driving more like human driving. Rather, we ought also to take seriously potential ways of making human driving more like robotic driving. Nor should we assume that complete automation is always the ideal to aim for; in some traffic-situations, the best results may be achieved through human-robot collaboration. Ultimately, our main aim in this paper is to argue that the new field of the ethics of automated driving needs take seriously the ethics of mixed traffic and responsible human-robot coordination. (shrink)

This book offers a systematic and thorough philosophical analysis of the ways in which driving automation crosses path with ethical values. Upon introducing the different forms of driving automation and examining their relation to human autonomy, it provides readers with in-depth reflections on safety, privacy, moral judgment, control, responsibility, sustainability, and other ethical issues. Driving is undoubtedly a moral activity as a human act. Transferring it to artificial agents such as connected and automated vehicles necessarily raises (...) many philosophical questions. When driving is automated, what happens to its ethical dimensions? Could artificial agents accomplish ethical objectives on our behalf, take moral decisions in our place, and drive us into a more ethical transportation future? In doing so, would they be “moral” as we are or in a way that is similar to, but also remarkably different from, our own? And what role is yet to be played by human responsibility and commitment? The book addresses these questions with the aim of stimulating an interdisciplinary dialogue between different stakeholders. They include automotive engineers, computer scientists, and moral philosophers, as well as industry representatives, policymakers, regulators, transportation experts, and the general public. Indeed, connected and automated vehicles will not take the high road for us . We must drive them there. (shrink)

In this paper, I will argue that automated vehicles should not swerve to avoid a person or vehicle in its path, unless they can do so without imposing risks onto others. I will argue that this is the conclusion that we should reach even if we start by assuming that we should divert the trolley in the standard trolley case. In defence of this claim, I appeal to the distribution of moral and legal responsibilities, highlighting the importance of safe (...) spaces, and arguing in favour of constraints on what can be done to minimise casualties. My arguments draw on the methodology associated with the trolley problem. As such, this paper also defends this methodology, highlighting a number of ways in which authors misunderstand and misrepresent the trolley problem. For example, the ‘trolley problem’ is not the ‘name given by philosophers to classic examples of unavoidable crash scenarios, historically involving runaway trolleys’, as Millar suggests, and trolley cases should not be compared with ‘model building in the sciences’, as Gogoll and Müller suggest. Trolley cases have more in common with lab experiments than model building, and the problem referred to in the trolley problem is not the problem of deciding what to do in any one case. Rather, it refers to the problem of explaining what appear to be conflicting intuitions when we consider two cases together. The problem, for example, could be: how do we justify the claim that automated vehicles should not swerve even if we accept the claim that we should divert the trolley in an apparently similar trolley case? (shrink)

We motivate the desire for self-driving and explain its potential and limitations, and explore the need for—and potential implementation of—morals, ethics, and other value systems as complementary “capabilities” to the Deep Technologies behind self-driving. We consider how the incorporation of such systems may drive or slow adoption of high automation within vehicles. First, we explore the role for morals, ethics, and other value systems in self-driving through a representative hypothetical dilemma faced by a self-driving car. Through (...) the lens of engineering, we explain in simple terms common moral and ethical frameworks including utilitarianism, deontology, and virtue ethics before characterizing their relationship to the fundamental algorithms enabling self-driving. The concepts of behavior cloning, state-based modeling, and reinforcement learning are introduced, with some algorithms being more suitable for the implementation of value systems than others. We touch upon the contemporary cross-disciplinary landscape of morals and ethics in self-driving systems from a joint philosophical and technical perspective, and close with considerations for practitioners and the public, particularly as individuals may not appreciate the nuance and complexity of using imperfect information to navigate diverse scenarios and tough-to-quantify value systems, while “typical” software development reduces complex problems to black and white decision-making. (shrink)

Many ethicists writing about automated systems attribute agency to these systems. Not only that; they seemingly attribute an autonomous or independent form of agency to these machines. This leads some ethicists to worry about responsibility-gaps and retribution-gaps in cases where automated systems harm or kill human beings. In this paper, I consider what sorts of agency it makes sense to attribute to most current forms of automated systems, in particular automated cars and military robots. I argue (...) that whereas it indeed makes sense to attribute different forms of fairly sophisticated agency to these machines, we ought not to regard them as acting on their own, independently of any human beings. Rather, the right way to understand the agency exercised by these machines is in terms of human–robot collaborations, where the humans involved initiate, supervise, and manage the agency of their robotic collaborators. This means, I argue, that there is much less room for justified worries about responsibility-gaps and retribution-gaps than many ethicists think. (shrink)

Artificial intelligence is poised to eliminate millions of jobs, from finance to truck driving. But artisanal products are valued precisely because of their human origins, and thus have some inherent “immunity” from AI job loss. At the same time, artisanal labor, combined with technology, could potentially help to democratize the economy, allowing independent, small-scale businesses to flourish. Could AI, robotics and related automation technologies enhance the economic viability and environmental sustainability of these beloved crafting professions, perhaps even expanding their (...) niche to replace some job loss in other sectors? In this paper, we compare the problems created by the current mass production economy and potential solutions from an artisanal economy. In doing so, the paper details the possibilities of utilizing AI to support hybrid forms of human–machine production at the microscale; localized and sustainable value chains at the mesoscale; and networks of these localized and sustainable producers at the macroscale. In short, a wide range of automation technologies are potentially available for facilitating and empowering an artisanal economy. Ultimately, it is our hope that this paper will facilitate a discussion on a future vision for more “generative” economic forms in which labor value, ecological value and social value can circulate without extraction or alienation. (shrink)

This book reports on theoretical and practical analyses of the ethical challenges connected to driving automation. It also aims at discussing issues that have arisen from the European Commission 2020 report “Ethics of Connected and Automated Vehicles. Recommendations on Road Safety, Privacy, Fairness, Explainability and Responsibility”. Gathering contributions by philosophers, social scientists, mechanical engineers, and UI designers, the book discusses key ethical concerns relating to responsibility and personal autonomy, privacy, safety, and cybersecurity, as well as explainability and human-machine (...) interaction. On the one hand, it examines these issues from a theoretical, normative point of view. On the other hand, it proposes practical strategies to face the most urgent ethical problems, showing how the integration of ethics and technology can be achieved through design practices. All in all, this book fosters a multidisciplinary approach where philosophy, ethics, and engineering are integrated, rather than just juxtaposed. It is meant to inform and inspire an audience of philosophers of technology, ethicists, engineers, developers, manufacturers, and regulators, among other interested readers. (shrink)

We address the considerations of the European Commission Expert Group on the ethics of connected and automated vehicles regarding data provision in the event of collisions. While human drivers’ appropriate post-collision behavior is clearly defined, regulations for automated driving do not provide for collision detection. We agree it is important to systematically incorporate citizens’ intuitions into the discourse on the ethics of automated vehicles. Therefore, we investigate whether people expect automated vehicles to behave like humans (...) after an accident, even if this behavior does not directly affect the consequences of the accident. We find that appropriate post-collision behavior substantially influences people’s evaluation of the underlying crash scenario. Moreover, people clearly think that automated vehicles can and should record the accident, stop at the site, and call the police. They are even willing to pay for technological features that enable post-collision behavior. Our study might begin a research program on post-collision behavior, enriching the empirically informed study of automated driving ethics that so far exclusively focuses on pre-collision behavior. (shrink)

This work considers the future of driving in terms of both its short- and long-term horizons. It conjectures that human-controlled driving will follow in the footsteps of a wide swath of other, now either residual or abandoned human occupations. Pursuits that have preceded it into oblivion. In this way, driving will dwindle down into only a few niche locales wherein enthusiasts will still persist, much in the way that steam train hobbyists now continue their own aspirational inclinations. (...) Of course, the value of any such prognostication is in direct proportion to the degree that information is conveyed, and prospective uncertainty reduced. In more colloquial terms: the devil is in the details of these coming transitions. It is anticipated that we will see a progressive transformation of the composition of on-road traffic that will be registered most immediately in the realm of professional transportation in which the imperative for optimization exceeds that in virtually all other user segments. The transition from manual control to full automation will be more punctate than gradualist in is evolutionary development. As performance optimization slowly exhausts the commercial sector, it will progressively transition more into the discretionary realm by dint of simple technology transfer alone. The hedonic dimension of everyday driving will be dispersed and pursued by progressively fewer individuals. The traveling window of generational expectation will soon mean that human driving will be largely ‘forgotten,’ as each sequential generation matures without this, still presently common, experience. Indications of this stage of progress are beginning to... (shrink)

In 2017, the German ethics commission for automated and connected driving released 20 ethical guidelines for autonomous vehicles. It is now up to the research and industrial sectors to enhance the development of autonomous vehicles based on such guidelines. In the current state of the art, we find studies on how ethical theories can be integrated. To the best of the authors’ knowledge, no framework for motion planning has yet been published which allows for the true implementation of (...) any practical ethical policies. This paper makes four contributions: Firstly, we briefly present the state of the art based on recent works concerning unavoidable accidents of autonomous vehicles (AVs) and identify further need for research. While most of the research focuses on decision strategies in moral dilemmas or crash optimization, we aim to develop an ethical trajectory planning for all situations on public roads. Secondly, we discuss several ethical theories and argue for the adoption of the theory “ethics of risk.” Thirdly, we propose a new framework for trajectory planning, with uncertainties and an assessment of risks. In this framework, we transform ethical specifications into mathematical equations and thus create the basis for the programming of an ethical trajectory. We present a risk cost function for trajectory planning that considers minimization of the overall risk, priority for the worst-off and equal treatment of people. Finally, we build a connection between the widely discussed trolley problem and our proposed framework. (shrink)

In the near future, vehicles will gradually gain more autonomous functionalities. Drivers’ activity will be less about driving than about monitoring intelligent systems to which driving action will be delegated. Road safety, therefore, remains dependent on the human factor and we should identify the limits beyond which driver’s functional state (DFS) may no longer be able to ensure safety. Depending on the level of automation, estimating the DFS may have different targets, e.g. assessing driver’s situation awareness in lower (...) levels of automation and his ability to respond to emerging hazard or assessing driver’s ability to monitor the vehicle performing operational tasks in higher levels of automation. Unfitted DFS (e.g., drowsiness) may impact the driver ability respond to taking over abilities. This paper reviews the most appropriate psychophysiological indices in naturalistic driving while considering the DFS through exogenous sensors, providing the more efficient trade-off between reliability and intrusiveness. The DFS also originates from kinematic data of the vehicle, thus providing information that indirectly relates to drivers behaviour. The whole data should be synchronously processed, providing a diagnosis on the DFS, and bringing it to the attention of the decision maker in real time. Next, making the information available can be permanent or intermittent (or even undelivered), and may also depend on the automation level. Such interface can include recommendations for decision support or simply give neutral instruction. Mapping of relevant psychophysiological and behavioural indicators for DFS will enable practitioners and researchers provide reliable estimates, fitted to the level of automation. (shrink)

The aim of this chapter is to identify the most representative measures of subjective and objective mental workload in driving, and to understand how the subjective and objective levels of mental workload influence the performance as a function of situation complexity and driving experience, i.e. to verify whether the increase of situation complexity and the lack of experience increase the subjective and physiological levels of mental workload and lead to driving performance impairments. This chapter will be useful (...) to carry out an experimental study and to improve the driver training. In the first part, we will broach the theoretical approach with two factors of mental workload and performance, i.e. situation complexity and driving experience. Indeed, a low complex situation (e.g., highways), or conversely a high complex situation (e.g., town) can provoke an overload. Additionally, performing the driving tasks implies producing a high effort for novice drivers who have not totally automated the driving activity. In the second part, we will focus on subjective measures of mental workload. A comparison of questionnaires usually used in driving will allow identifying the most appropriate ones as a function of different criteria. Moreover, we will review the empirical studies to verify if the subjective level of mental workload is high in simple and very complex situations, especially for novice drivers compared to the experienced ones. In the third part, we will focus on physiological measures. A comparison of physiological indicators will be realized in order to identify the most correlated to mental workload. An empirical review will also take the effect of situation complexity and experience on these physiological indicators into consideration. Finally, a more nuanced comparison between subjective and physiological measures will be established from the impact on situation complexity and experience. (shrink)

In this paper, we explore and describe what is needed to allow connected and automated vehicles to break traffic rules in order to minimise road safety risk and to operate with appropriate transparency. Reviewing current traffic rules with particular reference to two driving situations, we illustrate why current traffic rules are not suitable for CAVs and why making new traffic rules specifically for CAVs would be inappropriate. In defining an alternative approach to achieving safe CAV driving behaviours, (...) we describe the use of ethical goal functions as part of hybrid AI systems, suggesting that functions should be defined by governmental bodies with input from citizens and stakeholders. Ethical goal functions for CAVs would enable developers to optimise driving behaviours for safety under conditions of uncertainty whilst allowing for differentiation of products according to brand values. Such functions can differ between regions according to preferences for safety behaviours within that region and can be updated over time, responding to continual socio-technological feedback loops. We conclude that defining ethical goal functions is an urgent and necessary step from governmental bodies to enable the safe and transparent operation of CAVs and accelerate the reduction in road casualties they promise to achieve. (shrink)

Karl Marx took technological development to be the heart of capitalism’s drive and, ultimately, its undoing. Machines are initially engineered to perform functions that otherwise would be performed by human workers. The economic logic pushed to its limits leads to the prospect of full automation: a world in which all labor required to meet human needs is superseded and performed by machines. To explore the future of automation, the paper considers a specific point of resemblance between human beings and machines: (...) intelligence. Examining the development of machine intelligence through the Marxist concepts of alienation and reification reveals a tension between certain technophilic post-labor visions and the reality of capitalistic development oriented towards intelligent technology. If the prospect of a post-labor world depends on technologies that closely resemble humans, the world can no longer be described as post-labor. The tension has implications for the potential moral status of machines and the possibility of full automation. The paper considers these implications by outlining four possible futures of automation. (shrink)

Automated vehicles (AVs) are expected to operate on public roads, together with non-automated vehicles and other road users such as pedestrians or bicycles. Recent ethical reports and guidelines raise worries that AVs will introduce injustice or reinforce existing social inequalities in road traffic. One major injustice concern in today’s traffic is that different types of road users are exposed differently to risks of corporal harm. In the first part of the paper, we discuss the responsibility of AV developers (...) to address existing injustice concerns regarding risk exposure as well as approaches on how to fulfill the responsibility for a fairer distribution of risk. In contrast to popular approaches on the ethics of risk distribution in unavoidable accident cases, we focus on low and moderate risk situations, referred to as routine driving. For routine driving, the obligation to distribute risks fairly must be discussed in the context of risk-taking and risk-acceptance, balancing safety objectives of occupants and other road users with driving utility. In the second part of the paper, we present a typical architecture for decentralized automated driving which contains a dedicated module for real-time risk estimation and management. We examine how risk estimation modules can be adjusted and parameterized to redress some inequalities. (shrink)

The use of automation in cars is increasing. In future vehicles, drivers will no longer be in charge of the main driving task and may be allowed to perform a secondary task. However, they might be requested to regain control of the car if a hazardous situation occurs. Performing a secondary task might increase drivers' mental workload and consequently decrease the takeover performance if the workload level exceeds a certain threshold. Knowledge about the driver's mental state might hence be (...) useful for increasing safety in conditionally automated vehicles. Measuring drivers' workload continuously is essential to support the driver and hence limit the number of accidents in takeover situations. This goal can be achieved using machine learning techniques to evaluate and classify the drivers' workload in real-time. To evaluate the usefulness of physiological data as an indicator for workload in conditionally automated driving, three physiological signals from 90 subjects were collected during 25 min of automated driving in a fixed-base simulator. Half of the participants performed a verbal cognitive task to induce mental workload while the other half only had to monitor the environment of the car. Three classifiers, sensor fusion and levels of data segmentation were compared. Results show that the best model was able to successfully classify the condition of the driver with an accuracy of 95%. In some cases, the model benefited from sensors' fusion. Increasing the segmentation level increased the performance of the model for windows smaller than 4 min, but decreased for windows larger than 4 min. In conclusion, the study showed that a high level of drivers' mental workload can be accurately detected while driving in conditional automation based on 4-min recordings of respiration and skin conductance. (shrink)

A human driver and an automated driving system might share control of automated vehicles in the near future. This raises many concerns associated with the assignment of responsibility for negative outcomes caused by them; one is that the human driver might be required to bear the brunt of moral and legal responsibilities. The psychological consequences of responsibility misattribution have not yet been examined. We designed a hypothetical crash similar to Uber’s 2018 fatal crash. We incorporated five legal (...) responsibility attributions. Participants chose their preferred liability attribution and then were randomly assigned into one of the five actual liability attribution conditions. They then responded to a series of questions concerning liability assignment, the crash, and AVs. Slightly more than 50% of participants thought that the human driver should bear full or primary liability. Legal responsibility misattribution negatively influenced these mentioned responses, regardless of overly attributing human or manufacturer liability. Overly attributing human liability had more negative influences. Improper liability attribution might hinder the adoption of AVs. Public opinion should not be ignored in developing a legal framework for AVs. (shrink)

The issue of how to identify the liability of subjects after a traffic accident takes place remains a puzzle regarding the SAE classification system. The SAE system is not good at dealing with the problem of responsibility evaluation; therefore, building a new classification system for self-driving cars from the perspective of the subject's liability is a possible way to solve this problem. This new system divides automated driving into three levels: i) assisted driving based on the (...) will of drivers, ii) automated driving based on the will of the manufacturers, and iii) fully automated driving based on social will. The corresponding responsible subjects for level one, level two, and level three are the drivers, the manufacturers, and society, respectively. (shrink)

An emerging discourse about automated agricultural machinery imagines farms as places where farmers and workers do not need to be, but also implicitly frames farms as intolerable places where people do not want to be. Only autonomous machines, this story goes, can relieve farmers and workers of this presumed burden by letting them ‘farm at a distance’. In return for this distanced autonomy, farmers are promised increased control over their work-life balance and greater farm productivity from letting ‘smart’ robots (...) assume control over the operational environment. Drawing upon the ways that these machines are promoted by manufacturers in various media, we trace the nascent contours of what we term a liberatory sociotechnical imaginary for agricultural automation across three cases—automated milking systems, self-driving tractors, and robotic strawberry pickers. We show how promises for new freedoms and autonomy are flexibly deployed to respond to distinct frictions that farmers, workers, and even farm animals experience in different modes of industrial agriculture. However, underlying these promises is the purposefully understated self-interest of manufacturers, who stand to gain further control over farms if automated technologies assume a central role in agriculture. Through the liberatory rhetoric, we contend, the imaginary seeks to enroll farmers into a socio-technical network that creates new relations of dependence upon the companies who design, sell, maintain, and often retain ownership over automated technologies. While potentially powerful, this imaginary may nonetheless fail to coalesce as farmers, workers, and agroecosystems exert their own agency on automated imaginaries and technological futures for agriculture. (shrink)

Self-driving cars currently face a lot of technological problems that need to be solved before the cars can be widely used. However, they also face ethical problems, among which the question of crash-optimization algorithms is most prominently discussed. Reviewing current debates about whether we should use the ethics of the Trolley Dilemma as a guide towards designing self-driving cars will provide us with insights about what exactly ethical research does. It will result in the view that although we (...) need the ethics of the Trolley Dilemma as important input for self-driving cars, the route towards simply implementing it into automated cars is blocked. (shrink)

With the rapid development of sensor technology for automated driving applications, the fusion, analysis, and application of multimodal data have become the main focus of different scenarios, especially in the development of mobile edge computing technology that provides more efficient algorithms for realizing the various application scenarios. In the present paper, the vehicle status and operation data were acquired by vehicle-borne and roadside units of electronic registration identification of motor vehicles. In addition, a motion model and an identification (...) system for the single-vehicle lane-change process were established by mobile edge computing and self-organizing feature mapping. Two scenarios were modeled and tested: lane change with no vehicles in the target lane and lane change with vehicles in the target lane. It was found that the proposed method successfully identified the stochastic parameters in the process of driving trajectory simulation, and the standard deviation between simulation and the measured results obeyed a normal distribution. The proposed methods can provide significant practical information for improving the data processing efficiency in automated driving applications, for solving single-vehicle lane-change applications, and for promoting the formation of a closed loop from sensing to service. (shrink)

Accidents involving autonomous vehicles raise difficult ethical dilemmas and legal issues. It has been argued that self-driving cars should be programmed to kill, that is, they should be equipped with pre-programmed approaches to the choice of what lives to sacrifice when losses are inevitable. Here we shall explore a different approach, namely, giving the user/passenger the task of deciding what ethical approach should be taken by AVs in unavoidable accident scenarios. We thus assume that AVs are equipped with what (...) we call an “Ethical Knob”, a device enabling passengers to ethically customise their AVs, namely, to choose between different settings corresponding to different moral approaches or principles. Accordingly, AVs would be entrusted with implementing users’ ethical choices, while manufacturers/programmers would be tasked with enabling the user’s choice and ensuring implementation by the AV. (shrink)

To operate in an unpredictable environment, a vehicle with advanced driving assistance systems, such as a robot or a drone, not only needs to register its surroundings but also to combine data from different sensors into a world model, for which it employs filter algorithms. Such world models, as this article argues with reference to the SLAM problem in robotics, consist of nothing other than probabilities about states and events arising in the environment. The model, thus, contains a virtuality (...) of possible worlds that are the basis for adaptive behavior. The article shows that the current development of these technologies requires new concepts because their complex adaptive behaviors cannot be explained by referring them to mere algorithmic processes. Instead, it proposes the heuristic instrument of microdecisions to designate the temporality of decisions between alternatives that are created by probabilistic procedures of world modeling. Microdecisions are more than the implementation of deterministic processes—they decide between possibilities and, thus, always open up the potential of their otherness. By describing autonomous adaptive technologies with this heuristic, the question of sovereignty inevitably arises. It forces us to re-think what autonomy means when decisions can be automated. (shrink)

As driving functions become increasingly automated, motorists run the risk of becoming cognitively removed from the driving process. Psychophysiological measures may provide added value not captured through behavioral or self-report measures alone. This paper provides a selective review of the psychophysiological measures that can be utilized to assess cognitive states in real-world driving environments. First, the importance of psychophysiological measures within the context of traffic safety is discussed. Next, the most commonly used physiology-based indices of cognitive (...) states are considered as potential candidates relevant for driving research. These include: electroencephalography and event-related potentials, optical imaging, heart rate and heart rate variability, blood pressure, skin conductance, electromyography, thermal imaging, and pupillometry. For each of these measures, an overview is provided, followed by a discussion of the methods for measuring it in a driving context. Drawing from recent empirical driving and psychophysiology research, the relative strengths and limitations of each measure are discussed to highlight each measures’ unique value. Challenges and recommendations for valid and reliable quantification from lab to (less predictable) real-world driving settings are considered. Finally, we discuss measures that may be better candidates for a near real-time assessment of motorists’ cognitive states that can be utilized in applied settings outside the lab. This review synthesizes the literature on in-vehicle psychophysiological measures to advance the development of effective human-machine driving interfaces and driver support systems. (shrink)

In this article we outline two different ways of ‘seeing’ autonomous driving (AD) cars. The first corresponds with the technological innovation narrative, published in online industry, policy, business and other news contexts, that pitches AD cars as the solution to societal problems, and urges users to trust and accept them so that such benefits can be accrued. The second is a narrative of everyday improvisation, which was visualized through our video ethnography and participant mapping exercises. Our research, undertaken in (...) Sweden, involved possible future everyday users of AD cars. We argue for a research and intervention agenda that examines how the visual narration of how AD cars might participate in human futures, could be shifted to create new modes of trust and reassurance for publics. (shrink)

"Golf day 2005 @ federal golf club, red hill." Ethos: Official Publication of the Law Society of the Australian Capital Territory, (196), pp. 7.

Conditional automated driving [level 3, Society of Automotive Engineers ] requires drivers to take over the vehicle when an automated system’s failure occurs or is about to leave its operational design domain. Two-stage warning systems, which warn drivers in two steps, can be a promising method to guide drivers in preparing for the takeover. However, the proper time intervals of two-stage warning systems that allow drivers with different personalities to prepare for the takeover remain unclear. This study (...) explored the optimal time intervals of two-stage warning systems with insights into the drivers’ neuroticism personality. A total of 32 drivers were distributed into two groups according to their self-ratings in neuroticism. Each driver experienced takeover under the two-stage warning systems with four time intervals. The takeover performance and subjective opinions for time intervals and situation awareness were recorded. The results showed that drivers in the 5-s time interval had the best takeover preparation. Furthermore, both the 5- and 7-s time intervals resulted in more rapid takeover reactions and were rated more appropriate and useful than the 3- and 9-s time intervals. In terms of personality, drivers with high neuroticism tended to take over immediately after receiving takeover messages, at the cost of SA deficiency. In contrast, drivers with low neuroticism responded safely by judging whether they gained enough SA. We concluded that the 5-s time interval was optimal for drivers in two-stage takeover warning systems. When considering personality, drivers with low neuroticism had no strict requirements for time intervals. However, the extended time intervals were favorable for drivers with high neuroticism in developing SA. The present findings have reference implications for designers and engineers to set the time intervals of two-stage warning systems according to the neuroticism personality of drivers. (shrink)

In this paper, in line with the general framework of value-sensitive design, we aim to operationalize the general concept of “Meaningful Human Control” in order to pave the way for its translation into more specific design requirements. In particular, we focus on the operationalization of the first of the two conditions investigated: the so-called ‘tracking’ condition. Our investigation is led in relation to one specific subcase of automated system: dual-mode driving systems. First, we connect and compare meaningful human (...) control with a concept of control very popular in engineering and traffic psychology, and we explain to what extent tracking resembles and differs from it. This will help clarifying the extent to which the idea of meaningful human control is connected to, but also goes beyond, current notions of control in engineering and psychology. Second, we take the systematic analysis of practical reasoning as traditionally presented in the philosophy of human action and we adapt it to offer a general framework where different types of reasons and agents are identified according to their relation to an automated system’s behaviour. This framework is meant to help explaining what reasons and what agents play a role in controlling a given system, thereby enabling policy makers to produce usable guidelines and engineers to design systems that properly respond to selected human reasons. In the final part, we discuss a practical example of how our framework could be employed in designing automated driving systems. (shrink)

To be successful, policy must anticipate a broad range of constituents. Yet, all too often, technology policy is written with primarily mainstream populations in mind. In this article, drawing on Value Sensitive Design and discount evaluation methods, we introduce a new method—Diverse Voices—for strengthening pre-publication technology policy documents from the perspective of underrepresented groups. Cost effective and high impact, the Diverse Voices method intervenes by soliciting input from “experiential” expert panels. We first describe the method. Then we report on two (...) case studies demonstrating its use: one with a white paper on augmented reality technology with expert panels on people with disabilities, people who were formerly or currently incarcerated, and women; and the other with a strategy document on automated driving vehicle technologies with expert panels on youth, non-car drivers, and extremely low-income people. In both case studies, panels identified significant shortcomings in the pre-publication documents which, if addressed, would mitigate some of the disparate impact of the proposed policy recommendations on these particular stakeholder groups. Our discussion includes reflection on the method, evidence for its success, its limitations, and future directions. (shrink)

The automobile is a tool like no other. There is much excitement and enthusiasm for new and emerging transportation tools such as vehicle automation and driver assistance systems. Although there are high hopes for these technologies, there are many unknowns including the extent to which these new transportation tools can realistically and reliably improve driver safety and affect the subjective driving experience. This paper explores these questions in the context of evaluating collision warning systems on the behavior and perceptions (...) of drivers with different amounts of experience and places the results in the greater context of tool use and skilled learning focusing on the adoption and utilization of mental models. (shrink)

Machine vision is one of the main applications of artificial intelligence. In China, the machine vision industry makes up more than a third of the national AI market, and technologies like face recognition, object tracking and automated driving play a central role in surveillance systems and social governance projects relying on the large-scale collection and processing of sensor data. Like other novel articulations of technology and society, machine vision is defined, developed and explained by different actors through the (...) work of imagination. In this article, we draw on the concept of sociotechnical imaginaries to understand how Chinese companies represent machine vision. Through a qualitative multimodal analysis of the corporate websites of leading industry players, we identify a cohesive sociotechnical imaginary of machine vision, and explain how four distinct visual registers contribute to its articulation. These four registers, which we call computational abstraction, human–machine coordination, smooth everyday, and dashboard realism, allow Chinese tech companies to articulate their global ambitions and competitiveness through narrow and opaque representations of machine vision technologies. (shrink)

The interaction of automated vehicles with vulnerable road users is one of the greatest challenges in the development of automated driving functions. In order to improve efficiency and ensure the safety of mixed traffic, ADF need to understand the intention of vulnerable road users, to adapt to their driving behavior, and to show its intention. However, this communication may occur in an implicit way, meaning they may communicate with vulnerable road users by using dynamic information, such (...) as speed, distance, etc. Therefore, investigating patterns of implicit communication of human drivers with vulnerable road users is relevant for developing ADF. The aim of this study is to identify the patterns of implicit communication of human drivers with vulnerable road users. For this purpose, the interaction between right-turning motorists and crossing cyclists was investigated at a traffic light controlled urban intersection. In the scenario, motorists and cyclists had a green signal at the same time, but cyclist had right-of-way. Using the Application Platform for Intelligent Mobility Research Intersection, trajectory and video data were recorded at an intersection in Braunschweig, Germany. Data had been recorded for 4 weeks. Based on the criticality metric post-encroachment time and quality of the recorded trajectory, 206 cases of interaction were selected for further analyses. According to the video annotation, when approaching the intersection, three common communication patterns were identified: no yield, motorists, who should yield to cyclists, crossed the intersection first while forcing right-of-way; active yield, motorists, who were in front of cyclists, gave the right-of-way; passive yield, motorists, who were behind cyclists, had to give the right-of-way. The analysis of the trajectory data revealed different patterns of changes in time advantage in these three categories. Additionally, the communication patterns were evaluated with regard to frequency of occurrence, efficiency, and safety. The findings of this study may provide knowledge for the implementation of a communication strategy for ADF, contributing to traffic efficiency as well as ensuring safety in the interaction with vulnerable road users. (shrink)

Riassunto: L’indagine sui fondamenti neurali del giudizio morale è uno dei principali ed attuali temi di ricerca della Neuroscienza, il quale si intreccia inevitabilmente con tematiche relative all’Intelligenza Artificiale, al futuro dei trasporti e alla Filosofia della Mente. Gli esseri umani sono naturalmente dotati di un innato senso della morale, il quale è governato dalle intuizioni, ma sono anche provvisti di alcuni principi razionali. Il giudizio e il comportamento morale sono il risultato dell’integrazione fra le emozioni e i processi razionali, (...) proprio come i processi cognitivi erano una combinazione di istinto e pura computazione. Nella parte finale di questo lavoro, ho preso in considerazione i problemi di natura morale derivanti dall’introduzione dei veicoli a guida autonoma, i quali si presentano come un’applicazione diretta del problema del carrello : come dovrebbe essere programmato un veicolo per comportarsi nel caso di un incidente inevitabile, nel quale deve scegliere tra due mali? Parole chiave: Processi decisionali; Neuroscienza; Giudizi morali; Prospettiva comparata; Problema del carrello Brains and Trolleys: Formalizing Morality in Automated Driving Cars: Inquiry into the neural bases of moral judgment is one of the current frontiers in neuroscientific research and is intertwined with issues in Artificial Intelligence, the future of transport, and Philosophy of Mind. Humans are naturally endowed with an innate moral sense, which is governed by intuition and informed by rational rules. Moral judgment and behavior are the byproduct of the integration of emotional and rational processes, just as cognitive processes are the result of a combination of emotional instinct and computational rationality. In the latter part of this work, I also consider the ethical issues raised by the introduction of Automated Driving Systems by reexamining the Trolley Problem : how should an autonomous vehicle be programmed to behave in the event of an unavoidable accident, in which it has to choose between two harmful consequences harms? Keywords: Decision-making; Neuroscience; Moral Judgments; Comparative Perspective; Trolley Problem. (shrink)

Background: While advanced driver assistance technologies have the potential to increase safety, there is concern that driver inattention resulting from overreliance on these features may result in crashes. Driver monitoring technologies to assess a driver’s state may be one solution. The purpose of this study was to replicate and extend the research on physiological responses to common driving hazards and examine how these may differ based on driving experience.Methods: Learner and Licensed drivers viewed a Driving Hazard Perception (...) Task while electrodermal activity was measured. The task presented 30 Event and 30 Non-Event videos. A skin conductance response score was calculated for each participant based on the percentage of videos that elicited an SCR.Results: Analysis of the SCR score during Event videos revealed a medium effect of group differences, whereby Licensed drivers were more likely to have an SCR than Learner drivers. Interaction effects revealed Licensed drivers were more likely to have an SCR earlier in the Event videos compared to the end, and the Learner drivers were more likely to have an SCR earlier in the Non-Event videos compared to the end.Conclusion: Our results support the viability of using SCR during driving videos as a marker of hazard anticipation differing based on experience. The interaction effects may illustrate situational awareness in licensed drivers and deficiencies in sustained vigilance among learner drivers. The findings demand further examination if physiological measures are to be validated as a tool to inform driver potential performance in an increasingly automated driving environment. (shrink)

Currently, the central questions in the philosophical debate surrounding the ethics of automated warfare are (1) Is the development and use of autonomous lethal robotic systems for military purposes consistent with (existing) international laws of war and received just war theory?; and (2) does the creation and use of such machines improve the moral caliber of modern warfare? However, both of these approaches have significant problems, and thus we need to start exploring alternative approaches. In this paper, I ask (...) whether autonomous robots ought to be programmed to be pacifists. The answer arrived at is “Yes”, if we decide to create autonomous robots, they ought to be pacifists. This is to say that robots ought not to be programmed to willingly and intentionally kill human beings, or, by extension, participate in or promote warfare, as something that predictably involves the killing of humans. Insofar as we are the ones that will be determining the content of the robot’s value system, then we ought to program robots to be pacifists, rather than ‘warists’. This is (in part) because we ought to be pacifists, and creating and programming machines to be “autonomous lethal robotic systems” directly violates this normative demand on us. There are no mitigating reasons to program lethal autonomous machines to contribute to or participate in warfare. Even if the use of autonomous lethal robotic systems could be consistent with received just war theory and the international laws of war, and even if their involvement could make warfare less inhumane in certain ways, these reasons do not compensate for the ubiquitous harms characteristic of modern warfare. In this paper, I provide four main reasons why autonomous robots ought to be pacifists, most of which do not depend on the truth of pacifism. The strong claim being argued for here is that automated warfare ought not to be pursued. The weaker claim being argued for here is that automated warfare ought not to be pursued, unless it is the most pacifist option available at the time, and other alternatives have been reasonably explored, and we are simultaneously promoting a (long term) pacifist agenda in (many) other ways. Thus, the more ambitious goal of this paper is to convince readers that automated warfare is something that we ought not to promote or pursue, while the more modest—and I suspect, more palatable—goal is to spark sustained critical discussion about the assumptions underlying the drive towards automated warfare, and to generate legitimate consideration of its pacifist alternatives,in theory, policy, and practice. (shrink)

This paper explores the ambiguous impact of new information and communications technologies on the cultivation of moral skills in human beings. Just as twentieth century advances in machine automation resulted in the economic devaluation of practical knowledge and skillsets historically cultivated by machinists, artisans, and other highly trained workers , while also driving the cultivation of new skills in a variety of engineering and white collar occupations, ICTs are also recognized as potential causes of a complex pattern of economic (...) deskilling, reskilling, and upskilling. In this paper, I adapt the conceptual apparatus of sociological debates over economic deskilling to illuminate a different potential for technological deskilling/upskilling, namely the ability of ICTs to contribute to the moral deskilling of human users, a potential that exists alongside rich but currently underrealized possibilities for moral reskilling and/or upskilling. I flesh out this general hypothesis by means of examples involving automated weapons technology, new media practices, and social robotics. I conclude that since moral skills are essential prerequisites for the effective development of practical wisdom and virtuous character, and since market and cultural forces are not presently aligned to bring about the more salutary of the ambiguous potentials presented here, the future shape of these developments warrants our close attention—and perhaps active intervention. (shrink)

Inspired by recent breakthroughs in predictive modeling, practitioners in both industry and government have turned to machine learning with hopes of operationalizing predictions to drive automated decisions. Unfortunately, many social desiderata concerning consequential decisions, such as justice or fairness, have no natural formulation within a purely predictive framework. In efforts to mitigate these problems, researchers have proposed a variety of metrics for quantifying deviations from various statistical parities that we might expect to observe in a fair world and offered (...) a variety of algorithms in attempts to satisfy subsets of these parities or to trade o the degree to which they are satised against utility. In this paper, we connect this approach to fair machine learning to the literature on ideal and non-ideal methodological approaches in political philosophy. The ideal approach requires positing the principles according to which a just world would operate. In the most straightforward application of ideal theory, one supports a proposed policy by arguing that it closes a discrepancy between the real and the perfectly just world. However, by failing to account for the mechanisms by which our non-ideal world arose, the responsibilities of various decision-makers, and the impacts of proposed policies, naive applications of ideal thinking can lead to misguided interventions. In this paper, we demonstrate a connection between the fair machine learning literature and the ideal approach in political philosophy, and argue that the increasingly apparent shortcomings of proposed fair machine learning algorithms reflect broader troubles faced by the ideal approach. We conclude with a critical discussion of the harms of misguided solutions, a reinterpretation of impossibility results, and directions for future research. (shrink)

In the contemporary debate on artificial morality, the trolley problem has found a new field of application, in the “ethics of crashes” with self-driving cars. The paper aims to show that the trolley dilemma is out of place, in the context of automated traffic, not only with regard to the object of the dilemma (which human being should be sacrificed, in crashes with inevitable fatal consequences), but also with regard to the subject to whom it is up to (...) decide. In States whose constitutional charters protect fundamental individual rights, laws contain definite constraints on solving dilemmas regarding self-driving cars. The idea that crashes of self-driving cars pose extraordinary moral questions, rather than safety, transparency, caution and control issues, as any other machine, derives perhaps from the human inclination to consider anthropomorphic objects as agents, or even as moral agents. The development of autonomous machines can lead to believe that three crowdsourcing and subrogation operations, variously intertwined, are possible and allowed: of law with ethics, in the first place; of moral evaluation, secondly, with a computational model of aggregated social preferences and, finally, of human moral agency with the autonomous, unpredictable and opaque functioning of machines. (shrink)

Focused on city-scale automation, and using self-driving cars (SDCs) as a case study, this article reflects on the role of AI—and in particular, computer vision systems used for mapping and navigation—as a catalyst for urban transformation. Urban research commonly presents AI and cities as having a one-way cause-and-effect relationship, giving undue weight to AI’s impact on cities and overlooking the role of cities in shaping AI. Working at the intersection of data science and social research, this paper aims to (...) counter this trend by exploring the reverse perspective: how do cities affect the development, and expose the present limits, of SDCs? The contribution of this paper is threefold. First, by comparing urban and nonurban environments and thoroughly examining the relationship between computer vision and city-specific sociality and form, it defines machine autonomy/automation as a function of the sociotechnical milieu in which an AI system operates. Second, and related, the paper problematizes the notion of SDCs as autonomous technologies and the role it plays in envisioning contending policy arrangements and technical solutions for achieving full driving automation. Finally, the article offers insight into a materialist and spatialized understanding of AI—namely, not as an abstract quality susceptible to replication within discrete machines, but rather as a distributed property emerging through embodied interactions among a multiplicity of agents (human, non-human, and technological) within/with their environments. (shrink)

Recent advances in automation technology have increased the opportunity for collaboration between humans and multiple autonomous systems such as robots and self-driving cars. In research on autonomous system collaboration, the trust users have in autonomous systems is an important topic. Previous research suggests that the trust built by observing a task can be transferred to other tasks. However, such research did not focus on trust in multiple different devices but in one device or several of the same devices. Thus, (...) we do not know how trust changes in an environment involving the operation of multiple different devices such as a construction site. We investigated whether trust can be transferred among multiple different devices, and investigated the effect of two factors: the similarity among multiple devices and the agency attributed to each device, on trust transfer among multiple devices. We found that the trust a user has in a device can be transferred to other devices and that attributing different agencies to each device can clarify the distinction among devices, preventing trust from transferring. (shrink)