II.A Preventing Accidents

Regulation is necessary to prevent accidents caused by malfunctioning robots and unforeseen interactive effects. Some of these rules might need to be backed up by sanctions. It is almost impossible to say much more on a general level about potential accidents and what should be prohibited or regulated to minimize the risk of harm, as a more detailed analysis would require covering a vast area. The exact nature of important “dos and don’ts” that might warrant enforcement by criminal laws obviously depends on the kinds of activities that robots perform, e.g., in manufacturing, transportation, healthcare, households, and warfare, and the potential risks involved. The more complex a robot’s task, the more that can go wrong. The kind and size of potential harm depends, among other things, on the physical properties of robots, such as weight and speed, the frequency with which they encounter the general public, and the closeness of their operations to human bodies. Autonomous vehicles and surgical robots, e.g., require tighter regulation than robot vacuum cleaners.

The task of developing proper regulations for potentially dangerous human–robot interaction is challenging. It begins with the need to determine the entity to whom rules and prohibitions are addressed: manufacturers; programmers; those who rely on robots as tools, such as owners or users; third parties who happen to encounter robots, e.g., in the case of automated cars, other road users; or malevolent intruders who, e.g., hack computer systems or otherwise manipulate the robot’s functions. Another question is who can – and who should – develop legal standards. Not only legislatures, but also criminal and civil courts can and do contribute to rule-setting. Their rulings, however, generally target a specific case. Systematic and comprehensive regulation seems to call for legislative action. But before considering the enactment of new laws, attention should be paid to existing criminal laws, i.e., general prohibitions that protect human life, bodily integrity, property, etc. These prohibitions can be applied to some human failures that involve robots, but due to their unspecific wording and broad scope, they do not give sufficient guidance for our scenarios. More specific norms of conduct, norms tailored to the production, programming, and use of robots, would certainly be preferable. This leads again to the question of what institution is best situated to develop these norms of conduct. This task requires constant attention to and monitoring of rapid technological developments and emerging trends in robotics. Ultimately, traditional modes of regulation by means of laws might not be ideally suited to respond effectively to emerging technologies. Another major difficulty is that regulations in domestic laws do not make much sense for products circulating in global markets. This may prompt efforts to harmonize national laws.Footnote ⁸ As an alternative, soft law in the form of standards and guidelines proposed by the private sector or regulatory agencies might be a way to achieve faster and perhaps also more universal agreement among the producers and users of robots.Footnote ⁹

For legal scholars and legal policy, the upshot is that we should probably not expect too much from substantive criminal law as an instrument to control the use of new technologies. Effective and comprehensive regulation to prevent harm arising out of human–robot interactions, and the difficult task of balancing societal interest in the services provided by robots against the risks involved, do not belong to the core competencies of criminal law.

II.B Beyond Accidents

Beyond the prevention of accidents, other concerns might call for criminal prohibitions. If there are calls to suppress certain conduct rather than to regulate it, the criminal law is a logical choice. Strict prohibitions would make sense if one were to fundamentally object to the creation of AI and autonomous robots, in part because the long-term consequences for humankind might be serious,Footnote ¹⁰ although it may be too late for that in some instances. A more selective approach would be to demand not a categorical decision against all research in the field of AI and the production of advanced robots in general, but rather efforts to suspend researchFootnote ¹¹ or to stop the production of some kinds of robots. An example of the latter approach would be prohibiting devices that apply deadly force against humans, such as remotely controlled or automated weapons systems, addressed in this volume by Marta Bo.Footnote ¹² Not only is the possibility of accidents a particularly serious concern in this area, but also the reliability of target identification, the precision of application, and the control of access are of utmost importance. Even if autonomous weapon systems work as intended, they might in the long run increase the death toll in wars, and ethical doubts regarding war might grow if the humans responsible for aggressive military operations do not face personal risks.Footnote ¹³ Arguments that point to the risk of remote harm are often based on moral concerns. This is most evident in the discussions about sex robots. Should sex robots in general or, more particularly, sex robots that imitate stereotypical characteristics of female prostitutes, be banned?Footnote ¹⁴ The proposition of such prohibitions would need to be supported by strong empirical and normative arguments, including explanations as to why sex robots are more problematic than sex dolls, whether it is plausible to expect such robots to have negative effects on a sizable number of persons, why sexual activity involving humans and robots is morally objectionable, and even if convincing arguments of this kind could be made, why the state should engage in the enforcement of norms regarding sexual morality.

For legal theorists, it is also interesting to ask whether, at some point, policy debates will no longer focus solely on remote harms to other human beings, collective human concerns such as gender equality, or human values and morals, but will instead expand to include the interests or rights of individual robots as well. Take the example of sex robots. Could calls to prohibit sexual interactions between humans and robots refer to the dignity of the robot and its right to dignity? Might we experience a re-emergence of debates about slavery? At present, it would certainly be premature to claim that humans and robots should be treated as equivalent, but discussions about these issues have already begun.Footnote ¹⁵ As long as robots are distinguishable from humans in several dimensions, such as bodies, social competence, and emotional expressivity, it is unlikely that the rights humans grant one another will be extended to them. As long as there are no truly humanoid robots, i.e., robots that resemble humans in all or most physiological and psychological dimensions,Footnote ¹⁶ tremendous cognitive abilities alone are unlikely to trigger widespread demands for equal treatment such as the recognition of robots’ rights. For the purpose of this introductory chapter, it must suffice to point out that thinking in this direction would also be relevant to debates concerning the need to criminalize selected conduct in order to protect the interests of robots.

III.A Human Liability for Unforeseen Accidents

III.B Human Liability for the Use of a Robot with the Intent to Commit a Crime

Robots can be purposefully used to commit crimes, e.g., to spy on other persons.Footnote ²⁴ If the accused human intentionally designed, manipulated, used, or abused a robot to commit a crime, he or she can be held criminally liable for the outcome.Footnote ²⁵ The crucial point in such cases is that the human who employs the robot uses it as a tool.Footnote ²⁶ If perpetrators pursue their criminal goals with the use of a tool, it does not matter whether the tool is of the traditional, merely mechanical kind, such as a gun, or whether it has some features of intelligence, such as an automated weapon that is, e.g., reprogrammed for a criminal purpose.

While this is clearly the case for many criminal offenses, particularly those that focus on outcomes such as causing the death of another person, the situation with regard to other criminal offenses is not so clear. It will not always be obvious that a robot will be able to fulfil the definitional elements of all offenses. It could, e.g., be argued that sexual offenses that require bodily contact between offender and victim cannot be committed if the offender causes a robot to touch another person in a sexual way. In such cases, it is a matter of interpretation if wrongdoing requires the physical involvement of the human offender’s body. I would answer this particular question in the negative, because the crucial point is the penetration of the victim’s body. However, answers must be developed for different crimes separately, based on the legal terminology used and the kind of interest protected.

III.C Human Liability for Foreseen but Unavoidable Harm

In the situation of an unsolvable, tragic dilemma, in which there is no alternative harmless action, a robot might injure humans as part of a planned course of action. The most frequently discussed examples of these dilemmas involve automated cars in traffic scenarios in which all available options, such as staying on track or altering course, will lead to a crash with human victims.Footnote ²⁷ If such events have been anticipated by human programmers, the question arises of whether they could perhaps be held criminally liable, should the dilemmatic situation in fact occur. When human drivers in a comparable dilemma knowingly injure others to save their own lives or the lives of their loved ones, criminal law systems recognize defenses that acknowledge the psychological and normative forces of strong fear, the will to survive, and personal attachments.Footnote ²⁸ The rationale of such defenses does not apply, however, if a programmer, who is not in acute distress, decides that the automated car should always safeguard passengers inside the vehicle, and thus chooses the course that will lead to the death of humans outside the car.

If a human driver has to choose between swerving to save the lives of two persons on the road directly in front of the car, thus hitting and killing a single person on the sidewalk, or staying the course, thus hitting and killing both persons on the road, criminal law doctrine does not provide clear-cut answers. Under German doctrine, which displays a built-in aversion to utilitarian reasoning, the human driver who kills one person to save two would risk criminal punishment.Footnote ²⁹ Whether this would change once the assessment shifts from the human driver at the wheel of the car at the crucial moment to the vehicle’s programmer is an interesting question. German law is shaped by a strong preference for remaining passive, i.e., one may not become active in order to save the greater number of lives, but for the programmer, this phenomenological difference dissolves completely. At the time the automated car or other robot is manufactured, it is simply a decision between programming option A or programming option B for dilemmatic situations.Footnote ³⁰

III.D Self-Defense against Robots

If a human faces imminent danger of being injured or otherwise harmed by a robot, and the human knowingly or purposefully damages or destroys that robot, the question arises as to whether this situation is covered by a justificatory defense. In some cases, a necessity/lesser evil defense could be raised successfully if the danger is substantial. In other cases, it could be questioned if a lesser evil defense would be applicable, e.g., if someone shoots down a very expensive drone to prevent it from taking pictures.Footnote ³¹ Under such circumstances, another justificatory defense might be that of self-defense. In German criminal law, self-defense does not require a proportionality test.Footnote ³² In the case of an unlawful attack, it is permissible to destroy valuable objects even if the protected interest might be of comparatively minor importance. The crucial question in the drone case is whether an “unlawful attack”Footnote ³³ or “unlawful force by another person”Footnote ³⁴ requires that the attacker is a human being.

III.E Criminal Liability of Robots

In the realm of civil liability, robots could be treated as legal persons, and this status could be combined with the duty of producers or owners to endow robots with sufficient funds to compensate potential accident victims.Footnote ³⁵ A different question is whether a case could also be made for the capacity of robots to incur criminal liability.Footnote ³⁶ This is a highly contested proposal and a fascinating topic for criminal law theorists. Holding robots criminally liable would not be compatible with traditional features of criminal law: its focus on human agency and the notion of personal guilt, i.e., Schuld, which is particularly prominent in German criminal law doctrine. Many criminal law theorists defend these features as essential to the very idea of criminal law and thus reject the idea of permitting criminal proceedings against robots. But this is at best a weak argument. Criminal law doctrine is not set in stone; it has adapted to changes in the real world in the past and can be expected to do so again in the future.

The crucial question is whether there are additional principled objections to subjecting robots to criminal liability. Scholars typically examine the degree to which the abilities of robots are similar to those of humansFootnote ³⁷ and ask whether robots fulfil the requirements of personhood, which is defined by means of concepts such as autonomy and free will.Footnote ³⁸ These positions could be described as status-centered, anthropocentric, and essentialist. Traditional concepts of personhood rely on ontological claims about what humans are and the characteristics of humans qua humans. As possible alternatives, notions such as autonomy and personhood could also be described in a more constructivist manner, as the products of social attribution,Footnote ³⁹ and it is worth considering whether the criminal liability of robots could at least be constructed for a limited subsection of criminal law, i.e., strict liability regulatory offenses, for legal systems that recognize such offenses.Footnote ⁴⁰

Instead of exploring the degree of a robot’s human-ness or personhood, the alternative is to focus on the functions of criminal proceedings and punishments. In this context, the crucial question is whether some goals of criminal punishment practices could be achieved if norms of conduct were explicitly addressed to robots and if defendants were not humans but robots. As we will see, it makes sense to distinguish between the preventive functions of criminal law, such as deterrence, and the expressive meaning of criminal punishment.

The purpose of deterring agents is probably not easily transferrable from humans to robots. Deterring someone presupposes that the receiver of the message is actually aware of a norm of conduct but is inclined not to comply with it, because other incentives seem more attractive or other personal motives and emotions guide his or her decision-making. AI will probably not be prone to the kind of multi-layered, sometimes blatantly irrational type of decision-making practiced by humans. For robots, the point is to identify the right course of conduct, not to avoid being side-tracked by greed and emotions. But preventive reasoning could, perhaps, be brought to bear on the humans involved in the creation of robots who might be indirectly influenced. They might be effectively driven toward higher standards of care in order to avoid public condemnation of their products’ behavior.Footnote ⁴¹

In addition to their potentially preventive effects, criminal law responses have expressive features. They communicate that certain kinds of wrongful conduct deserve blame, and more specifically they reassure crime victims that they were indeed wronged by the other party to the interaction, and not that they themselves made a mistake or simply suffered a stroke of bad luck.Footnote ⁴² Some of the communicative and expressive features of criminal punishment might retain their functions, and address the needs of victims, if robots were the addressees of penal censure.Footnote ⁴³ Even if robots will not for a long time, if ever, be capable of feeling remorse as an emotional state, the practice of assigning blame could persist with some modifications.Footnote ⁴⁴ It might suffice if robots had the cognitive capacity to understand what their environment labels as right and wrong and the reasons behind these judgments, and if they adapted their behavior to norms of conduct. Communication would be possible with smart robots that are capable of explaining the choices they have made.Footnote ⁴⁵ In their ability to respond and to modify parameters for future decision-making, advanced robots are distinguishable from others not held criminally liable, e.g., animals, young children, and persons with severe mental illness.

Admittedly, criminal justice responses to the wrongful behavior of robots cannot be the same as the responses to delinquent humans. It is difficult, e.g., to conceive of a “hard treatment” component of criminal punishmentFootnote ⁴⁶ that would apply to robots, and such a component, if conceived, might well be difficult to enforce.Footnote ⁴⁷ It could, however, be argued that punishment in the traditional sense is not necessary. For an entirely rational being, the message that conduct X is wrongful and thus prohibited, and the integration of this message into its future decision-making, would be sufficient. The next question would be if blaming robots and eliciting responses could provide some comfort to human victims and thus fulfil their emotional needs. It is conceivable that a formal, solemn procedure might serve some of the functions that traditional criminal trials fulfil, at least in the theoretical model, but study would be required to determine whether empathy or at least the potential for empathy are prerequisites for calling a perpetrator to account. Criminal law theorists have argued that robots could only be held criminally liable if they were able to understand emotional states such as suffering.Footnote ⁴⁸ In my view, a deeply shared understanding of what it means, emotionally, to be hurt is not necessarily essential for the communicative message delivered to victims who have been harmed by a robot.

Another question, however, is whether a merely communicative “criminal trial,” without the hard treatment component of sanctions, would be so unlike criminal punishment practices as we know them that the general human public would consider it pointless and not worth the effort, or even a travesty. This question moves the inquiry beyond criminal law theory. Answers would require empirical insight into the feasibility and acceptance of formal, censuring communication with robots. If designing procedures with imperfect similarities to traditional criminal trials would make sense, the question of criminal codes for robots should perhaps also be addressed.Footnote ⁴⁹

III.F Robots as Victims of Crime

Another area that might require more attention in the future is the interpretation of criminal laws if the victim of the crime is not a human, as assumed by the legislators when they passed the law, but a robot. Crimes against personality rights, e.g., might lead to interesting questions. Might it be a criminal offense to record spoken words, a criminal offense under §201 of the Strafgesetzbuch (German Criminal Code), if the speaker is a robot rather than a human being? Thinking in this direction would require considering whether advanced robots should be afforded constitutional and other rightsFootnote ⁵⁰ and, should such a discussion gain seriousness, which rights these would be.

II.A Are Programmers in Control of Algorithm and Data-Related Risks in AVs?

Before turning to the risks and failures that might lie in algorithm design and thus potentially under programmer control, this section describes the tasks required when producing an AV, and then reviews some of the rules that need to be coded to achieve this end.

The main task of AVs is navigation, which can be understood as the AV’s behavior as well as the algorithm’s effect. Navigation on roads is mostly premised on rules-based behavior requiring knowledge of traffic rules and the ability to interpret and react to uncertainty. In AVs, automated tasks include the identification and classification of objects usually encountered while driving, such as vehicles, traffic signs, traffic lights, and road lining.Footnote ²¹ Furthermore, “situational awareness and interpretation”Footnote ²² is also being automated. AVs should be able “to distinguish between ordinary pedestrians (merely to be avoided) and police officers giving direction,” and conform to social habits and rules by, e.g., “interpret[ing] gestures by or eye contact with human traffic participants.”Footnote ²³ Finally, there is an element of prediction: AVs should have the capability to anticipate the behavior of human traffic participants.Footnote ²⁴

In AV design, the question of whether traffic rules can be accurately embedded in algorithms, and if so who is responsible for translating these rules into algorithms, becomes relevant in determining the accuracy of the algorithm design as well as attributing potential criminal responsibility. For example, are only programmers involved, or are lawyers and/or manufactures also involved? While some traffic rules are relatively precise and consist of specific obligations, e.g., a speed limit represents an obligation not to exceed that speed,Footnote ²⁵ there are also several open-textured and context-dependent traffic norms, e.g., regulations requiring drivers to drive carefully.Footnote ²⁶

AV incidents might stem from a failure of the AI to identify objects or correctly classify them. For example, the first widely reported incident involving an AV in May 2016 was allegedly caused by the vehicle sensor system’s failure to distinguish a large white truck crossing the road from the bright spring sky.Footnote ²⁷ Incidents may also arise due to failures to correctly interpret or predict the behavior of others or traffic conditions, which may sometimes be interlinked with or compounded by problems of detection and sensing.Footnote ²⁸ In turn, mistakes in both object identification and prediction might occur as a result of faulty algorithm design and/or derived from flawed data. In the former case, prima vista, if mistakes in object identification and/or prediction occur due to an inadequate algorithm design, the criminal responsibility of programmers could be engaged.

In relation to the latter, the increasing and almost dominant use of machine learning (ML) algorithms in AVsFootnote ²⁹ means that issues of algorithms and data are interrelated. The performance of algorithms has become heavily dependent on the quality of data. A multitude of different algorithms are used in AVs for different purposes, with supervised and unsupervised learning-based algorithms often complementing one another. Supervised learning, in which an algorithm is fed instructions on how to interpret the input data, relies on a fully labeled dataset. Within AVs, the supervised learning models are usually: (1) “classification” or “pattern recognition algorithms,” which process a given set of data into classes and help to recognize categories of objects in real time, such as street signs; and (2) “regression,” which is usually employed for predicting events.Footnote ³⁰ In cases of supervised learning, mistakes can arise from incorrect data annotation instead of a faulty algorithm design per se. If incidents do occur,Footnote ³¹ the programmer arguably would not be able to foresee those risks and be considered in control of the subsequent navigation decisions.

Other issues may arise with unsupervised learningFootnote ³² where an ML algorithm receives unlabeled data and programmers “describe the desired behaviour and teach the system to perform well and generalise to new environments through learning.”Footnote ³³ Data can be provided in the phase of simulating and testing, but also during the use itself by the end-user. Within such methods, “deep learning” is increasingly used to improve navigation in AVs. Deep learning is a form of unsupervised learning that “automatically extracts features and patterns from raw data [such as real-time data] and predicts or acts based on some reward function.”Footnote ³⁴ When an incident occurs due to deep learning techniques using real data, it must be assessed whether the programmer could have foreseen that specific risk and the resulting harm, or whether it derived, e.g., from an unforeseeable interaction with the environment.

II.B Programmer or User: Who Is in Control of AVs?

As shown in the March 2018 Uber incident,Footnote ³⁵ incidents can also derive from failures of the user to regain control of the AV, with some AV manufacturers attempting to shift the responsibility for ultimately failing to avoid collisions onto the AVs’ occupants.Footnote ³⁶ However, there are serious concerns as to whether an AV’s user, who depending on the level of automation is essentially in an oversight role, is cognitively in the position to regain control of the vehicle. This problem is also known as automation bias,Footnote ³⁷ a cognitive phenomenon in human–machine interaction, in which complacency, decrease of attention, and overreliance on the technology might impair the human ability to oversee, intervene, and override the system if needed.

Faulty human–machine interface (HMI) design, i.e., the technology which connects an autonomous system to the human, such as a dashboard or interface, could cause the inaction of the driver in the first place. In these instances, the driver could be relieved from criminal responsibility. Arguably, HMIs do not belong to programmers’ functional obligations and therefore fall outside of a programmer’s control.

There are phases other than actual driving where a user could gain control of an AV’s decisions. Introducing ethics settings into the design of AVs may ensure control over a range of morally significant outcomes, including trolley-problem-like decisions.Footnote ³⁸ Such settings may be mandatorily introduced by manufacturers with no possibility for users to intervene and/or customize them, or they may be customizable by users.Footnote ³⁹ Customizable ethics settings allow users “to manage different forms of failure by making autonomous vehicles follow [their] decisions” and their intention.Footnote ⁴⁰

II.C Are Some AV-Related Risks Out of Programmer Control?

There are a group of risks and failures that could be considered outside of programmer control. These include communications failures, hacking of the AV by outside parties, and unforeseeable bystander behavior. One of the next steps predicted in the field of vehicle automation is the development of software enabling AVs to communicate with one another and to share real-time data gathered from their sensors and computer systems.Footnote ⁴¹ This means that a single AV “will no longer make decisions based on information from just its own sensors and cameras, but it will also have information from other cars.”Footnote ⁴² Failures in vehicular communication technologiesFootnote ⁴³ or inaccurate data collected by other AVs cannot be attributed to a single programmer, as they might fall beyond their responsibilities and functions, and also beyond their control.

Hacking could also cause AV incidents. For example, “placing stickers on traffic signs and street surfaces can cause self-driving cars to ignore speed restrictions and swerve headlong into oncoming traffic.”Footnote ⁴⁴ Here, the criminal responsibility of a programmer could depend on whether the attack could have been foreseen and whether the programmer should have created safeguards against it. However, the complexity of AI systems could make them more difficult to defend from attacks and more vulnerable to interference.Footnote ⁴⁵

Finally, imagine an AV that correctly follows traffic rules, but hits a pedestrian who unforeseeably slipped and fell onto the road. Such unforeseeable behavior of a bystander is relevant in criminal law cases on vehicular homicide, as it will break the causal nexus between the programmer and the harmful outcome.Footnote ⁴⁶ In the present case, it must be determined which unusual behavior should be foreseen at the stage of programming, and whether standards of foreseeability in AVs should be higher for human victims.

III.A Are Programmers in Control of Algorithm and Data-Related Risks in AWs?

Autonomous drones provide an example of one of the most likely applications of autonomy within the military domain,Footnote ⁴⁸ and this example will be used to highlight the increasingly autonomous tasks in AWs. This section will address the rules to be programmed, and identify where some risks might lie in the phase of algorithm design.

The two main tasks being automated in autonomous drones are: (1) navigation, which is less problematic than on roads and a relatively straightforward rule-based behavior, i.e., they must simply avoid obstacles while in flight; and (2) weapon release, which is much more complex as “ambiguity and uncertainty are high when it comes to the use of force and weapon release, bringing this task in the realm of expertise-based behaviours.”Footnote ⁴⁹ Within the latter, target identification is the most important function because it is crucial to ensure compliance with the international humanitarian law (IHL) principle of distinction, the violation of which could also cause individual criminal responsibility for war crimes. The principle of distinction establishes that belligerents and those executing attacks must distinguish at all times between civilians and combatants, and not target civilians.Footnote ⁵⁰ In target identification, the two main automated tasks are: (1) object identification and classification on the basis of pattern recognition;Footnote ⁵¹ and (2) prediction, e.g., predicting that someone is surrendering, or based on the analysis of patterns of behavior, predicting that someone is a lawful target.Footnote ⁵²

Some of the problems in the algorithm design phase may derive from translating the open-textured and context-dependentFootnote ⁵³ rules of IHL,Footnote ⁵⁴ such as the principle of distinction, into algorithms, and from incorporating programmer knowledge and expert-based rules,Footnote ⁵⁵ such as those needed to analyze patterns of behavior in targeted strikes and translate them into code.

There are some differences compared with the algorithm design phase in AVs. Due to the relatively niche and context-specific nature of IHL, compared to traffic law which is more widely understood by programmers, programming IHL might require a stronger collaboration with outside expertise, i.e., military lawyers and operators.

However, similar observations to AVs can be made in relation to supervised and unsupervised learning algorithms. Prima vista, if harm results from mistakes in object identification and prediction based on an inadequate algorithm design, the criminal responsibility of the programmer(s) could be engaged. Depending on the foreseeability of such data failures to the programmer and the involvement of third parties in data labeling, and assuming mistakes could not be foreseen, criminal responsibility might not be attributable to programmers. Also similar to AVs, the increasing use of deep learning methods in AWs makes the performance of algorithms dependent on both the availability and accuracy of data. Low quality and incorrect data, missing data, and/or discrepancies between real and training data may be conducive to the misidentification of targets.Footnote ⁵⁶ When unsupervised learning is used in algorithm design, environmental conditions and armed conflict-related conditions, e.g., smoke, camouflage, and concealment, may inhibit the collection of accurate data.Footnote ⁵⁷ As with AVs, programmers of AWs may at some point gain sufficient knowledge and experience regarding the robustness of data and unsupervised machine learning that would subject them to due diligence obligations, but the chapter assumes that programmers have not reached that stage yet. In the case of supervised learning, errors in data may lie in a human-generated data feed,Footnote ⁵⁸ and incorrect data labeling could lead to mistakes and incidents that might be attributable to someone, but not to programmers.

III.B Programmer or User: Who Is in Control of AWs?

The relationship between programmers and users of AWs presents different challenges than AVs. In light of current trends in AW development, arguably toward human–machine interaction rather than full autonomy of the weapons system, the debate has focused on the degree of control that militaries must retain over the weapon release functions of AWs.

However, control can be shared and distributed among programmers and users in different phases, from the design phase to deployment. As noted above, AI engineering in the military domain might require a strong collaboration between programmers and military lawyers in order to accurately code IHL rules in algorithms.Footnote ⁵⁹ Those arguing for the albeit debated introduction of ethics settings in AWs maintain that ethics settings would “enable humans to exert more control over the outcomes of weapon use [and] make the distribution of responsibilities [between manufacturers and users] more transparent.”Footnote ⁶⁰

Finally, given their complexity, programmers of AWs might be more involved than programmers of AVs in the use of AWs and in the targeting process, e.g., being required to update the system or implement some modifications to the weapon target parameters before or during the operation.Footnote ⁶¹ In these situations, it must be evaluated to what extent a programmer could foresee a certain risk entailed in the deployment and use of an AW in relation to a specific attack rather than just its use in the abstract.

III.C Are Some AW-Related Risks Out of Programmer Control?

In the context of armed conflict, it is highly likely that AWs will be subject to interference and attacks by enemy forces. A UN Institute for Disarmament Research (UNIDIR) report lists several pertinent examples: (1) signal jamming could “block systems from receiving certain data inputs (especially navigation data)”; (2) hacking, such as “spoofing” attacks, might “replace an autonomous system’s real incoming data feed with a fake feed containing incorrect or false data”; (3) “input” attacks could “change a sensed object or data source in such a way as to generate a failure,” e.g., enemy forces “may seek to confound an autonomous system by disguising a target”; and (4) “adversarial examples” or “evasion,” which are attacks that “involve adding subtle artefacts to an input datum that result in catastrophic interpretation error by the machine.”Footnote ⁶² In such situations, the issue of criminal responsibility for programmers will depend on the modalities of the adversarial interference, whether it could have been foreseen, and whether the AW could have been protected from foreseeable types of attacks.

Similar to the AV context, failures of communication technology, caused by signal jamming or by failures of communication systems between a human operator and the AI system or among connected AI systems, may lead to incidents that could not be imputed to a programmer.

Finally, conflict environments are likely to drift constantly as “[g]roups engage in unpredictable behaviour to deceive or surprise the adversary and continually adjust (and sometimes radically overhaul) their tactics and strategies to gain an edge.”Footnote ⁶³ The continuously changing and unforeseeable behavior of opposing belligerents and the tactics of enemy forces can lead to “data drift,” whereby changes that are difficult to foresee can lead to a weapon system’s failure without it being imputable to a programmer.Footnote ⁶⁴

IV.A Actus Reus in AV-Related Crimes

This section focuses on the domestic criminal offenses of negligent homicide and manslaughter in order to assess whether the actus reus of AV-related crimes could be performed by a programmer. It does not address traffic and road violations generally,Footnote ⁶⁶ nor the specific offense of vehicular homicide.Footnote ⁶⁷

Given the increasing use of AVs and pending AV-related criminal cases in the United States,Footnote ⁶⁸ it seems appropriate to take the Model Penal Code (MPC) as an example of common law legislation.Footnote ⁶⁹ According to the MPC, the actus reus of manslaughter consists of “killing for which the person is reckless about causing death.”Footnote ⁷⁰ Negligent homicide concerns instances where a “person is not aware of a substantial risk that a death will result from his or her conduct, but should have been aware of such a risk.”Footnote ⁷¹

While national criminal law frameworks differ considerably, there are similarities regarding causation which are relevant here. Taking Germany as a representative example of civil law traditions, the Strafgesetzbuch (German Criminal Code) (StGB) distinguishes two forms of intentional homicide: murderFootnote ⁷² and manslaughter.Footnote ⁷³ Willingly taking the risk of causing death is sufficient for manslaughter.Footnote ⁷⁴ Negligent homicide is proscribed separately,Footnote ⁷⁵ and the actus reus consists of causing the death of a person through negligence.Footnote ⁷⁶

These are crimes of result, where the harm consists of the death of a person. While programmer conduct may be remote with regard to AV incidents, some decisions taken by AV programmers at an early stage of development could decisively impact the navigation behavior of an AV that results in a death. In other words, it is conceivable that a faulty algorithm designed by a programmer could cause a fatal road accident. The question then becomes what is the threshold of causal control exercised by programmers over an AV’s unlawful behavior of navigation and its unlawful effects such as a human death.

IV.B Actus Reus in AW-Related War Crimes

This section addresses AW-related war crimes and whether programmers could perform the required actus reus. Since the actus reus would most likely stem from an AW’s failure to distinguish between civilian and military targets, the war crime of indiscriminate attacks, which criminalizes violations of the aforementioned IHL rule of distinction,Footnote ⁷⁷ takes on central importance.Footnote ⁷⁸ The war crime of indiscriminate attacks refers inter alia to an attack that strikes military objectives and civilians or civilian objects without distinction. This can occur as a result of the use of weapons that are incapable of being directed at a specific military objective or accurately distinguishing between civilians and civilian objects and military objectives; these weapons are known as inherently indiscriminate weapons.Footnote ⁷⁹

While this war crime is neither specifically codified in the Rome Statute nor in AP I, it has been subsumedFootnote ⁸⁰ under the war crime of directing attacks against civilians. Under AP I, the actus reus of the crime is defined in terms of causing death or injury.Footnote ⁸¹ In crimes of result with AWs, a causal nexus between the effects resulting from the deployment of an AW and a programmer’s conduct must be established. Under the Rome Statute, the war crime is formulated as a conduct crime, proscribing the actus reus as the “directing of an attack” against civilians.Footnote ⁸² A causal nexus must be established between the unlawful AW’s behavior and/or the attack and the programmer’s conduct.Footnote ⁸³ Under both frameworks, the question is whether programmers exercised causal control over the behavior and/or effects, e.g., death or attack, of an AW.

A final issue relates to the required nexus with an armed conflict. The Rome Statute requires that the conduct must take place “in the context of and was associated with” an armed conflict.Footnote ⁸⁴ However, while undoubtedly there is a temporal and physical distance between programmer conduct and the armed conflict, it is conceivable that programmers may program AW software or upgrade it during an armed conflict. In certain instances, it could be argued that programmer control continues even after the completion of the act of programming, when the effects of their decisions materialize in the behavior and/or effects of AWs in armed conflict. Programmers can be said to exercise a form of control over the behavior and/or effects of AWs that begins with the act of programming and continues thereafter.

IV.C The Causal Nexus between Programming and AV- and AW-Related Crimes

A crucial aspect of programmer criminal responsibility is the causal control they exercise over the behavior and/or effects of AVs and AWs. The assessment of causation refers to the conditions under which an AV’s and AW’s unlawful behavior and/or effects should be deemed the result of programmer conduct for purposes of holding them criminally responsible.

Causality is a complex topic. In common law and civil law countries, several tests to establish causation have been put forward. Due to difficulties in establishing a uniform test for causation, it has been argued that determining conditions for causation are “ultimately a matter of legal policy.”Footnote ⁸⁵ But this does not render the formulation of causality tests in the relevant criminal provisions completely beyond reach. While a comprehensive analysis of these theories is beyond the scope of this chapter, for the purposes of establishing when programmers exercise causal control, some theories are more aligned with the policy objectives pursued by the suppression of AV- and AW-related crimes.

First, in common law and civil law countries, the “but-for”/conditio sine qua non test is the dominant test for establishing physical causation, and it is intended as a relationship of physical cause and effect.Footnote ⁸⁶ In the language of MPC §2.03(1)(a), the conduct must be “an antecedent but for which the result in question would not have occurred.” The “but for” test works satisfactorily in cases of straightforward cause and effect, e.g., pointing a loaded gun toward the chest of another person and pulling the trigger. However, AV- and AW-related crimes are characterized by a temporal and physical gap between programmer conduct and the behavior and effect of AVs and AWs. They involve complex interactions between AVs and AWs and humans, including programmers, data providers and labelers, users, etc. AI itself is also a factor that could intervene in the causal chain. The problem of causation in these cases must thus be framed in a way that reflects the relevance of intervening and superseding causal forces which may break the causal nexus between a programmer’s conduct and AV- and AW-related crime.

Both civil law and common law systems have adopted several theories to overcome the shortcomingsFootnote ⁸⁷ and correct the potential over-inclusivenessFootnote ⁸⁸ of the “but-for” test, in complex cases involving numerous necessary conditions. Some of these theories include elements of foreseeability in the causality test.

The MPC adopts the “proximate cause test,” which “differentiates among the many possible ‘but for’ causal forces, identifying some as ‘necessary conditions’ – necessary for the result to occur but not its direct ‘cause’ – and recognising others as the ‘direct’ or ‘proximate’ cause of the result.”Footnote ⁸⁹ The relationship is “direct” when the result is foreseeable and as such “this theory introduces an element of culpability into the law of causation.”Footnote ⁹⁰

German theories about adequacy assert that whether a certain factor can be considered a cause of a certain effect depends on “whether conditions of that type do, generally, in the light of experience, produce effects of that nature.”Footnote ⁹¹ These theories, which are not applied in their pure form in criminal law, include assessments that resemble a culpability assessment. They bring elements of foreseeability and culpability into the causality test, and in particular, a probability and possibility judgment regarding the actions of the accused.Footnote ⁹² However, these theories leave unresolved the different knowledge perspectives, i.e., objective, subjective, or mixed, on which the foreseeability assessment is to be based.Footnote ⁹³

Other causation theories include an element of understandability, awareness, or foreseeability of risks. In the MPC, the “harm-within-the risk” theory considers that causation in reckless and negligent crimes is in principle established when the result was within the “risk of which the actor is aware or … of which he should be aware.”Footnote ⁹⁴ In German criminal law, some theories describe causation in terms of the creation or aggravation of risk and limit causation to the unlawful risks that the violated criminal law provision intended to prevent.Footnote ⁹⁵

In response to the drawbacks of these theories, the teleological theory of causation holds that in all cases involving a so-called intervening independent causal force, the criterion should be whether the intervening causal force was “produced by ‘chance’ or was rather imputable to the criminal act in issue.”Footnote ⁹⁶ Someone would be responsible for the result if their actions contributed in any manner to the intervening factor. What matters is the accused’s control over the criminal conduct and whether the intervening factor was connected in a but/for sense to their criminal act,Footnote ⁹⁷ thus falling within their control.

In ICL, a conceptualization of causation that goes beyond the physical relation between acts and effects is more embryonic. However, it has been suggested that theories drawn from national criminal law systems, such as risk-taking and linking causation to culpability, and thus to foreseeability, should inform a theory of causation in ICL.Footnote ⁹⁸ It has also been suggested that causality should entail an evaluation of the functional obligations of an actor and their area of operation in the economic sphere. According to this theory, causation is “connected to an individual’s control and scope of influence” and is limited to “dangers that he creates through his activity and has the power to avoid.”Footnote ⁹⁹ As applied in the context of international crimes, which have a collective dimension, these theories could usefully be employed in the context of AV and AW development, which is collective by nature and is characterized by a distribution of responsibilities.

Programmers in some instances will cause harm through omission, notably by failing to avert a particular harmful risk when they are under a legal duty to prevent harmful events of that type (“commission by omission”).Footnote ¹⁰⁰ In these cases, the establishment of causation will be hypothetical as there is no physical cause-effect relationship between an omission and the proscribed result.Footnote ¹⁰¹ Other instances concern whether negligence on the side of the programmers, via, e.g., a lack of instructions and warnings, have contributed to and caused the omission, constituting a failure to intervene on behalf of the user. Such omissions amount to negligence, i.e., violations of positive duties of care,Footnote ¹⁰² and since it belongs to mens rea, will be addressed in the following section.

IV.D Criminal Negligence: Programming AVs and AWs

In light of the integration of culpability assessments in causation tests, an assessment of programmers’ criminal responsibility would be incomplete without addressing mens rea issues. In relation to mens rea, while intentionally and knowingly programming an AV or AW to commit crimes falls squarely under these prohibitions, in both these contexts, the most expected and problematic issue is the unintended commission of these crimes, i.e., cases in which the programmer did not design the AI system to commit an offense, but harm nevertheless arises during its use.Footnote ¹⁰³ In such situations, programmers had no intention to commit an offense, but still might incur criminal liability for risks that they should have known and foreseen. To define the scope of criminal responsibility for unintended harm, it is crucial to determine which risks can be known and foreseen by an AV or AW programmer.

There are important differences in the mens rea requirements of AV- and AW-related crimes. Under domestic criminal law, the standards of recklessness and negligence apply to the AV-related crimes of manslaughter and negligent homicide. While “[a] person acts ‘recklessly’ with regard to a result if he or she consciously disregards a substantial risk that his or her conduct will cause the result; he or she acts only ‘negligently’ if he or she is unaware of the substantial risk but should have perceived it.”Footnote ¹⁰⁴ The MPC provides that “criminal homicide constitutes manslaughter when it is committed recklessly.”Footnote ¹⁰⁵ In the StGB, dolus eventualis, i.e., willingly taking the risk of causing death, would encompass situations covered by recklessness and is sufficient for manslaughter.Footnote ¹⁰⁶ For negligent homicide,Footnote ¹⁰⁷ one of the prerequisites is that the perpetrator can foresee the risk to a protected interest.Footnote ¹⁰⁸

Risk-based mentes reae are subject to more dispute in ICL. The International Tribunal for the former Yugoslavia accepted that recklessness could be a sufficient mens rea for the war crime of indiscriminate attacks under Article 85(3)(a) of AP I.Footnote ¹⁰⁹ However, whether recklessness and dolus eventualis could be sufficient to ascribe criminal responsibility for war crimes within the framework of the Rome Statute remains debated.Footnote ¹¹⁰

Unlike incidents with AVs, incidents in war resulting from a programmer’s negligence cannot give rise to their criminal responsibility. Where applicable, recklessness and dolus eventualis, which entail understandability and foreseeability of risks of developing inherently indiscriminate AWs, become crucial to attribute responsibility to programmers in scenarios where programmers foresaw and took some risks. Excluding these mental elements would amount to ruling out the criminal responsibility of programmers in most expected instances of war crimes.

III.A Basic Rules with Examples Regarding the Due Diligence of Surgeons

Unlike other jurisdictions, Swiss law explicitly defines the basic rule determining criminal negligence. In Article 12, paragraph 3 of the SCC, a “person commits a felony or misdemeanour through negligence if he fails to consider or disregards the consequences of his conduct due to a culpable lack of care. A lack of care is culpable if the person fails to exercise the care that is incumbent on him in the circumstances and commensurate with his personal capabilities.”Footnote ²²

Determining a person’s precise due diligence obligations can be a complex endeavor. In Swiss criminal law a myriad of due diligence rules underpin negligence and are used to specify the relevant obligations, including legal norms, private regulations, and a catch-all-clause, dubbed the risk principle (Gefahrensatz).Footnote ²³ The risk principle establishes that everyone has to behave in a reasonable way that minimizes threats to the relevant legal interest as best as possible.Footnote ²⁴ For example, a surgeon must take all reasonable possible precautions to avoid increasing a pre-existing danger to the patient.Footnote ²⁵

To apply the risk principle, the maximum permissible risk must be determined.Footnote ²⁶ For this purpose, the general risk range must first be determined, and this range is limited by human skill;Footnote ²⁷ no one can be reproached for not being able to prevent the risk in spite of doing everything humanly possible (ultra posse nemo tenetur).Footnote ²⁸ The risk range is therefore limited by society’s understanding of the permissible risk, and by the abilities possessed by a capable, psychologically, and physically normal person; no superhuman performance is expected.Footnote ²⁹ However, if a person’s ability is lower than what is required in a situation, the performed activity should be refrained from.Footnote ³⁰ In the context of medical personnel, a surgeon who is not familiar with the use of robots may not perform such an operation.

As the law does not list the exact duties of care of a surgeon, it is left to the courts to specify in more detail the content and scope of the medical duties of care based on the relevant statutes and regulations. In that respect, it is not of significance whether the treatment is governed by public or private law.Footnote ³¹

III.B Due Diligence Standards Specific to Surgeons

Swiss criminal law is applied in the medical field, and every healthcare professional who hurts a patient intentionally or with criminal negligence can be liable.Footnote ³² Surgery is an activity that is, in principle, hazardous, and a surgeon may be prosecuted if he or she, consciously or unconsciously,Footnote ³³ neglects a duty of care.Footnote ³⁴ According to the Swiss Federal Supreme Court, the duty of care when applying conventional methods of treatment is based on “the circumstances of the individual case, i.e., the type of intervention or treatment, the associated risks, the discretionary scope and time available to the physician in the individual case, as well as his objectively expected education and ability to perform.”Footnote ³⁵

This reference of the Swiss Federal Supreme Court to the educational background and efficiency of the physician does not indicate that the standard is entirely subjective. Rather, the physician should be assessed according to the knowledge and skills assumed to be available to representatives of his specialty at the time the measures are taken.Footnote ³⁶ This objective approach creates an ongoing obligation for the further education of surgeons.

Part of a surgeon’s obligation is that they owe the patient a regime of treatment that complies with the generally recognized state of medical art (lex artis),Footnote ³⁷ determined at the time of treatment. Lex artis is the guiding principle for establishing due diligence in an individual case in Swiss criminal law.Footnote ³⁸ It encompasses the entire medical procedure, from the examination, diagnosis, therapeutic decision, and implementation of the treatment, and in the case of surgeons from preparing the operation to aftercare.Footnote ³⁹ The standard is therefore not what is individually possible and reasonable, but the care required according to medical indications and best practice.Footnote ⁴⁰ A failure to meet this medical standard leads to a breach of duty of care. Legal regulation, such as the standards of the Medical Professions Act (“MedBG”),Footnote ⁴¹ especially Article 40 lit. a, may be used to determine the respective state of medical art. Together, the regulatory provisions provide for the careful and conscientious practice of the medical profession.Footnote ⁴²

Doctors must also observe and not exceed the limits of their own competence. A surgeon must recognize when they are not able to perform a surgery and need to consult a specialist. This obligation includes the duty to cooperate with other medical personnel, because performing an operation without the required expertise is a breach of duty of care in itself.Footnote ⁴³ As with other areas of medical care, the surgeon’s obligations do not exceed the human ability to foresee events and to influence them in a constructive way.Footnote ⁴⁴

If there are no legal standards for an area of medical practice, courts may refer to guidelines from medical organizations.Footnote ⁴⁵ In practice, courts usually refer to the private guidelines of the Swiss Academy of Medical SciencesFootnote ⁴⁶ and the Code of Conduct of the Swiss Medical Association (“FMH”).Footnote ⁴⁷ Additionally, general duties derived from court decisions, such as “practising the art of medicine according to recognized principles of medical science and humanity,” can be used in a secondary way to articulate a doctor’s specific due diligence obligation.Footnote ⁴⁸

III.C Due Diligence of a Surgeon in Robot-Assisted Surgery

New technologies have long been making appearances in operating rooms. Arthrobot assisted for the first time in 1983; responding to voice command, the robot was able to immobilize patients by holding them steady during orthopedic surgery.Footnote ⁴⁹ Arthrobots are still in use today.Footnote ⁵⁰

The introduction of robots to surgery accomplishes two main aims: (1) they perform more accurate medical procedures; and (2) they enable minimally invasive surgeries, which in turn increases surgeon efficacy and patient comfort by providing a faster recovery. A doctor is, generally, not responsible for the dangers and risks that are inherent in every medical action and in the illness itself.Footnote ⁵¹ However, the surgeon’s obligation of due diligence applies when using a robot. The chapter argues that the precise standards of care should differ, depending on whether the surgeon has control of the robot’s actions or whether the robot reacts independently in the environment, and depending on the extent of the surgeon’s control, including the ability to intervene in a procedure.Footnote ⁵²

The next section introduces and explains the functioning of several examples of surgical robots. These robots qualify as medical devices under Swiss law,Footnote ⁵³ and as such are subject to statutes governing medical devices. Medical devices are defined as instruments, equipment, software, and other objects intended for medical use.Footnote ⁵⁴ Users of medical devices must take all measures required by the state of the art in science and technology to ensure that they pose no additional risk. The lex artis for treatment incorporating robots under Swiss criminal law requires users to apply technical aids lege artis and operate them correctly. For example, when the robot is used again at a later time, its functionality and correct reprocessing must be checked.Footnote ⁵⁵ A surgeon does not have to be a trained technician, but he or she must have knowledge of the technology used, similar to the way that a driver must “know” a car, but need not be a mechanic.

On its own, the concept of lex artis does not imply specific obligations, and the specific parameters of the obligations must be determined based on individual circumstances. According to Article 45, paragraph 1 of the Therapeutic Products Act (TPA), a medical device must not endanger the health of patients when used as intended. If a technical application becomes standard in the field, falling below or not complying with the standard (lex artis) is classified as a careless action.Footnote ⁵⁶ Lack of knowledge of the technology, as well as a lack of control over a device during an operation, leads to an assumption of liability (“Übernahmeverschulden”).Footnote ⁵⁷

A final aspect of the surgeon’s obligations regarding surgical robots is that a patient must always be informedFootnote ⁵⁸ about the robot before an operation, and the duty of documentationFootnote ⁵⁹ must be complied with. Although the precise due diligence obligations of surgeons always depend on the circumstances of individual cases, the typical duties of care regarding two different kinds of robots that incorporate elements of remote-control, and the situation in which a robot provides a warning to the surgeon, are outlined below.

III.D Limiting the Surgeon’s Due Diligence Obligations regarding Surgical Robots through the Principle of Trust (Vertrauensgrundsatz)?

The surgeon’s obligation of supervision currently imposes excessive amounts of liability for the use of surgical robots, because, as discussed above, while surgeons rightfully have obligations to monitor the robot, they should not be required to check every movement the robot makes before it proceeds. The chapter argues that in the context of robot supervision, variations of the principle of trust (Vertrauensgrundsatz) should apply to limit the surgeon’s criminal liability.

When a surgeon works with human team members, the legitimate expectation is that individuals are responsible only for their own conduct and not that of others. The principle of trust is a foundational legal concept, one that enables effective cooperation by identifying spheres of responsibility and limiting the duties of due diligence to those spheres. It relieves individuals from having to evaluate the risk-taking of every individual in the team in every situation, and allows for the effective division of expertise and labor. The principle of trust was developed in the context of road traffic regulation, but it has widespread relevance and is applied today in medical law as well as other areas.Footnote ⁸⁴

The principle of trust has limits and does not provide a carte blanche justifying all actions. If there are concrete indications that trust is unjustified, one must analyze and address that situation.Footnote ⁸⁵ An example regarding surgical robots might be the DaVinciFootnote ⁸⁶ robot. It has been in use for a long time, but if a skilled surgeon notices that the robot is defective, the surgeon must intervene and correct the defect.

The limitations of due diligence arising out of the principle of trust are well established in medical law, an environment where many participants work together based on a division of expertise and labor. In an operating room, several different kinds of specialists are normally at work, such as anesthesiologists, surgeons, and surgical nurses. The principle of trust in this environment limits responsibility to an individual’s own area of expertise and work.Footnote ⁸⁷

One way of understanding the division of labor in surgery is that the primary area is the actual task, i.e., the operation, and the secondary area is supervisory, i.e., being alert to and addressing the misconduct of others.Footnote ⁸⁸ Supervisory responsibility can be imposed horizontally (surgeon–surgeon) or vertically (surgeon–nurse), depending on the position a person occupies in the operating room. An example of the horizontal division of labor in the medical context would be if several doctors are assigned equal and joint control, with all having an obligation to coordinate the operation and monitor one another. If an error is detected, an intervention must take place, and if no error is detected, the competence of the other person can be trusted.Footnote ⁸⁹ With vertical division of labor, a delegation to surgical staff such as assistants or nursing professionals requires supervisory activities such as selection, instruction, and monitoring. The important point here is that whether supervision is horizontal or vertical, the applicability of the principle of trust is not predicated upon constant control.Footnote ⁹⁰

So far, the principle of trust has only been applied to the behavior of human beings. This chapter argues that the principle of trust should be applied to surgical robots, when lex artis requires it. First, as a general principle, delegation of certain activities must be permitted. Surgeons cannot perform an operation on their own, as this would, in itself, be a mistake in treatment.Footnote ⁹¹ Second, regarding robots in particular, given the degree to which surgical robots offer better surgical treatment, surgeons should use them as part of the expected standard of medical treatment.

But can robots, even certified robots, be equated with another human in terms of trustworthiness? Should a surgeon trust the functioning of a robot, and in what situations is trust warranted? The chapter argues that a variation of the principle of trust should be applied to a surgeon’s use of surgical robots. Specifically, an exception to the non-application of the principle of trust for robots should be created for robots that have been certified by competent authority as safe, referred to here as certification-based trust. Before and until the certification is awarded, the principle of mistrust (Misstrauensgrundsatz) should apply. This approach would also impose greater responsibility on the surgeon if, e.g., the robot used by the surgeon was still in a trial phase, or had a lower level of approval from the relevant authorities.Footnote ⁹²

The concept of certified-based trust is supported by the principle of permissible risk. It is a fact that people die in the operating room, because medical and surgical procedures are associated with a certain degree of risk to health or life, but in Switzerland, this is included in the permissible risk.Footnote ⁹³ There is no reason why this level of acceptable risk should not apply to surgical robots. According to Olaf Dössel:Footnote ⁹⁴

A certification-based trust approach is also consistent with other current practices, e.g., cooperating with newcomers in a field always requires a higher duty of care. When the reliability and safety of surgical robots becomes sufficiently established in practice, the principle of trust should then be applied, to establish the surgeon’s due diligence obligations within the correct parameters.

III.E Certified for Trust

This chapter argues that surgeons working with surgical robots can develop a legitimate expectation of trust consistent with principles of due diligence if the robot they use is certified. This approach to surgeon liability places increased importance on the process of the medical device certification, which is discussed further here.

Certification of medical devices is a well-developed area. In addition to the TPAFootnote ⁹⁵ and the Medical Devices Ordinance,Footnote ⁹⁶ other standards apply, including Swiss laws and ordinances, international treaties, European directives, and other international requirements.Footnote ⁹⁷ These standards define the safety standards for the production and distribution of medical devices.Footnote ⁹⁸

Swiss law requires that manufacturers keep up with the current state of scientific and technical knowledge, and comply with applicable standards when distributing the robot.Footnote ⁹⁹ Manufacturers of surgical robots must successfully complete a conformity assessment procedure in Switzerland.

A robot with a CE-certification can be placed on the market in Switzerland and throughout the European Union.Footnote ¹⁰⁰ A CE-certification mark means that a product has been “assessed by the manufacturer and deemed to meet EU safety, health and environmental protection requirements.”Footnote ¹⁰¹ For the robot to be used in an operating room in Switzerland, a CE-certificationFootnote ¹⁰² must be issued by an independent certification body.Footnote ¹⁰³ After introducing the robot to the market, the manufacturer remains obliged to check its product.Footnote ¹⁰⁴

This chapter argues that a surgeon’s due diligence obligations when using a surgical robot should be limited by a principle of trust, and that the principle should apply when the robot is certified. A certification-based trust approach is consistent with Dössel’s suggestion that trust in technology is well-founded if, inter alia, the manufacturer has professionally designed, constructed, and operated the machinery.Footnote ¹⁰⁵ It is currently not an accepted point of law that the CE-certification is a sufficient basis for the user to trust the robot and not be held criminally responsible, but the chapter suggests that as a detailed, well-established standard, the CE-certification is an example of a certification that could form the basis of application of the principle of trust.

If the principle of certification-based trust is adopted, the surgeon would still retain other due diligence obligations, including the duty to inform patients about the risks involved in a robot’s use.Footnote ¹⁰⁶ This particular duty will likely become increasingly important over time, as the performance range of surgical robots increases.

II.A Approximating the Responsibilities of Machines and Legal Persons

As robots can be made to look and act more and more like humans, the idea of approximating their movements to human acts becomes more plausible – which might pave the way to attributing the notion of actus reus to robots’ activities. By the same token, robots’ ways of processing information and turning it into a motive for getting active may approach the notion of mens rea. The law might, as Ryan Abbott and Alex Sarch have argued, “deem some AIs to possess the functional equivalent of sufficient reasoning and decision-making abilities to manifest insufficient regard” of others’ protected interests.Footnote ¹¹

Probably the most sophisticated argument to date in favor of robots’ criminal responsibility has been advanced by Monika Simmler and Nora Markwalder.Footnote ¹² These authors reject as ideologically based any link between the recognition of human free will and the ascription of culpability;Footnote ¹³ they instead subscribe to a strictly functionalist theory of criminal law that bases criminal responsibility on an “attribution of freedom as a social fact.”Footnote ¹⁴ In such a system, the law is free to “adopt a concept of personhood that depends on the respective agent’s capacity to disappoint normative expectations.”Footnote ¹⁵ The essential question then becomes “whether robots can destabilize norms due to the capacities attributed to them and due to their personhood and if they produce a conflict that requires a reaction of criminal law.”Footnote ¹⁶ The authors think that this is a probable scenario in a foreseeable future: robots could be “experienced as ‘equals’ in the sense that they are constituted as addressees of normative expectations in social interaction like humans or corporate entities are today.”Footnote ¹⁷ It would then be a secondary question in what symbolic way society’s disapproval of robots’ acts were to be expressed. It might well make sense to convict an AI device of a crime – even if it lacks the sensory, intellectual, and moral sensibility of feeling the impact of any traditional punishment.Footnote ¹⁸ Since the future is notoriously difficult to foresee, this concept of robots’ criminal responsibility can hardly be disproved, however unlikely it may appear today that humans could have normative expectations of robots and that disappointment of these expectations would call for the imposition of sanctions. However, in the brave new functional world envisioned by these authors, the term “criminal sanctions” appears rather old-fashioned, because it relies on concepts more relevant to human beings, such as censure, moral blame, and retribution (see Section II.B).

One recurring argument in favor of imposing criminal responsibility on AI devices is the asserted parallel to the criminal responsibility of corporations (CCR).Footnote ¹⁹ CCR will be discussed in more detail in the following section of this chapter, but it is addressed briefly here because calls for the criminal responsibility of corporations and of robots are reactions to a similar dilemma. In each case, it is difficult to trace responsibility for causing harm to an individual person. If, e.g., cars produced by a large manufacturing firm are defective and cause fatal accidents, it is safe to say that something must have gone wrong in the processes of designing, testing, or manufacturing the relevant type of car. But it may be impossible to identify the person(s) responsible for causing the defect, especially since the companies involved are unlikely to actively assist in the police investigation of the case. As we have seen, harm caused by robots leads to similar problems concerning the identification of responsible humans in the background. Regarding commercial firms, the introduction of CCR, which has spread from the United States to many other jurisdictions,Footnote ²⁰ has helped to resolve the problem of the diffusion of responsibility by making corporations criminally liable for any fault of their officers or even – under the respondeat superior doctrine – of their employees. The main goals of CCR are to obtain redress for victims and give corporations a strong incentive to improve their compliance with relevant legal rules. If criminal liability is imposed on the corporation whenever it can be proved that one of its employees must have caused the harm, it can be expected that corporations will do everything in their power to properly select, train, and supervise their personnel. The legal trick that leads to this desired result is to treat corporations as or like responsible subjects under criminal law, even though everyone knows that a corporation is a mere product of legal rules and therefore cannot physically act, cannot form an intent, and cannot understand what it means to be punished. If applying this fiction to corporations has beneficial effects,Footnote ²¹ why should this approach not be used for robots as well?

II.B Critical Differences

However attractive that idea sounds, one cannot help but note that there exist significant differences between corporations and AI devices. Regarding the basic requirements of criminal responsibility, robots at their present stage of development cannot make free decisions, whereas corporations can do so through their statutory organs.Footnote ²² At the level of sanctioning, corporations can – through their management – be deterred from committing further offenses, they can compensate victims, and they can improve their operation and become better corporate citizens. Robots have none of these abilities,Footnote ²³ although it is conceivable that their performance can be improved through reprogramming, retraining, and special supervision. The imposition of retributive criminal sanctions on robots would presuppose, however, that they can in some way feel punished and can link the consequences visited upon them to some prior malfeasance on their part. Today’s robots lack this key feature of punishability, although their grandchildren may well be imbued with the required sensitivity to moral blame.

The differences between legal persons and robots do not necessarily preclude the future possibility of treating robots as criminal offenders. But the fact that corporations, although they are not human beings, can be recognized as subjects of the criminal law does not per se lend sufficient plausibility to the idea of granting the same status to today’s robots.

There may, however, be another way of establishing criminal responsibility for robots’ harmful actions: corporations that use AI devices and/or benefit from their services could be held responsible for the harm they cause. To make this argument, one would have to show that: (1) corporate responsibility as such is a legitimate feature of the law; and (2) corporations can be held responsible for robots as well as for their human agents.

III.A Should There Be Corporate Criminal Responsibility?

Before we investigate this option, we should reflect on the legitimacy of the general concept of CCR. If that concept is ethically or legally doubtful or even indefensible, we should certainly refrain from extending its reach from holding corporations responsible for the acts of their human employees to holding them responsible for their robots.

Two sets of theories have been developed for justifying the imposition of criminal responsibility of legal persons for the harmful acts of their managers and employees. One approach regards certain decision-makers within the corporation as its alter ego and therefore proposes that acts of these persons are attributed to the corporation; the other approach targets the corporation itself and bases its responsibility on its criminogenic or improper self-organization.Footnote ²⁴ These two theories are not mutually exclusive. For example, Austrian law combines both approaches: its statute on the responsibility of corporations imposes criminal liability on a corporation if a member of its management or its control board committed a criminal offense on the corporation’s behalf or in violation of its obligations, or if an employee unlawfully committed a criminal offense and the management could have prevented or rendered significantly more difficult the perpetration by applying due diligence.Footnote ²⁵

Whereas in the United States CCR has been recognized for more than a century,Footnote ²⁶ its acceptance in Europe has been more hesitant.Footnote ²⁷ In Germany, a draft law on corporate responsibility with semi-criminal features failed in 2021 due to internal dissent within the coalition government of the time.Footnote ²⁸ Critics claim that CCR violates fundamental principles of criminal law.Footnote ²⁹ They maintain that a corporation cannot be a subject of criminal law because it can neither act nor make moral judgments.Footnote ³⁰ Moreover, a fine imposed on a corporation is said to be unfair because it does not punish the corporation itself, but its shareholders, creditors, and employees, who cannot be blamed for the faults of managers.Footnote ³¹

It can hardly be denied that CCR is a product of crime-preventive pragmatism rather than of theoretically consistent legal thinking. The attribution of managers’ and/or employees’ harmful acts to the corporation, cloaked with sham historical dignity by the Latin phrase respondeat superior, is difficult to justify because it leads to a duplication of responsibility for the same crime.Footnote ³² It is doubtful, moreover, whether the moral blame inherent in criminal punishment can adequately be addressed to a legal person, an entity that has no conscience and cannot feel guilt.Footnote ³³ An alternative basis for CCR could be a strictly functional approach to criminal law which links the responsibility of corporations to the empirical and/or normative expectation that they abide by the legal norms applying to their scope of activities.Footnote ³⁴

There exists an insoluble conflict between the pragmatic and political interest in nudging corporations toward legal compliance and the theoretical problems of extending the criminal law beyond natural persons. It is thus ultimately a policy question whether a state chooses to limit the liability of corporations for faults of their employees to tort law, extends it to criminal law, or places it somewhere in between,Footnote ³⁵ as has been done in Germany.Footnote ³⁶ In what follows, I assume that the criminal law version of CCR has been chosen. In that case, the further policy question arises as to whether CCR should include criminal responsibility for harm caused by AI devices used by the corporation.

III.B Legitimacy of CCR for Robots

As we have seen, retroactively identifying the fault of an individual human actor can be as difficult when an AI device was used as when some unknown employee of a corporation may have made a mistake.Footnote ³⁷ The problem of allocating responsibility for robot action is further exacerbated by the black box element in self-teaching robots used on behalf of a corporation.Footnote ³⁸

It could be argued that the responsibility gap can be closed by treating the robot as a mere device employed by a human handler, which would turn the issue of a robot’s harmful action into a regular instance of corporate liability. But even assuming that the doctrine of respondeat superior provides a sufficient basis for holding a corporation liable for faults of its employees, extending that doctrine to AI devices employed by humans would raise additional doubts about a corporation’s responsibility. It may neither be known how the robot’s harmful action came about nor whether there was a human at fault,Footnote ³⁹ nor whether the company could have avoided the employee’s potential malfeasance.Footnote ⁴⁰ It is therefore unlikely that many cases of harm caused by an AI device could be traced back to recklessness or criminal negligence on the part of a human employee for whom the corporation can be made responsible.

Effectively bridging the responsibility gap would therefore require the more radical step of treating a company’s robots like its employees, with the consequence of linking CCR directly to the robot’s malfeasance. This step could set into motion CCR’s beneficial compliance mechanism: if the robot’s fault is transferred by law to the company that employs it, that company will have a strong incentive to design, program, and constantly monitor its robots to make sure that they function properly.

How would a corporation’s direct responsibility for actions of its robots square with the general theories on CCR?Footnote ⁴¹ The alter ego-type liability model based on a transfer of the responsibility of employees to the corporation is not well suited to accommodating activities of robots because their actions lack the quality of blameworthy human decision-making.Footnote ⁴² Transfer of liability would work only if the mere existence of harmful activity on the part of an employee or robot would be sufficient to trigger CCR, i.e., in an absolute liability model. Such a model would address the difficulties raised by corporations using robots in situations where the robot’s behavior is unpredictable; however, it is difficult to reconcile absolute liability with European concepts of criminal justice. A more promising approach to justifying CCR for robots relates to the corporation’s overall spirit of lawlessness and/or its inherently defective organization as grounds for holding it responsible.Footnote ⁴³ It is this theory that might provide an explanation for the corporation’s liability for the harmful acts of its robots; if a corporation uses AI devices, but fails to make sure that they operate properly, or uses a robot when it cannot predict that the robot will act safely, there is good reason to impose sanctions on the corporation for this deficiency in its internal organization. This is true even where such AI devices contain elements of self-teaching. Who but the corporation that employs them should be able to properly limit and supervise this self-teaching function?

In this context, an analogy has been discussed between a corporation’s liability for robots and a parent’s or animal owner’s liability for harm caused by children or domestic animals.Footnote ⁴⁴ Even though the reactions of a small child or a dog cannot be completely predicted, it is only fair to hold the parent or dog owner responsible for harm that could have been avoided by training and supervising the child or the animal so as to minimize the risks emanating from them.Footnote ⁴⁵ Similar considerations suggest a corporation’s liability for its robots, at least where it can be shown that the robot had a recognizable propensity to cause harm. By imposing penalties on corporations in such cases, the state can effectively induce companies to program, train, and supervise AI devices so as to avoid harm.Footnote ⁴⁶ Moreover, if there is insufficient liability for harm by robots, business firms might be tempted to escape traditional CCR by replacing human employees by robots.Footnote ⁴⁷

III.C Regulating and Limiting Robot CCR

Before embracing an extension of CCR from employees to robots, however, a counterargument needs to be considered. The increased deployment of AI devices is by and large a beneficial development, saving not only cost, but also human labor in areas where such labor is not necessarily satisfying for the worker, as in conveyor-belt mechanical manufacturing. Robots do have inherent risks, but commercial interests will provide strong incentives for their companies to control these risks. Adding criminal responsibility might produce an over-reaction, inhibiting the use and further development of AI devices and thus stifling progress. An alternative to CCR for robot malfunction may be for society to accept certain risks associated with the widespread use of AI devices and to restrict liability to providing compensation for harm through insurance.Footnote ⁴⁸ These considerations do not necessarily preclude the introduction of a special regime of corporate liability for robots, but they counsel restraint. Strict criminal liability for robotic faults would have a chilling effect on the development of robotic solutions and therefore does not recommend itself as an adequate solution.

Legislatures should therefore limit CCR for robots to instances where human agents of the corporation were at least negligent with regard to designing, programming, and controlling robots.Footnote ⁴⁹ Only if that condition is fulfilled can it be said that the corporation deserves to be punished because it failed to organize its operation so as to minimize the risk of harm to others. Potential control over the robot by a human agent of the corporation is thus a necessary condition for the corporation’s criminal liability. Mihailis E. Diamantis plausibly explains that “control” in the context of algorithms means “the power to design the algorithm in the first place, the power to pull the plug on the algorithm, the power to modify it, and the power to override the algorithm’s decisions.”Footnote ⁵⁰ But holding every company that has any of these types of control liable for any harm that the robot causes, Diamantis continues, would draw the net wider than “sound policy or fairness would dictate.”Footnote ⁵¹ He therefore suggests limiting liability for algorithms to companies which not only control a robot, but also benefit from its activities.Footnote ⁵² The combination of these factors is in fact perfectly in line with the requirements of traditional CCR, where liability presupposes that the corporation had a duty to supervise the employee who committed the relevant fault and that the employee’s activity or culpable passivity was meant to benefit the corporation.

This approach appropriately limits CCR to corporations that benefit from the employment of AI devices. Even so, liability should not be strict in the sense that a corporation is subject to punishment whenever any of its robots causes harm and no human actor responsible for its malfunction can be identified.Footnote ⁵³ In line with the model of CCR that is based on a dysfunctional organization of the corporation, criminal liability should require a fault on the part of the corporation that has a bearing on the robot’s harmful activity.Footnote ⁵⁴ This corporate fault can consist, e.g., in a lack of proper training or oversight of the robot, or in an unmonitored self-teaching process of the AI device.Footnote ⁵⁵ There should in any event be proof that the corporation was at least negligent concerning its obligation to do everything in its power to prevent robots that work for its benefit from causing harm to others. In other words, CCR for robots is proper only where it can be shown that the corporation could, with proper diligence, have avoided the harm. This model of liability could be adopted even in jurisdictions that require some fault on the part of managers for CCR, because the task of properly training and supervising robots is so important that it should be organized on the management level.

Corporate responsibility for harm caused by robots differs from CCR for activities of humans and therefore should be regulated separately by statute. The law needs to determine under what conditions a corporation is to be held responsible for robot malfeasance. The primary issue that needs to be addressed is the necessary link between a corporation and an AI device. Taking an automated car as an example, there are several candidates for potential liability for its harmful operation: the firm that designed the car, the manufacturing company, the programmer of the software, the seller, and the owner of the car, if that is a corporation. If it can be proved that the malfunctioning of the car was caused by an agent of one of these companies, e.g., the programmer was reckless in installing defective software, that company will be liable under the normal CCR rules of the relevant jurisdiction. Special “Robot CCR” will come into play only if the car’s aberration cannot be traced to a particular human source, for example, if the reason for the malfunction remains inexplicable even to experts, if there was a concurrence of several causes, or if the harmful event resulted from the car’s unforeseeable defective self-teaching. In any of these instances, it must be determined which of the corporate entities identified above should be held responsible.