A Regulatory Challenges Facing Automated Driving Systems and the “Moral Machine” Dilemma

At the outset, the use of terminology and taxonomy must be clarified. There exist various terms that are used to refer to vehicles equipped with different levels of driving automation systems (a generic term that covers all levels of automation), such as self-driving cars, unmanned vehicles, autonomous cars, and automated vehicles. However, for reasons to be elaborated later, this chapter consciously uses “ADSs” – namely, level 3–5 systems as defined by the SAE International’s taxonomy and definitionsFootnote ¹² – to refer to the kinds of driving automation that require only limited human intervention and that more appropriately denote the essence of commonly known terms such as “self-driving cars” or “autonomous vehicles.” Indeed, the inconsistent and sometimes confusing use of terms such as “self-driving cars” or “autonomous vehicles” may lead to problems not only related to misleading marketing practices, mistaken consumer perceptions, and information asymmetry, but also insufficient and ineffective regulatory design. For instance, in the robotics and AI literature, the term “autonomous” has been used to denote systems capable of making decisions and acting “independently and self-sufficiently,”Footnote ¹³ but the use of such terms “obscures the question of whether a so-called ‘autonomous vehicle’ depends on communication and/or cooperation with outside entities for critical functionality (such as data acquisition and collection).”Footnote ¹⁴ Some products may be fully autonomous as long as their functions are executed entirely independently and self-sufficiently to the extent entailed in level 5, while others may depend on external cooperation and connection to work (which may fall under the scope of level 3 or level 4). Yet when the term “autonomous vehicle” is commonly used to refer to level 5, levels 3 and 4, or even all levels of driving automation as defined in various legislation enacted in different states,Footnote ¹⁵ regulatory confusion ensues. Comparable conceptual and practical problems can also be found with the use of “self-driving,” “automated,” or “unmanned” in regulatory discourse.

While ADSs offer many benefits to road safety, economic growth, and transportation modernization,Footnote ¹⁶ myriad regulatory issues – such as safety, testing and certification, liability and insurance, cybersecurity, data flow, ethics, connectivity, infrastructure, and service – must be appropriately addressed.Footnote ¹⁷ First, reducing human errors does not mean that ADSs are free from machine error, especially when the technology continues to grow in complexity.Footnote ¹⁸ A review of recent incidents involving Tesla and Volvo-Uber systems suggests that ADSs may be subject to different standards of care, considering the many new safety threats and consumer expectations for the technology.Footnote ¹⁹ Other commentators also point to cybersecurity and industry risks related to ADSs, given their reliance on data collection, processing, and transmission through vehicle-to-vehicle and vehicle-to-infrastructure communications.Footnote ²⁰ The multifaceted yet under-addressed issues of privacy and personal freedom also call for clearer rules and standards.Footnote ²¹ Issues including the Internet of Things (IoT), 5G networks, and smart city development – which are beyond the scope of this chapter – also play a crucial role in the regulatory discourse surrounding ADSs.Footnote ²² The different risks posed by ADSs and IoT and their consequential interactions with the physical world may have crucial ramifications for international trade and investment law.Footnote ²³

This chapter will not exhaust all of these regulatory issues, but rather focuses on the most controversial, ethical dimension of ADSs. There are concerns about the “crash algorithms” of ADSs, which are the programs that decide how to respond at the time of unavoidable accidents.Footnote ²⁴ Ethical issues stem from the infamous “Trolley Problem,” a classic thought experiment of utilitarianism vis-à-vis deontological ethics introduced in 1967 by Philippa Foot.Footnote ²⁵ It involves a runaway, out-of-control trolley moving toward five people who are tied up and lying on the main track. You are standing next to a lever that can switch the trolley to a side track, on which only one tied-up person is lying. The problem? Would you pull the lever to save five and kill one? What is the right thing to do? In modern times, the advent of ADSs makes the Trolley Problem, once an exercise of applied philosophy, a real-world challenge rather than an ethical thought experiment.Footnote ²⁶ Should ADSs prioritize the lives of the vehicle’s passengers over those of pedestrians? Should ADSs kill the baby, the doctor, the mayor, the jaywalker, or the grandma? Or should ADSs be programmed to reach a decision that is most beneficial to society as a whole, taking into account a massive range of factors? Researchers at the Massachusetts Institute of Technology (MIT) designed scenarios representing ethical dilemmas that call upon people to identify preferences for males, females, the young, the elderly, low-status individuals, high-status individuals, law-abiding individuals, law-breaking individuals, and even fit or obese pedestrians in a fictional, unavoidable car crash.Footnote ²⁷ They collected and consolidated around 40 million responses provided by millions of individuals from 233 jurisdictions and published their results in an article titled “The Moral Machine Experiment.”Footnote ²⁸ How does the world respond to the Trolley Problem? While a general, global moral preference can be found, there exist strong and diverse demographic variations specifically associated with “modern institutions” and “deep cultural traits.”Footnote ²⁹ For instance, respondents from China, Japan, Taiwan, South Korea, and other East Asian countries prefer saving the elderly over the young, while those in North America and Europe are the opposite.Footnote ³⁰

As ADSs cannot be subjectively assessed ex post for blame or moral responsibility, it seems necessary – yet it is unclear how – to design rules to regulate the reactions of ADSs when faced with moral dilemmas.Footnote ³¹ Presumably, ethics as well as cultural, demographic, and institutional factors may play a role in likely heterogeneous regulatory measures that could increase frictions in international trade. From a practical, legalist perspective, different tort systems in varying jurisdictions may also have an anchoring effect on ADS designs.Footnote ³² While the decision at the time of unavoidable accidents has immense legal, economic, and moral consequences, it is predetermined when the algorithms are written and built into ADSs. Algorithms are not objective. Rather, they carry the existing biases and discriminations against minority groups in human society, which are reflected and reinforced by the training data used to power the algorithms.Footnote ³³ Further, algorithms do not build themselves, so they may carry the values and preferences of people who write or train them.Footnote ³⁴ Therefore, ADS manufacturers are increasing exposed to legal and reputational risks associated with these moral challenges.Footnote ³⁵ Governments have not yet addressed these ethical puzzles posed by ADS algorithms.

B Regulatory Initiatives at National and Transnational Levels

One may ask whether there are existing or emerging international standards that can serve as a reference for domestic regulations. What approaches are regulators in different jurisdictions taking to address these issues? This chapter maps out some representative regulatory initiatives that have taken place at both the national and the transnational level and are respectively backed by public, private, and hybrid institutions – without concrete harmonization.Footnote ³⁶

What are the relevant positions of the governments of these countries in the global value chain of automated vehicles? What are their respective regulatory governance strategies in light of concerns related to economic growth, national security, and business competition?Footnote ³⁷ To what extent are these countries competing (or cooperating) with one another to lead the global standard-setting process in various international arenas?Footnote ³⁸ At the national level, crucial questions have largely been left unaddressed. A leader in regulating ADSs, the United States Department of Transportation (US DoT) has been stocktaking and monitoring current ADS standards development activities, including those led by, inter alia, the SAE International, the International Organization for Standardization (ISO), the Institute of Electrical and Electronics Engineers (IEEE), the Federal Highway Administration (FHWA), the American Association of Motor Vehicle Administrators (AAMVA), and the National Highway Traffic Safety Administration (NHTSA), in relation to issues such as cybersecurity framework, data sharing, functional safety, event data recorders, vehicle interaction, encrypted communications, infrastructure signage and traffic, and testing approaches.Footnote ³⁹ While a couple of initiatives might partly touch upon some issues with ethical implications,Footnote ⁴⁰ nothing concrete has been designated to address ADSs’ ethical issues. In the United Kingdom, the British Standard Institution published a prestandardization document based on relevant guidelines developed by the UK Department for Transport and Centre for the Protection of National Infrastructure to facilitate further standardization on cybersecurity.Footnote ⁴¹ Taiwan also set up a sandbox scheme for the development and testing of vehicles equipped with ADSs,Footnote ⁴² and the sandbox is open to a broadly defined scope of experimentation, including automobiles, aircraft, and ships, and even a combination of these forms.Footnote ⁴³

Again, none has been initiated to specifically address the ethical issues of ADSs. The world’s firstFootnote ⁴⁴ concrete government initiative specifically on ADS ethical issues at the moment is the report with twenty ethical rules issued by the Ethics Commission for Automated and Connected Driving, a special body appointed by Germany’s Federal Ministry of Transport and Digital Infrastructure.Footnote ⁴⁵ The report consists of twenty ethical rules for ADSs.Footnote ⁴⁶ Of importance are the ethical rules, which ask that “[t]he protection of individuals takes precedence over all other utilitarian considerations,”Footnote ⁴⁷ that “[t]he personal responsibility of individuals for taking decisions is an expression of a society centred on individual human beings,”Footnote ⁴⁸ and that “[i]n hazardous situations that prove to be unavoidable, the protection of human life enjoys top priority in a balancing of legally protected interests.”Footnote ⁴⁹ In particular, Ethical Rule 8 provides that:

Ethical Rule 9 further prescribes that “[i]n the event of unavoidable accident situations, any distinction based on personal features,” such as age, gender, and physical or mental conditions, “is strictly prohibited.”Footnote ⁵¹ While the ethical rules are not mandatory, they certainly mark the first step toward addressing ADSs’ ethical challenges.Footnote ⁵² It remains to be seen how these ethics rules will be translated into future legislations and regulations in Germany and beyond.Footnote ⁵³

Other relevant initiatives, while not specifically addressing ADS ethical issues, include algorithmic accountability rules (generally applicable to data protection and AI applications) that may inform future regulations. For instance, the European Union’s General Data Protection Regulation (GDPR) sets out rights and obligations in relation to algorithmic explainability and accountability in automated individual decision-making.Footnote ⁵⁴ The European Commission also established the High-Level Expert Group on Artificial Intelligence in 2018, which published the final version of its Ethics Guidelines for Trustworthy Artificial Intelligence in April 2019.Footnote ⁵⁵ At the same time, lawmakers in the United States recently tabled a new bill, the Algorithmic Accountability Act of 2019, which intends to require companies to audit systems based on machine learning algorithms, to examine instances of potential bias and discrimination therein, and to fix any issues found in a timely manner.Footnote ⁵⁶

There have been active and dynamic regulatory initiatives at the transnational level.Footnote ⁵⁷ The United Nations Economic Council for Europe (UNECE)Footnote ⁵⁸ and the 1968 Vienna Convention on Road TrafficFootnote ⁵⁹ have struggled to change the formal rules under their existing framework, given the complexity of the issues, high negotiation costs, and institutional inflexibility.Footnote ⁶⁰ The Vienna Convention was somewhat passive in the development of driving automated systems until an amendment to its Articles 8 and 39 entered into force in March 2016.Footnote ⁶¹ The amendment allows for the transfer of driving tasks from humans to vehicles under certain conditions, lifting the formalistic requirement that a “human” must be in charge of driving tasks.Footnote ⁶² In September 2018, the UNECE’s Global Forum on Road Traffic Safety (WP.1) adopted a resolution to promote the deployment of vehicles equipped with ADSs in road traffic.Footnote ⁶³ This resolution is rather soft and represents an informal approach to guiding Contracting Parties to the 1968 Vienna Conventions on the safe deployment of ADSs in road traffic.Footnote ⁶⁴ In any case, because major ADS players like the United States, China, and Japan are not contracting parties to the Vienna Convention, what will be done under the treaty body may not readily generate direct policy relevance and normative influence at the national level (at least for the moment). A few additional private and hybrid organizations have also been engaging in ADS standard-setting, including the SAE International,Footnote ⁶⁵ the ISO,Footnote ⁶⁶ and the IEEE.Footnote ⁶⁷ Among such standard-setting bodies, the SAE International and the ISO are the most comprehensive, cited, and embraced references. Given the complex and dynamic nature of ADS technologies, the SAE International and the ISO, as informal, private/hybrid bodies with more institutional flexibility, have been able to incorporate their members’ expertise to work together in developing common standards – SAE/ISO standards on road vehicle and intelligent transportation systems.Footnote ⁶⁸ The SAE International further offers the ISO a Secretariat function and services for ISO’s TC204 Intelligent Transport System work.Footnote ⁶⁹ With its transnational scope, domain expertise, and industry support, the SAE International’s standards, especially the recent clarification and definition of the J3016 standard’s six levels of driving automation, serve as the “most-cited reference” for the ADS industry and governance.Footnote ⁷⁰ While there has been progress at the transnational level, these regulatory initiatives have yet to touch upon contentious ethical issues that extend beyond the narrower understanding of road safety of ADS.Footnote ⁷¹

A Public Moral Exception, Technical Regulations, and International Standards

First, import bans on vehicles equipped with ADSs because they are designed and manufactured in a jurisdiction and a manner that reflect a different value set, even if they are reasonable, could violate the national treatment or most favored nation obligations under the GATT. Certainly it would be interesting to see whether vehicles equipped with ADS algorithms that are trained with different data reflecting different cultural and ethical preferences are “like products,”Footnote ⁷⁴ or whether ADSs with “pet-friendly,” “kids-friendly,” and “elderly-friendly” algorithms are like products. How would diverse consumer morals in a given market influence the determination of likeness? The determination of likeness is “about the nature and extent of a competitive relationship between and among the products at issue,”Footnote ⁷⁵ and underlying regulatory concerns “may play a role” only if “they are relevant to the examination of certain ‘likeness’ criteria and are reflected in the products’ competitive relationship.”Footnote ⁷⁶ Given the compliance costs and the distributional role of the global value chain, “even-handed regulation would be found to treat like products less favorably.”Footnote ⁷⁷ Furthermore, to discipline algorithm designs in terms of how source codes are written and what/how training data are fed, WTO members would need to regulate not only the end product, but also the process and production methods, which remain controversial issues in WTO jurisprudence.Footnote ⁷⁸ Nevertheless, even if a violation of Article I or III is found, such measures may well be justified under GATT Article XX(a), namely when they are “necessary to protect public morals” and “not applied in a manner which would constitute a means of arbitrary or unjustifiable discrimination between countries where the same conditions prevail, or a disguised restriction on international trade” – the so-called two-tier test.Footnote ⁷⁹ Most other free trade agreements also contain such a standard exception, allowing parties to derogate from their obligations to protect public morals. Similar clauses can also be found in GATS Article XIV(a)Footnote ⁸⁰ and TBT Agreement Article 2.2. Further examinations include whether the measures are “designed to protect public morals,”Footnote ⁸¹ and whether they are “necessary” based on a weighing and balancing process.Footnote ⁸² Such a process has been the yardstick of the GATT Article XX necessity test, which is, as reaffirmed by the Appellate Body in China – Publications and Audiovisual Products, “a sequential process of weighing and balancing a series of factors,” including assessing the relative importance of the values or interests pursued by the measure at issue, considering other relevant factors, and comparing the measure at issue with possible alternatives in terms of reasonable availability and trade restrictiveness.Footnote ⁸³ Most importantly, the definition and scope of “public morals” can be highly contentious, and WTO adjudicators have embraced a deferential interpretation:

More recently, the Appellate Body in EC–Seal Products also emphasized that WTO members must be given some scope to define and apply the idea of “public morals” pursuant to their own systems and values.Footnote ⁸⁵ Given this deferential approach, WTO members appear to enjoy ample leeway in defining and applying public moral-based measures according to their own unique social systems and communal values.

Further, because the TBT Agreement cumulatively applies in conjunction with the GATT, an ADS regulatory measure that is justified may still violate the TBT Agreement, which similarly contains nondiscrimination obligations but lacks public moral exceptions. According to Trachtman, “the scope of the TBT national treatment requirement has been interpreted somewhat narrowly compared to that of GATT, excluding from violation measures that ‘stem exclusively from a legitimate regulatory distinction,’ in order to avoid invalidating a broader scope of national technical regulations than the GATT.”Footnote ⁸⁶ Under Articles 2.1 and 2.2 of the TBT Agreement, ADS regulatory measures are required to be sufficiently “calibrated” to different conditions in different areas, and to not be “more trade-restrictive than necessary to fulfill a legitimate objective, taking account of the risks non-fulfillment would create.”Footnote ⁸⁷ That is, similarly to the jurisprudence in the GATT, a holistic weighing and balancing process taking into account the degree of contribution, levels of trade restrictiveness, and the risks of non-fulfillment of the stated objectives as well as a comparison with possible alternatives are mandated.Footnote ⁸⁸ As will be demonstrated next, the necessity of regulatory measures that focus on mandatory disclosure of source codes and training data (both the substance and form of the regulation) may be fiercely challenged; at the same time, locating a reasonably available alternative can be equally problematic.

Given the transnational regulatory initiatives, Article 2.4 of the TBT Agreement also plays a crucial role here. WTO members are required to use the standards developed by the SAE/ISO and UNECE (so long as they are “relevant international standards”) as the bases for domestic regulations unless such standards cannot effectively or appropriately fulfill the legitimate objective of protecting public morals in the ADS issue area.Footnote ⁸⁹ While this may impose certain (albeit weak) restrictions on the regulatory autonomy and flexibility of WTO members when designing and imposing their ADS algorithm rules and standards in the ethical dimension,Footnote ⁹⁰ the implausible (if not impossible) global consensus on ethical decision-making means that such international standards remain far out of reach. In the long run, there might be more and more initiatives of international standards in this regard, potentially resulting in concerns over the structure, process, and participation in a standard-setting body as well as political confrontations at the TBT Committee.Footnote ⁹¹

B Automated Driving System Algorithms, Source Codes, and Training Data as “Undisclosed Information” under the TRIPS Agreement

Even if the substance of ADS regulatory measures does not violate existing obligations under the GATT and TBT Agreement, WTO members may require ADS manufacturers to disclose their algorithms designs, source code, and training data to verify compliance and achieve their regulatory objectives. If WTO members force ADS vehicle manufacturers or programmers to disclose their trade secrets – proprietary algorithm designs, source codes, and training data – can they survive the test of the TRIPS Agreement? To be sure, entities that own ADS algorithms can seek protection via various channels including patents, copyrights, and trade secrets.Footnote ⁹² However, the commercial practice in the ADS field (and many other AI applications) has been to hold both source code and training data as trade secrets to maximize the protection of interests and to remain competitive in the market.Footnote ⁹³

Article 39.1 of the TRIPS Agreement requires members, when “ensuring effective protection against unfair competition as provided in Article 10bis of the Paris Convention (1967),” to “protect undisclosed information in accordance with paragraph 2.”Footnote ⁹⁴ Article 39.2 further provides that information – when it is secret (not “generally known among or readily accessible to persons within the circles that normally deal with the kind of information in question”), has commercial value, and is controlled by the lawful custodian – shall be protected “from being disclosed to, acquired by, or used by others without their consent in a manner contrary to honest commercial practices.”Footnote ⁹⁵ This requires WTO members to provide minimum protections for undisclosed information, recognized in Article 1.2 as a category of intellectual property,Footnote ⁹⁶ in accordance with the conditions and criteria provided in Article 39.2.Footnote ⁹⁷

Article 39 does not explicitly prohibit members from promulgating laws, consistent with other provisions of the TRIPS Agreement, to allow lawful disclosure or create exceptions where trade secrets may lawfully be forced to be disclosed. Yet what may constitute a lawful disclosure under the TRIPS Agreement can also be controversial. Can members promulgate any law that requires disclosure of trade secrets to serve certain regulatory objectives? Are all measures regulating ADSs and requiring disclosure of source code and training data for conformity assessment lawful and consistent with the TRIPS Agreement? There has been no case law related to Article 39, but the fact that the United States’ proposal to include “theft, bribery, [and] espionage” of secrets in “a manner contrary to honest commercial practices”Footnote ⁹⁸ was rejected in the negotiation process indicates that what may constitute a lawful disclosure can also prove contentious.Footnote ⁹⁹ A contextual reading of TRIPS Agreement Articles 7 and 8 suggests that “Members may … adopt measures necessary to … promote the public interest in sectors of vital importance to their socio-economic and technological development,”Footnote ¹⁰⁰ and “a balance of rights and obligations”Footnote ¹⁰¹ is called for, but such measures cannot “unreasonably restrain trade.”Footnote ¹⁰² The scope of disclosure, the regulated entities, the manner of disclosure, and enforcement and safeguard may therefore be crucial factors in determining consistency. In this sense, in China’s social credit scenario, a limited approach that requires essential source code and training data (from companies that program the algorithms making ethical decisions, instead of all of the actors along the global ADS supply chain) to be disclosed to an expert committee (or similar institutional designs)Footnote ¹⁰³ for review and certification, rather than a wholesale, systematic forced disclosure, may appear to be more TRIPS-consistent. Additional safeguards that prohibit government agencies from sharing disclosed proprietary information with others may also help to avoid inappropriate forced technology transfers, unfair competition, and unfair commercial use.Footnote ¹⁰⁴ Relatedly, some recent megaregional free trade agreements (mega-FTAs) have included provisions that explicitly prevent governments from demanding access to an enterprise’s proprietary software source code.Footnote ¹⁰⁵ Demands for stronger protection of source code and training data and limitations on governments’ regulatory room for maneuver are likely to grow in the age of AI.

C “Algorithmic Black Box” and the Limits of Regulatory Measures

An additional layer of regulatory challenge that may prevent the effectiveness (therefore necessity) of these measures stems from the technological nature of machine/deep learning algorithms – its opaque characteristic, or as criticized by a leading commentator, the “black box” problem.Footnote ¹⁰⁶ This problem refers to the complexity and secrecy of algorithm-based (especially deep learning-based) decision-making processes, which frustrates meaningful scrutiny and regulation. Without understanding and addressing the black box challenge, it may be unrealistic to rely on disclosure or source codes as a regulatory approach. The black box problem can further be disentangled into “legal black box” and “technical black box.”Footnote ¹⁰⁷ The “legal black box” is opaque because of the proprietary status of complex statistical models or source codes, as they are legally protected by trade secret laws.Footnote ¹⁰⁸ Regulatory measures focusing on forced disclosure are one way to fix such black box problems by unpacking the algorithms therein to secure a certain level of compliance.

However, the “technical black box,” which arises in applications based on machine/deep learning algorithms, is much more problematic.Footnote ¹⁰⁹ A technically inherent lack of transparency persists as decisions and classifications emerge automatically in ways that no one – even the programmers themselves – can adequately explain in human-intelligible terms why and how certain decisions and classifications are reached.Footnote ¹¹⁰ There exists “no well-defined method to easily interpret the relative strength of each input and to each output in the network” due to the highly nonlinear technological characteristic.Footnote ¹¹¹ Therefore, the measures that are limited to legally forced disclosure can hardly address this technical black box problem. Even if the regulator forces ADS manufacturers to disclose source codes and algorithm designs, the level of compliance may not be effectively ascertained and evaluated. Because of this technical black box problem, regulatory measures designed to disclose source codes and ensure compliance with ethical rules on ADSs (hence the rational nexus between regulatory means and objectives) may be significantly frustrated.