Hostname: page-component-cd9895bd7-p9bg8 Total loading time: 0 Render date: 2024-12-22T16:12:04.242Z Has data issue: false hasContentIssue false
  • English
  • Français

Accountability for the Taking of Human Life with LAWS in War

Published online by Cambridge University Press:  01 December 2023

Esther D. Reed Open the ORCID record for Esther D. Reed [Opens in a new window]
Esther D. Reed*
Affiliation:
University of Exeter, Exeter, England ([email protected])
Rights & Permissions [Opens in a new window]

Abstract

Accountability for developing, deploying, and using any emerging weapons system is affirmed as a guiding principle by the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems. Yet advances in emerging technologies present accountability challenges throughout the life cycle of a weapons system. Mindful of a lack of progress at the Convention on Certain Conventional Weapons since 2019, this essay argues for a mechanism capable of imputing accountability when individual agent accountability is exceeded, forensic accountability unreliable, and aspects of political accountability fail.

Keywords

accountabilityLAWSlethal autonomous weapon systemsimputationConvention on Certain Conventional WeaponsCCWethics
Type
Roundtable: Global Governance and Lethal Autonomous Weapon Systems
Information
Ethics & International Affairs , Volume 37 , Issue 3 , Fall 2023 , pp. 299 - 308
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
Copyright © The Author(s), 2023. Published by Cambridge University Press on behalf of Carnegie Council for Ethics in International Affairs

Accountability for developing, deploying, and using any emerging weapons system was affirmed as a guiding principle by the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems.Footnote 1 Yet advances in emerging technologies present accountability challenges throughout the life cycle of a weapons system. Mindful of a lack of progress at the Meeting of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons since 2019, when the Guiding Principles were affirmed, this essay argues for a mechanism capable of imputing accountability when individual agent accountability is exceeded, forensic accountability unreliable, and aspects of political accountability fail.Footnote 2 It offers a working definition of accountability and builds an ethical case for imputed accountability to be incorporated practicably as a priority for policy-making in global governance. Assuming an ethical baseline that human beings should be able to demand a reckoning from other human beings for killings, the challenge is to make practicable the requirement for humans to remain at all times accountable for the use of lethal force. The concept of imputed accountability is shown to be needed in order to link together two broad categories of ethical judgment with lethal autonomous weapon systems (LAWS) deployed during war: the criterion of legality under international law and the locus of culpability for violations and omissions to fulfill obligations under the law.

This essay begins by discussing a general understanding of accountability and proceeds to show the limitations of agential accountability, forensic accountability, and political accountability. It argues that these are insufficient to capture the type of accountability that is needed for LAWS. Instead, it proposes the need for imputed accountability as a way of identifying who ought to be accountable for actions that take place outside of the existing limited framework.

Agent Accountability

Accountability is a particular form of responsibility: “the obligation to explain one's action to a forum.”Footnote 3 To be accountable is to be required to stand answerable before other(s) for something. Agent accountability is the view that a human agent is required to render an account within a governance framework for actions performed (or not performed). In criminal jurisprudence, agent accountability entails the relationship of the agent (whether an individual person, team, recognizable social group, company, or other such entity) within a context that yields an expectation of account-giving. Anthony Duff describes a triadic relationship in which agent A may be held answerable before the law for action X done to person B, by virtue of both A and B being members of a unit of accountability, typically a nation-state.Footnote 4 As a form of “backward-looking responsibility,” which provides an account of events after their occurrence, agent accountability presupposes a normative framework within which judgment is made.Footnote 5 Agent accountability can extend to actions or failures beyond commissions or omissions by the individuals in question. Companies and managers can be held liable, for instance, for accidents in their workplaces or organizations if they did not reasonably ensure compliance with relevant safety and health codes.

The importance of agent accountability for the taking of human life is recognized across the world's many cultures, religions, and philosophies. The Hebrew Bible establishes universal standards for answerability when blood is shed (Gen. 9:5–6). The requirement is for a reckoning (in Hebrew, adrsh) for bloodshed at the hand of every human being. The verb dāraš, meaning to enquire or seek with care, is repeated in the chapter to increase the intensity of an investigative and insistent demand. The Qur'an requires respect for human life because the murder of one is an insult to all: “It is as though he slew all men” (Qur'an 5:2). Russian Orthodox Christians are taught to regard murder as arising from “the darkened state of the human heart.”Footnote 6 Daoism has its views of murder, crime, and punishment within the cosmic lifeforce known as qi.Footnote 7 Qi principles maintain stability and harmony in society by exacting punishment upon wrongdoing.Footnote 8 International human rights law recognizes the right to life (Art. 3, Universal Declaration of Human Rights), and rules of customary international humanitarian law prohibit the “arbitrary deprivation of the right to life.”Footnote 9 How then should agent accountability be approached in the light of such widespread and broadly similar recognition of the requirement?

As a religious organization that participates at the Convention on Certain Conventional Weapons (CCW), the Holy See set out standards for human agent accountability for LAWS at war during the Sixth Review Conference of the High Contracting Parties in December 2021. The Holy See outlined a normative and operational framework for LAWS centered around adequate, meaningful, and consistent human supervision.Footnote 10 Insistent that LAWS can never be morally accountable for their actions, the Holy See called for human supervision sufficient to preserve ethical principles, able to guide the use of weapon systems, and consistent such that LAWS would never “have the capacity to contradict what the human authority has prescribed as the main purpose or result of its intervention.”Footnote 11 While the authority level of this CCW contribution falls below that of an apostolic declaration or encyclical, the Holy See's intervention on the international stage clarifies, in effect, what is necessary for “a reckoning for bloodshed” (Gen. 9:5–6). At its simplest: decision-making about the taking of human life should not be delegated to a weapon system but maintained as a matter of human agent accountability.

Interpreting and applying this standard is far from simple, however, as LAWS vary in design and capability, as politico-strategic and tactical pressures build, and as the places and types of accountable decision-making multiply. Features available to LAWS mean that decisions about how they perform extend across their life cycle of creation and use, from commissioning and procurement, through industry interpretation of user needs and technical development, to postmission analysis and review. Accountability decisions about the taking of human life precede and follow choices made during use. As characterized by Floridi and Sanders, the available features of artificial agency include: (1) interactivity, whereby the environment and systems can act upon each other to change an output or previously made decision; (2) autonomy, as the ability, without reference to a human, to “perform internal transitions” that change its state; and (3) adaptability, whereby LAWS “learn their own mode of operation” in a way that depends critically on experience.Footnote 12 These features require restrictive decision-making at the design and readying phases if international humanitarian law (IHL) violations are to be avoided; that is, to prevent a lethal autonomous weapons system from adapting and “learning” new targets during use that IHL prohibits. Unless trained and restricted to preclude unacceptable unpredictability, no human agent accountability on the battlefield can compensate for decisions made (or not made) elsewhere.

Forensic Accountability

As we have seen, an explanation of accountability for taking human lives cannot be limited to agent accountability in a world in which LAWS exist. Instead, accountability needs to include something more. An additional type of explanation is “forensic accountability,” which for our purposes denotes the multifaceted nature of accountability that has, at its core, an obligation to inform and justify one's conduct to an appropriate authority. In AI, this kind of account is widely recognized in engineering and industry as an indispensable imperative for the professions involved inter alia in the design and eventual use of LAWS: “Accountability is a cornerstone of the governance of artificial intelligence (AI).”Footnote 13 Novelli, Taddeo, and Floridi define “accountability in AI” as “a relation of answerability requiring authority recognition, interrogation, and limitation of power.”Footnote 14 They specify forensic accountability through seven features: context, range, agent, forum, standard, process, and implications, and further discuss a set of values, practices, measures, and goals for an accountability “architecture” that accountability may serve in a governance framework. Within such a framework, forensic accountability is integral to juridical accountability whereby the deployment of LAWS would be subject to investigation by appropriate professionals, such as judges and prosecutors, within a court or tribunal system.Footnote 15

Forensic accountability can be derived as an obligation owed by states to the international community as outlined in Article 36 of Protocol Additional to the Geneva Conventions of August 12, 1949 (Protocol I), which puts a High Contracting Party under the obligation to determine whether the development, acquisition, or adoption of a new weapon, or whether the means or method of warfare, would, “in some or all circumstances, be prohibited by this Protocol or by any other rule of international law applicable to the High Contracting Party.”Footnote 16 States must ensure that all weapon systems, including LAWS, comply with Article 36 and other requirements of IHL and conduct weapons review processes.Footnote 17 Regrettably, as Natalia Jevglevskaja summarizes, “Article 36 has been marked by poor implementation.”Footnote 18 While some states meet their obligation by deploying independent delegations for international law monitoring of arms projects, a majority do not have such delegations or other review mechanisms in place.Footnote 19

Such weaknesses in forensic accountability at the state level, combined with lack of progress at the CCW toward a binding legal measure—due to the absence of support from a minority of states, notably the United States, United Kingdom, China, and Russia—pushes companies and industries to self-regulate and explore these ethical questions themselves.Footnote 20 In a blog post on its website, the defense technology company QinetiQ asks, “As a community of innovators, how great is our share of the responsibility for ensuring the ethical development of defence technology?” The post acknowledges a tension in defense between taking sufficient time to consider the ethical implications of new technologies and deploying new capabilities quickly and decisively enough to maintain the advantage over adversaries. “This tension is growing in the 21st century as technology advances at an ever-increasing rate; becoming more powerful, diverse, and accessible.”Footnote 21 Similarly, aerospace and defense company SAAB publishes a podcast that featured an episode looking at the rights and wrongs of autonomous systems.Footnote 22

In the absence of clear governance, military ethicist George Lucas concluded his book Law, Ethics and Emerging Military Technologies: Confronting Disruptive Innovation, which focuses on the ethical challenges of emerging military technologies, with eleven baseline guiding principles of voluntary behavior aimed at incorporating ethical concerns into clear engineering design specifications and operational outcomes.Footnote 23 REAIM 2023, or the Summit on Responsible Artificial Intelligence in the Military Domain, is an international conference focused on the use of AI in military technologies that exemplified good multistakeholder engagement, including government representatives.Footnote 24 Given lack of progress at the CCW since 2019, however, states and companies will inevitably adopt highly variable interpretations of Article 36 and industry principles. Calls for adequate, meaningful, and consistent forensic accountability will be interpreted and applied differently across states, industry, locations, and cultures.

Political Accountability

The failure to achieve consistent forensic accountability at the international level can be charged to a failure of political accountability more broadly. Political accountability is an essentially contestable notion. Since its practicable meaning “depends on the resolution of contestable normative issues[,] disagreements are therefore to be expected.”Footnote 25 Political accountability with respect to weapons control nationally and internationally concerns the functioning of institutions obligated to ensure the law, procedures, and oversight mechanisms that forensic accountability requires.Footnote 26 Agents owing political accountability are those with responsibility nationally or internationally for continuous lawmaking in the face of technological advance, most notably the High Contracting Parties to the Convention on Certain Conventional Weapons and their Governmental Experts. For our purposes, political accountability denotes a duty owed by parties to the CCW to fellow parties and the global citizens whom they represent for expectation setting and legal provision for forensic accountability.

Of concern is how political accountability falters when relationships of trust are absent among those obligated both to devise and to operate according to the relevant rules, since many states have self-interested reasons to avoid governance.Footnote 27 Witness the lack of congruence between the Contracting Parties to the CCW, which threatens the meaningfulness of political accountability among them. “It's b****hit and they know it's b****hit . . . because it's the same, every time they come out with this rubbish about what consensus means for bringing people together and building bridges,” remarked Richard Moyes, managing director of the nonprofit organization Article 36, on January 25, 2022, following the CCW Review Conference.Footnote 28 Some states, Moyes argued, want the breakdown of norm setting and to maintain a failure to attain granularity in the shared understanding of how international law applies, because a loose legal framework permits an unrestrained quest for competitive advantage: “The CCW mode of work allows them to prevail.” The CCW can, of course, exercise its authority only on the sufferance of its members; it is dependent wholly on “the common will of states to act in concert.”Footnote 29 In the absence of this common will, states cannot routinely rely upon the international community to hold states to account around agent responsibility and the conduct of forensic accountability. The international community faces, at least potentially, a modern variant of iudicium cessans  (judgment unavailable) whereby accountability mechanisms are absent and weapons control degenerates into state-specific or regional standards and regulatory requirements, or what is in effect a global “free-for-all.”Footnote 30

Imputed Accountability

This brings us to a crucial point in our consideration of accountability for the taking of human life with LAWS at war. If competition between states in the development, acquisition, and adoption of new weapons technologies is not to become a global free-for-all, the above-discussed types of accountability must be supplemented with an accountability that not only describes the state or condition of being required to render an account but also links together two broad categories of ethical judgment in security and defense: the legality of LAWS under international law and the locus of culpability for failures with respect to accountability.

The necessary accountability function may be denoted “imputation,” where imputation means “to put something on someone's account”—whether a condemnable action, fault, or good action. To impute accountability is to hold an agent(s) accountable for actions that they might not have committed themselves but are yet liable for.Footnote 31 Imputation places a violation of the law on the account, so to speak, of someone who can be deemed culpable, or some organization or institution that may bear collective moral and potentially legal guilt for acts of omission or commission. “Imputation serves to link a particular action with all its ethical weight to a particular agent.”Footnote 32 Imputed accountability does not substitute for substantive principles of the kind developed by the Holy See, but is a way of recognizing that human accountability with LAWS at war is no longer a matter only of individual agent accountability but is also tied to forensic accountability and aspects of political accountability too. Given that LAWS could so easily become unpredictable, that forensic accountability is variable, and that political institutions are failing, the need for imputed accountability is clear.

No illusions should be harbored that a credible mechanism capable of imputing accountability for regulating LAWS is easily achievable in the international community.Footnote 33 The practical and political challenges are many, including support for them from companies, states, and international governmental organizations. We face the reality that, across the world's many cultures, religions, and philosophies, there is a demand for accountability when human blood is shed, but there are vanishingly small prospects for achieving it. Many of the world's religions appeal to divine judgment at such a juncture. Some might recall the prophet Amos thundering against the nations for idolatry, military aggression, economic exploitation, and inhumanity to the vulnerable: “For three transgressions . . . and for four, I will not revoke the punishment” (Amos 2:3, 6, 9, 13). As one of many biblical voices expressing divine judgment, Amos was sensitive to the complex business and national dimensions of accountability that could not be reduced to individual fault: “I will cut off the ruler from its midst, and will kill all its princes [Hebrew, sar; also meaning ‘merchants’ or ‘officials’] with him” (Amos 2:3). Divine judgment extended beyond the rulers of the people to traders and administrators. The belief being expressed here is that God will impute accountability at the final judgment, if not before. In modern secular contexts, where accountability as imputation is needed but seemingly unattainable, we find ourselves facing an accountability gap and are without means of negotiating it.

At the end of the day, accountability for the taking of human life is more of a human and political challenge than a technical one. A functioning CCW, robust weapons reviews by states, public interest in such matters, and vibrant nonprofit and campaigning organizations all have a role to play and can help with the drafting of standards and pushing for legislation. The ethical question for all is whether and how much accountability for the taking of human life is wanted. As Aquinas argued, action follows choices, which are judgments of reason about what is to be done. Judgments of reason are influenced by what is perceived as desirable or good. Wanting matters.Footnote 34 The ethical challenge is to want accountability in ways that drive action toward achieving it. Wanting accountability generates inclinations toward those things that, while perhaps not immediately sufficient to generate change, form habits that tend toward the production of attractive and sustainable benefits. With the development of LAWS, when individual agent accountability for the taking of human life is exceeded, forensic accountability is unreliable, and aspects of political accountability fail, all relevant modes of accountability are needed, including accountability imputed.

References

Notes

1 Annex 3(d), “Guiding Principles Affirmed by the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons System,” in Final Report, CCW/MSP/2019/9, December 13, 2019 (Meeting of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, Geneva, November 13–15, 2019) digitallibrary.un.org/record/3856241?ln=en.

2 NATO has an AI strategy and is developing an AI certification standard.

3 Verdiesen, Ilse, de Sio, Filippo Santoni, and Dignum, Virginia, “Accountability and Control over Autonomous Weapon Systems: A Framework for Comprehensive Human Oversight,” Minds and Machines 31, no. 1 (August 2020), pp. 137–63CrossRefGoogle Scholar, in abstract.

4 Duff, Anthony, Answering for Crime: Responsibility and Liability in the Criminal Law (London: Bloomsbury, 2007)Google Scholar.

5 Verdiesen et al., “Accountability and Control over Autonomous Weapon Systems,” pp. 139–40.

6 Department for External Church Relations, Russian Orthodox Church, “The Basis of the Social Concept” 9(1), old.mospat.ru/en/documents/social-concepts/ix/.

7 Livia Kohn, “Daoism,” in Encyclopedia of Crime and Punishment, Sage Knowledge, 2002, ed. David Levenson, sk.sagepub.com/reference/crimepunishment/n118.xml.

8 Zhang, Xiuhua, Yao, Tianyu, Xu, Wenjun, and Zhai, Yujia, “Crime and Punishment in Ancient China and Its Relevance Today,” American Journal of Economics and Sociology 76, no. 5 (November 2017), pp. 1191–1218, at p. 1192CrossRefGoogle Scholar.

9 Henckaerts, Jean-Marie, Doswald-Beck, Louise, and Alvermann, Carolin, “Murder Is Prohibited,” rule 89 in Customary International Humanitarian Law, vol. 2 (Cambridge, U.K.: Cambridge University Press, 2005)CrossRefGoogle Scholar, ch. 32, sec. C.

10 Holy See, “Translating Ethical Concerns into a Normative and Operational Framework for Lethal Autonomous Weapons Systems,” CCW/CONF.VI/WP.3, December 20, 2021 (Sixth Review Conference of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, Geneva, December 13–17, 2021), docs-library.unoda.org/Convention_on_Certain_Conventional_Weapons_-SixthReview_Conference_(2021)/CCW-CONF.VI-WP.3.pdf.

11 Sec. 3(17), in ibid.

12 These descriptors were borrowed from Floridi, Luciano and Sanders, J. W., “On the Morality of Artificial Agents,” Minds and Machines 14, no. 3 (2004), pp. 349–379CrossRefGoogle Scholar, at p. 357.

13 Claudio Novelli, Mariarosaria Taddeo, and Luciano Floridi, “Accountability in Artificial Intelligence: What It Is and How It Works,” AI & Society (February 2023), in abstract, link.springer.com/article/10.1007/s00146-023-01635-y. See also sec. 7.

14 Ibid., sec. 7.

15 For a similar working definition, see Jeremy Waldron, “Accountability: Fundamental to Democracy” (Working Paper No. 14-13, New York University School of Law, Public Law & Legal Theory Research Paper Series, April 2014), papers.ssrn.com/sol3/papers.cfm?abstract_id=2410812.

16 Diplomatic Conference on the Reaffirmation and Development of International Humanitarian Law Applicable in Armed Conflicts, Art. 36, “New Weapons,” in “Protocol Additional to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of Non-International Armed Conflicts (Protocol I),” June 8, 1977, www.ohchr.org/en/instruments-mechanisms/instruments/protocol-additional-geneva-conventions-12-august-1949-and. See also United Nations General Assembly, “Final Report of the Conference to the General Assembly,” A/Conf.95/15 (United Nations Conference on Prohibitions or Restrictions of Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, October 27, 1980). For background information on this CCW report, see United Nations General Assembly, “A Comparative Study of the Scope Originally Proposed or Aimed at in Draft Multilateral Disarmament Treaties of a Universal Character Concluded under United Nations Auspices and the Scope Finally Fixed in Those Treaties, including the Contemplated Measures for Expanding That Scope,” A/AC.206/10, June 16, 1981; and Justin McClelland, “The Review of Weapons in Accordance with Article 36 of Additional Protocol I,” International Review of the Red Cross 85, no. 850 (June 2003), pp. 397–420, www.icrc.org/en/doc/assets/files/other/irrc_850_mcclelland.pdf.

17 “What Are Weapons Reviews?,” International Weapons Review, internationalweaponsreview.com/legal-reviews-of-weapons/.

18 Natalia Jevglevskaja, International Law and Weapons Review: Emerging Military Technology under the Law of Armed Conflict (Cambridge, U.K.: Cambridge University Press, 2021), p. 3.

19 For a similar conclusion, see Jacobsson, Marie, “Modern Weaponry and Warfare: The Application of Article 36 of Additional Protocol I by Governments,” International Law Studies 82 (2006)Google Scholar, digital-commons.usnwc.edu/cgi/viewcontent.cgi?article=1231&context=ils.

20 This point is made by Michael T. Klare in “Conference Makes No Progress on Robotic Weapons,” Arms Control Association, January/February 2022, www.armscontrol.org/act/2022-01/news/conference-makes-progress-robotic-weapons.

21 “Technology Ethics in Defence and Security” (from TechWatch 10), QinetiQ blog, March 14, 2022, qinetiq.com/en/blogs/techwatch-deep-dive-tech-ethics.

22 “Autonomous Systems and the Rules of Right and Wrong,” July 2021, Zero Pressure podcast, 50:04, SAAB, saab.com/newsroom/stories/2021/july/the-rights-and-wrongs-of-autonomous-systems.

23 Lucas, George, Law, Ethics and Emerging Military Technologies: Confronting Disruptive Innovation (London: Routledge, 2023), pp. 177–81Google Scholar. See also “Palantir Technologies’ Approach to AI Ethics,” Palantir, March 2, 2023, www.palantir.com/pcl/palantir-ai-ethics/.

24 “REAIM 2023,” Government of the Netherlands, www.government.nl/ministries/ministry-of-foreign-affairs/activiteiten/reaim.

25 Waldron, Jeremy, “Is the Rule of Law an Essentially Contested Concept (in Florida)?,” Law and Philosophy 21, no. 2 (March 2002), pp. 137–64Google Scholar, at p. 148.

26 Waldron, “Accountability,” p. 2.

27 On difficulties conceptualizing trust in international relations, see Hoffman, Aaron M., “A Conceptualization of Trust in International Relations,” European Journal of International Relations 8, no. 3 (September 2002), pp. 375–401CrossRefGoogle Scholar.

28 Richard Moyes, “Article 36 Podcast Episode 8: CCW Review Conference Special Part III,” discussion with Elizabeth Minor, Richard Moyes, and Uldduz Sohrabi, January 25, 2022, Article 36 podcast, 27:57, article36.org/updates/publication/article-36-podcast-episode-8-ccw-review-conference-special-part-iii/. See further Klare, “Conference Makes No Progress on Robotic Weapons.”

29 O'Donovan, Oliver, The Ways of Judgment (Grand Rapids, Mich.: Eerdmans, 2005), pp. 226–27Google Scholar.

30 Pope Francis, Fratelli Tutti: On Fraternity and Social Friendships (Washington, D.C.: United States Conference on Catholic Bishops, 2020), § 36.

31 Imputation and attribution may be used interchangeably.

32 W. David Hall's interpretation of Paul Ricoeur on imputation is helpful: W. David Hall, Paul Ricoeur and the Poetic Imperative: The Creative Tension between Love and Justice (Albany, N.Y.: SUNY Press, 2007), p. 90.

33 Terms like “international community” attract an immense literature and the meaning is heavily contested. The definition I assume is “a fragile achievement in need of continual support,” as described by Alexander Latham-Gambi in “Jeremy Waldron and the Circumstances of Politics” (Review of Politics 83, no. 2 [Spring 2021], pp. 242–63, at, esp., p. 242).

34 “Human actions stem from natural appetites, including the passions and the will, which operate through inclinations towards perceived goods.” Jean Porter, The Perfection of Desire: Habit, Reason, and Virtue in Aquinas's Summa Theologiae (Milwaukee: Marquette University Press, 2018), p. 17.