1.1 Three Cases

1.2 What Is an Algorithm?

The academic literature and wider public discourse about the sorts of systems we have been discussing involve a constellation of concepts such as “algorithms,” “big data,” “machine learning,” and “predictive analytics.”Footnote ²⁷ However, there is some ambiguity about these ideas and how they are related, and any discussion of emerging technologies requires some ground-clearing about the key concepts. There are, however, some general points of overlap in the literature. We won’t attempt to settle any taxonomical debates here once and for all, but we will fix some of the important concepts for the sake of clarity.

Among the key concepts we will use, “algorithm” is among the most important, but its usage also invites confusion. At its most basic, an algorithm is just an explicit set of instructions for solving a problem. The instructions may be for a digital computer, but not necessarily so: a recipe for chocolate chip cookies, a set of instructions for operating a combination lock, and even the familiar procedure for long division are all algorithms. In contrast to this broad concept, we are considering algorithms in terms of their functional roles in complex technological systems.Footnote ²⁸ The term “algorithm” is also ambiguous in this more specific setting. It can be used to refer either to a set of instructions to complete a specific task or to a system that is driven by such algorithms. This distinction makes a difference in patent law. Inventions built upon an abstract mathematical algorithm (such as a special mechanical process for molding synthetic rubber) can be patented, while the algorithm itself (meaning the equations used to guide the process or system) cannot.Footnote ²⁹

Our focus here, however, is algorithms in the more applied, systematic sense. That is, we are concerned with algorithms that are incorporated into decision systems. These systems take a variety of forms. Some are parts of mechanical systems, for example, sensor systems in modern cars that activate warnings (e.g., for obstacles nearby) or control safety features (e.g., emergency brakes). Others are parts of information systems, for example, recommendation systems for videos (e.g., Netflix, YouTube), music (Spotify, Pandora), books (Amazon, Good Reads), and maps (Google maps). Still others are incorporated into complex social structures (supply chain logistics, benefits services, law enforcement, criminal justice). These systems have become ubiquitous in our lives; everything from border security to party planning is now managed by algorithms of one sort or another. When we discuss COMPAS, EVAAS, and the Facebook News Feed in one breath, we are discussing algorithms in this broad sense. Moreover, algorithms in this sense are best understood as constitutive parts of socio-technical systems. They are not purely sets of instructions for carrying out a task and they are not mere technological artifacts. Rather, they are used by individuals and groups and affect other individuals and groups such that they constitute an interrelated system that is both social and technological. For the remainder of the book we will refer to these kinds of systems in several ways, including “automated decision systems,” “algorithmic decision systems,” and (for the sake of terseness) simply “algorithms.”

Another key concept is “big data.” This term is often used to describe any data-mining approach to a problem using large datasets, but this washes over much of what makes such datasets a distinctive ingredient of modern technological systems. Datasets that are “big” in the sense of big data are usually enormous and high dimensional; often they consist of hundreds of thousands of rows and thousands of columns. However, a dataset that is merely big in this sense will not render the statistical magic often discussed under the rubric of predictive analytics. Rather, the systems and datasets that underlie algorithmic decision systems also have a number of other special properties.Footnote ³⁰ These additional properties are often summarized in terms of the “three V’s”: volume, velocity, and variety. In other words, datasets that are big in the relevant sense are not only big in volume. They also have high velocity, meaning that they are often continuously updated or are created in real time, for example, systems offering driving route instructions that are updated to account for traffic conditions. Finally, they are diverse in variety, meaning that they encompass both data that is structured (i.e., organized in a predefined format), in the sense of being organized and comprehensible for analysis, and data that is unstructured (i.e., not organized in a predefined format).

As with the concepts of algorithms and big data, “predictive analytics” is not defined by a well-codified set of rules, systems, or practices. At root, the term describes the application of data-mining techniques in developing predictive models, but it is more than that. Many of the model-building techniques, such as linear regression, are standard statistical methods that have been known for hundreds of years.Footnote ³¹ The characteristic feature of modern predictive analytics is not its use of algorithms or even the size or complexity of its datasets, but rather the analytical possibilities offered by machine learning.

Machine learning involves training computers to perform tasks according to statistical patterns and inferences rather than according to human-coded logical instructions. This approach incorporates different kinds of processes, the broadest categories of which are “supervised” and “unsupervised” learning. Supervised learning is the more straightforward and familiar of the two forms of machine learning. It involves systems that have been trained on large numbers of examples, either for classification (i.e., for classifying future examples) or for regression (i.e., for performing regression analysis). What makes the computer’s learning supervised in these cases is that both classification and regression processes involve a “supervision signal,” which is constructed from training on a set of pre-labeled examples and which defines the desired sort of output in advance. Classification, for instance, involves sorting novel examples into a known set of discrete values (e.g., determining whether a given image is of a cat, a dog, or a rabbit), given a set of pre-labeled training examples. Regression involves predicting some real-valued output (e.g., determining the value of a rental property in a complex market), given some set of examples.

In contrast to supervised learning, unsupervised learning involves analysis using large numbers of examples but lacks a supervision signal. Unsupervised learning algorithms, then, are not given right answers in advance for the purposes of future prediction; rather, they are designed to somehow discern or reduce the deep structure of the (often high dimensional) dataset for explanatory purposes. This can take the form of “clustering,” in which the data is “naturally” grouped according to the distances between its data points, or “dimensionality reduction,” in which the dataset is either compressed or broken down for intuitive visualization. In recent years, these techniques have found applications in data center regulation, social media sentiment analysis, and disease analysis based on patient clustering.

There is widespread recognition that there are ethical issues surrounding complex algorithmic systems and that there is a great deal of work to be done to better understand them. To some extent, concern about these issues is related to beliefs about the potential of unsupervised learning to help realize strong forms of AI.Footnote ³² The reality is more pedestrian.Footnote ³³ Outside of cutting-edge AI labs such as OpenAI or DeepMind, machine learning is mainly a matter of employing familiar techniques such as classification, regression, clustering, or dimensionality reduction, at a big data scale. So rather than grappling with ghosts in machines that have not yet begun to haunt us, we aim to address the practical issues we already face.

1.3 Algorithms, Ethics, and Autonomy

We began this introduction by describing several recent legal disputes. Loomis, Wagner, and Houston Teachers will be polestar cases throughout the book. But at root, this book addresses moral questions surrounding algorithmic decision systems. Whether use of COMPAS violates legal rights is a distinct (though related) question from whether it impinges moral claims. Moreover, the proper scope of legal claims and how the law and legal systems ought to treat algorithmic systems are moral questions. Concerns about algorithmic systems have come from a range of sectors and include guidance from nongovernmental organizations, government agencies, legislators, and academics. For example, the UK’s Nuffield Foundation published a road map for research on ethical and societal implications of algorithmic systems. They argue that there are important conceptual gaps that need to be facilitated by philosophical analysis. In their canvas of various sets of AI principles offered by scientific, engineering, corporate, and government groups, “most of the principles prosed for AI ethics are not specific enough to be action guiding.”Footnote ³⁴ Likewise, they point to a gap in the philosophical literature on ethics in algorithms, data, and AI.Footnote ³⁵

Government entities have also recognized moral concerns and the need for greater research on these issues as well. The US President’s National Science and Technology Council’s 2016 report, “Preparing for the Future of Artificial Intelligence,” outlined a number of ethical concerns surrounding AI and algorithmic systems.Footnote ³⁶ While the report focuses on transparency and fairness, the issues it raises have autonomy implications as well. The Ethics Advisory Group to the European Data Protection Supervisor (EDPS-EAG) issued a report in 2018 outlining a slate of ethical concerns surrounding digital technologies, including algorithmic decision systems. In particular, the advisory group explained the importance of linking foundational values – among them autonomy, freedom, and democracy – to digital technologies. The UK parliament appointed a Lords Select Committee on Artificial Intelligence in 2017 to examine a handful of issues in development and adoption of AI (within which they include algorithmic systems), one of which is “What are the ethical issues presented by the development and use of artificial intelligence?”Footnote ³⁷ Among their recommendations are principles protecting “fairness and intelligibility” and prohibiting automated systems from having the power to “hurt, destroy, or deceive human beings.”Footnote ³⁸ Members of both houses of the U.S. Congress have introduced an Algorithmic Accountability Act that would impose requirements to create impact assessments and address bias and security issues.Footnote ³⁹

The academic literature is also expanding in its criticism of algorithmic systems. Kirsten Martin, for instance, argues that big data generates negative externalities in the form of additional surveillance, which she calls “surveillance as pollution.”Footnote ⁴⁰ Tal Zarsky argues that automated decision-making introduces both efficiency- and fairness-based problems.Footnote ⁴¹ Danielle Citron and Frank Pasquale argue for imposing auditing procedures on the process of algorithmic development on the basis of those problems.Footnote ⁴² Karen Yeung argues that the inoffensive practice of nudging, which classically involves only subtle forms of behavioral modification through manipulation of “choice architectures,” can be galvanized by predictive analytics to produce “hypernudging” platforms whose effects at scale wind up being radically paternalistic.Footnote ⁴³ Cathy O’Neil groups such systems under the banner of “weapons of math destruction,”Footnote ⁴⁴ arguing that they enjoy an aura of epistemic respectability that encourages us to use them beyond their actual capacities. Citron argues that addressing these problems requires nothing short of a new constitutional paradigm – a new “technological due process.”Footnote ⁴⁵

Algorithmic systems (including both predictive systems and digital platforms) have come under substantial economic, political, and philosophical criticism. We agree with much of it. However, for a few reasons we do not defend any overall moral or ethical conclusion about the technologies themselves. First, the fact that they are rapidly advancing as part of an ongoing process means that the horizon for productive commentary on current technology is time delimited. Second, we acknowledge that these technologies – whatever their life spans might be – can be employed for useful aims as well as for pernicious ones. There are few global, all-things-considered moral judgments that can be made about, for instance, the governance of Facebook’s News Feed or use of risk assessment algorithms. Third, we acknowledge that the algorithmic landscape of predictive analytics and digital platforms is here to stay in some form or other. It is possible to exert some influence on how these systems are employed and perhaps even develop new conceptions of fair play to cope with these changes, but predictive analytics and digital platforms will not be eliminated altogether.

For those reasons we aim to look beyond the particular features of the technologies as much as possible, treating the technologies themselves as case studies that are useful for making certain moral and social issues vivid and concrete, rather than as the sources of ontologically distinctive philosophical issues. In many cases, the philosophical issues have more to do with our psychological features and our social structures than with the inherent hazards of technological systems considered abstractly. Hence, what unifies the systems discussed in the book is not strictly about the technologies per se, but rather about the human values that are implicated by the designs and modes of operation of all of those socio-technical systems. All these systems, we argue, raise philosophical issues about autonomy.

Focusing on autonomy is important for several reasons. Primarily, it connects unease about algorithmic systems to a good that has deep, stable, and plausible moral value. This is a rich seam to mine in the same way that autonomy issues have been fundamentally important in other domains of applied ethics. Arguments grounded in autonomy connect concerns about algorithmic systems to an area with a broad and well-developed philosophical literature.

Moreover, by drawing out the importance of autonomy, our account can address concerns about algorithmic decision systems that are not captured by accounts that focus on fairness, harm, and bias. Algorithmic systems often reflect, harden, and create unfair structures; this is an enormous moral concern. However, that is only part of the moral importance of algorithmic systems. Loomis shows why: Loomis was plausibly wronged, but it is not clear that he has been treated unfairly (at least in the sense that COMPAS treats him differently from other, similarly situated defendants) and it does not appear that he has been materially harmed by use of COMPAS. Note, too, that while there are some scholars who have addressed whether certain kinds of algorithmic systems conflict with autonomy via manipulation,Footnote ⁴⁶ our view is that autonomy is grounds for a much broader evaluation of algorithmic systems. Loomis, Wagner, and Houston Schools do not appear to involve manipulation in any strong sense.

Our focus on autonomy also provides a foundation for moral concerns that are often under-explained (e.g., transparency, filter bubbles). Specifically, a focus on autonomy can serve to route around at least some disputes about bias and fairness. It has become clear that there are different ways that a single system can be plausibly unfair to members of different groups.Footnote ⁴⁷ Determining which facets of fairness matter the most requires considering different values. We will argue that an appeal to autonomy can help in that regard. Finally, note that our approach is consistent with other important critiques; that is, a concern about autonomy for the most part adds to rather than contradicts extant critiques.

1.4 Overview of the Book

To get at the different ways in which algorithmic systems bear upon autonomy, we have divided the book into four main parts. Part I is introductory and ground-clearing. Chapters 1 and 2 serve as introductory chapters, outlining the conceptual foundations and philosophical commitments that ground our arguments throughout the book. The primary task of Chapter 2 is providing an account of autonomy and its importance. It begins with a high-level explanation of autonomy itself and then canvasses several key conceptions of autonomy and the philosophical concerns underlying them. We then advance our ecumenical account of autonomy. Specifically, we draw on procedural, psychological accounts of autonomy as well as social accounts of personal autonomy. We argue that while they have important differences in how they explain key facets of autonomy, in practice they are substantially overlapping. Hence, we can draw on both as a foundation for the arguments we make in the book. That is because fully realized autonomy demands both procedural independence (which includes both epistemic competence conditions and authenticity conditions) and substantive independence (which includes conditions of reciprocal support and non-domination from society more broadly).

Part II builds upon our account of autonomy and our polestar cases to address the kinds of moral claims and the nature of the respect that persons are owed in virtue of their autonomy. In Chapter 3 we argue that people are owed systems of rules and practices that they can reasonably endorse. We begin the chapter with a closer consideration of VAMs used for K-12 teacher assessment. We argue that the problem with such tools cannot be reduced to concerns about their reliability or their potential for bias. Rather, teachers have a claim to incorporate their values into their lives as they see fit. And respecting teachers requires recognizing them as value-determiners, neither thwarting nor circumventing their ability to act according to those values without good reason. Moreover, as agents they are capable of abiding fair terms of social agreement (so long as others do too), and hence “good reasons” will be reasons that they can abide as fair terms of cooperation. Teachers can endorse those reasons either as consistent with their own values or as a manifestation of fair social agreement.

We argue that VAMs fail to respect teachers as agents because they are used in a way that teachers cannot reasonably endorse, for four interrelated reasons. First is their reliability, which many have questioned. Second is that their results can be based on factors for which teachers are not morally responsible (e.g., student performance may correlate with teachers’ age, ethnicity, or gender). Third is stakes. The fact that an algorithmic system is unreliable or measures factors for which persons are not responsible is important primarily as a function of the stakes involved. Fourth is the relative burdens placed upon people subject to them. We conclude by applying our framework to our polestar cases (Loomis, Wagner, and Houston Schools).

One important, oft-cited criticism of algorithmic systems is that they lack transparency. Such systems can be opaque because they are complex, protected by patent or trade secret, or deliberately obscure. In the EU, there is a debate about whether the General Data Protection Regulation (GDPR) contains a “right to explanation,” and if so what such a right entails. Our task in Chapter 4 is to address this informational component of algorithmic systems. We argue that information access is integral for respecting autonomy, and transparency policies should be tailored to advance autonomy.

To make this argument we distinguish two facets of agency (i.e., capacity to act). The first is practical agency or the ability to act effectively according to one’s values. The second is what we call cognitive agency, which is the ability to exercise what Pamela Hieronymi calls “evaluative control” (i.e., the ability to control our affective states such as beliefs, desires, and attitudes). We argue that respecting autonomy requires providing persons sufficient information to exercise evaluative control and properly interpret the world and one’s place in it. We draw this distinction out by considering algorithmic systems used in background checks, and we apply the view to our polestar cases.

While Part II of the book considers what we owe people given that they have autonomy, Part III addresses our responsibility to secure the conditions under which people can act autonomously. Chapter 5 considers the relationship between algorithmic systems and freedom. There is substantial dispute about the concept and moral value of freedom. A key area of dispute is whether freedom is best understood in terms of negative, positive, or republican freedom. We offer an account according to which freedom is ecological and includes both republican freedom (which is to say, freedom from others’ exercise of arbitrary power) and positive freedom, properly understood (i.e., where positive freedom is a function of quality of agency). We argue that algorithmic systems in several ways conflict with ecological freedom.

Chapter 6 addresses a specific condition of autonomy from Chapter 2, viz., epistemic competence. It has been clear since the early 2000s that internet communication technologies generally and algorithmically driven information systems in particular create epistemically noxious environments. These include phenomena like filter bubbles and echo chambers as well as more insidious phenomena like toxic recommendation systems.Footnote ⁴⁸ Most every media platform now employs content moderation systems to screen for the truly awful content that would make sites like YouTube, Facebook, Reddit, and others unnavigable.Footnote ⁴⁹ That practice is relatively uncontroversial. We argue that there are further obligations on sites to exercise a kind of epistemic paternalism.

While Parts II and III begin from the premise that people have certain claims as agents (specifically, claims to the conditions that foster autonomy), Part IV shifts focus to the obligations of agents. Chapter 7 considers the autonomy and responsibility of those who deploy information technologies (as collectors of big data, users of algorithmic decision systems, developers of social media sites, and so on). Specifically, we argue that there is a type of wrong that arises when autonomous agents obscure responsibility for their actions, which we call “agency laundering.” At root, agency laundering involves a failure to meet one’s moral responsibility for an outcome by attributing causal responsibility to another person, group, process, or technology, and it does so by undermining a key component of responsibility itself, viz., accountability. We apply our conception of agency laundering to a series of examples, including Facebook-automated advertising suggestions, Uber driver interfaces, as well as to our polestar cases.

We then turn to the ways in which autonomy underwrites democratic governance. Political authority, which is to say the ability of a government to exercise power, may be justifiable or not. Whether it is justified and how it can come to be justified is a question of political legitimacy. In Chapter 8 we consider several views of legitimacy and argue for a hybrid version of normative legitimacy based on one recently offered by Fabienne Peter.Footnote ⁵⁰ We explain that each facet of the hybrid requires a legitimation process that is itself grounded in autonomy. We argue that the autonomy view is a basis for criticism of the legitimation processes related to a predictive policing technology and to actions of Cambridge Analytica and the Internet Research Agency in recent elections. In Chapter 9 we offer some conclusions and caveats.

1.5 A Heuristic

It is worth stepping back for a moment before launching into the main substance of the book’s arguments in order to explain its purpose and approach. This book is about ethics and algorithmic decision systems, such as COMPAS, EVAAS, YouTube and other recommendation systems, Facebook Ad services, and others. Its aim is to better understand moral concerns about those systems. So we will consider questions such as the following: Is it morally justifiable for a court to use an algorithmic system such as COMPAS in determining whether, and if so for how long, to sentence a defendant to incarceration? Should school systems use algorithmic systems such as EVAAS to promote, reward, and fire teachers? Does YouTube have an obligation to better police the videos that are suggested to viewers?

These questions are multifaceted, and there are lots of different ways we might construe them. There is a meta-ethical issue of what the “should” and “justifiable” in the questions mean or refer to.Footnote ⁵¹ There are questions of legal doctrine such as whether the use of a system like COMPAS impinges upon due process rights and whether algorithmic systems violate contract terms or statutory protections. Those kinds of questions have ethical implications and will come up now and again in the book, but they are not the focus.

Instead we will focus on ethics (which we use interchangeably with “morals”). How, though, does one do that? There is no univocal recipe for answering moral questions. However, we can offer a heuristic for evaluating moral questions.Footnote ⁵² It is not the only way to think through problems, and it is useful in part because it shows just how difficult resolving some moral problems actually is. Nonetheless, it is a way to keep oneself in check and (perhaps more importantly) letting others understand what exactly one is doing. The hope is that following this kind of heuristic will help readers recognize what (if anything!) an ethical argument contributes and what its limitations are.

The first step is clarifying relevant concepts. In considering whether it is justifiable to use algorithmic systems in criminal justice decisions, for example, we will need to specify a number of concepts in order to make progress. An obvious one is “algorithmic system,” which we have tried to clarify in Section 1.2. Do we mean literally any algorithm, including an ink-and-paper set of sentencing guidelines? Or are we talking about only sophisticated, big data–driven systems? Or even machine learning–based systems? Another concept to clarify is “criminal justice decisions.” That could mean decisions about sentencing, supervision, early release, or something else altogether. As we will see in our discussion of COMPAS throughout the book, those issues matter. We will also spend substantial time working to clarify moral concepts. These include “autonomy,” “agency,” “fairness,” “freedom,” “legitimacy,” and others.

The next step is to get one’s empirical factsFootnote ⁵³ straight. Of course, this is easier said than done. If we want to address the question of whether some use of algorithmic systems in criminal justice decisions is justifiable or not, we will want to know something about how such systems operate, what effects they have, and what the underlying picture of the criminal justice system looks like. We might want to know something about how people are likely to use such systems; how those systems interact with social structures like courts, prisons, parole boards, and the like. There is a world of facts surrounding any moral question, some of which are contested, some of which are unknowable, and some of which people get wrong. Regardless, any moral claim will rest on some understanding of the facts on the ground, and getting those straight, getting a clear-eyed view of what facts are tenuous and unknown, and having a sense of what we would like to know is vital in addressing moral questions.

Step three is discerning and applying the correct moral theory to the questions at hand. A few thousand years of religious and philosophical disputes have not answered the question of which moral theories are correct. It is debatable as to whether (and to what extent and by what metric) there has been progress, and there is an important debate about the degree of cultural convergence and divergence on fundamental questions of right and wrong. However, the entire premise of the project of debating the justifiability of anything – using animals for food, abortion, policing, capital punishment, universal suffrage, civil disobedience, mandating vaccinations, and, our topic, applying algorithmic systems in various contexts – is that there can be better and worse (i.e., more and less morally justifiable) answers. And hence, we need some basis on which to find our way on these kinds of questions.

Of course, the question of which moral theory is correct comprises its own, vast area of inquiry. Among the possibilities are consequentialism, deontology, virtue ethics, and contractarianism, each of which has myriad versions. And even once one has some set of moral values one believes to be correct, it is yet another daunting task to figure out just how to apply them. If one is a thorough-going consequentialist who believes that an action is right just in case that act leads to the best overall consequences, it remains an open question of how to figure out how to apply that to actions on the ground.

There are, however, a number of views of morality that are critical of the notion that there are moral principles to apply in the first place, even while accepting that actions can be more or less justified. Some of these views are forms of pragmatism, holding that the practice of theorizing is inseparable from practical consideration. Others adhere to “casuistry,” according to which one can find justification by comparing contested situations to similar “paradigmatic” cases. Still others are forms of moral particularism, according to which general moral principles (e.g., “lying is bad,” “killing is wrong,” “people should be treated equally”) carry no moral weight and provide no role in explaining why actions are right or wrong. Rather, particular acts are right or wrong, and any generalities we draw from particular acts are neither explanatory nor applicable to other cases. But even these “anti-theory” positions continue to insist that there is an account of moral justification (even if it is not one based on explanatorily prior moral principles). And so, if you are searching for the answer, as we are, to the question of whether using algorithmic systems under certain conditions is right or wrong, then you are committed at some level (and perhaps only implicitly) to there being better and worse accounts of moral justifiability on which to base such judgments. We should therefore understand “moral theory” in this third step of the heuristic (i.e., discern and apply the correct moral theory) to be quite broad. It is not only the discernment and application of moral principles but also (or, on the anti-theory views, rather) the discernment and application of the correct understanding of moral justification.

The final part of the heuristic is to apply principles of critical reflection to the concepts, empirical claims, and moral theories one has at one’s disposal. This does not imply that one can simply apply rote, bloodless, “pure” reason to problems. That would be impossible, and it would be unwarranted in light of the inevitable conceptual questions, factual lacunae, and steep burdens of determining the correct moral theories. Rather, it requires a much more limited (but no less important) set of constraints. One should follow one’s reasons where they lead. If the concepts, empirical claims, and moral theory entail that some action is impermissible, one should recognize that entailment and either accept the conclusion or aim to look for the flaw in the premises that led one there. Another element of good reasoning is to recognize just how fallible reasoning is; in other words, one should avoid dogmatism. Related is to accept that one’s conceptual apparatus, understanding of facts, and moral theory are revisable. That’s a good thing. After all, we are bound to have mistaken beliefs and commitments.

This is a project in applied moral philosophy and it aims to use the heuristic we have outlined. It considers some questions that are legal, and it adopts some factual claims. Its contribution comes in several parts. First, it clarifies some relevant concepts, including (among others) autonomy, agency, freedom, paternalism, responsibility, and legitimacy. Second, as to the empirical facts, we largely take our cues from others, drawing on court cases and related documents, other academics’ empirical work on algorithmic systems, and journalists’ investigations on related issues. Third, we advance some claims of normative moral theory. Most of these are grounded in autonomy. However, we do not advance the view that autonomy is the only moral value relevant to analyzing and evaluating algorithmic systems. Rather, our view is that autonomy is an important value, and many moral concerns about algorithmic systems are best understood as, at bottom, issues of autonomy.

Put slightly differently, our project is based on the premise that people are, to some degree, and within important limitations, able to govern themselves. They can determine what kinds of things matter to them, how to use those values to order their lives, and come to agreements with others about how to navigate differences in belief about values. The capacity for autonomy, we argue, gives rise to moral claims that others respect (to some degree and within important limits) persons’ decisions about self-government. Our task, then, is to craft a set of considerations surrounding algorithmic systems based on autonomy. There are many other considerations. Some involve consequences. Some are about the law, including to what extent the law itself provides moral reasons. Yet another is virtue. There are also deeper questions of justice. Still others involve religious and quasi-religious issues. Others involve the proper scope of freedom for technologists to develop and implement systems. Still others involve trade secrets and capitalism. There is no way to adequately address all of these in a volume like this. But we submit that many potential readers will agree with our rock-bottom premise that autonomy matters. So we will start there, clarifying the concept and positing some moral principles that rest on it. On to Chapter 2.

2.1 Autonomy Basics

Autonomy is at root self-government. Individuals can (to some degree and with more or less success) develop their own sense of value, make decisions about what matters most to them, and act accordingly. But just below the surface of that general statement is a rich, complex, and deeply contested set of questions about the scope, nature, and moral importance of autonomy. Our task in this chapter is to offer an account that shows our philosophical commitments, positions itself within some of the weightiest philosophical debates about autonomy, and explains where (and why) we decide not to make stands. The account we offer here is lightweight and ecumenical. It is lightweight because it takes on minimal commitments. It is ecumenical in that it is compatible with a broad range of views. This approach can go a long way in helping understand the moral importance of algorithmic systems without having to resolve some of the deepest and most vexing disputes about the nature and scope of autonomy.

We can get a sense of the basic contours of autonomy by considering some archetypical ways that autonomy can be undermined.Footnote ¹ For one, a person’s actions and preferences are not autonomous to the extent that they are the result of coercion or deceit. Where an employer gives an employee the option of working after they have clocked out or being fired, the employee’s choice to keep their job is of course their genuine preference. However, that choice is not autonomous because it is constrained in an illegitimate and coercive way. Suppose instead that the employer constrains the employee’s options through deceit. The employer lies to the employee that the company will fail if the employee does not put in extra, unpaid hours after they have clocked out. Again, the employee’s choice to do unpaid work is based on their genuine desire to keep the company afloat and retain their job. It is not, however, an autonomous choice because they have been deliberately deceived.

A more subtle type of case concerns adaptive preferences. Suppose that the employer continuously asks the employee to work extra, unpaid hours in order for their business to thrive. The employee believes that other employment options are limited and fears losing their job if they do not do the unpaid work. Although it is a burden, the employee does indeed want the business to thrive. They may consciously embrace the practice to make the burden seem manageable. This would seem to conflict with their autonomy in that they formed their preference to work extra, unpaid hours in response to illegitimate pressure from the employer and their belief that other alternatives were bad. Finally, suppose the employer comes to rely on the employee in a close working relationship, provides appropriate encouragement, and genuinely values the employee’s skill and professionalism. Indeed, the employer values it so much that they regularly promote other, similarly qualified employees to better-paid, management positions so that they can keep drawing on the employee’s talents directly. The employee may come to think that the management positions are not really attractive anyway, as they involve a steep learning curve, increased time demands, and working with people they do not know. These are, in the employee’s mind, grapes that are out of reach and probably sour anyway. What the employee prefers conflicts with their autonomy in that they formed their preferences in response to their limited opportunity.Footnote ²

The key to each of these examples is that just what it means for individuals to develop their own sense of value, the conditions under which individuals’ decisions are their own, and precisely when people act according to their own preferences is itself a nuanced question. Hence, it is not enough to say that autonomy is simply a matter of people being able to do what they want. Rather, the deeper question is this: Under what conditions are people and their choices, preferences, and values properly understood as autonomous?

The view that we will advocate here is that although autonomy of preference and choice is important, this sort of autonomy is limited. A fuller understanding of autonomy will focus on autonomy of persons, which is to say the social conditions under which a person is autonomous. Our view, though, is ecumenical in that it incorporates both psychological views and personal autonomy views. The reason is that the normative considerations substantially overlap, as we explain in Section 2.4.

2.2 Some Distinctions

With our first gloss in mind and having described several ways in which persons’ autonomy may be impinged, it will be useful to make a few distinctions. After doing so, we can fill out our conception.

2.3 The Key Split

These distinctions are important and useful, but we still owe our own account. The view we advance in this book incorporates two different conceptions of autonomy. The first focuses on the relationship between a person and their motives, intentions, values, and preferences; that is, it holds that autonomy is primarily understood in terms of how an individual’s desires, preferences, and actions relate to their history and psychology. For lack of a better term, we will call this “psychological autonomy.” A different way of understanding autonomy focuses on an individual’s social conditions, relationships to other people, status within a community and polity, and the range of options and opportunities in which one develops a sense of self and others. Following Marina Oshana, we will call this “autonomy of persons” or “personal autonomy.”Footnote ¹² The boundaries between these two ways of understanding autonomy are blurry, as a person’s social circumstances are closely related to their desires, values, and preferences. Nonetheless, both facets are important in understanding the relationship between algorithmic decision systems and autonomy. We will argue that both components are morally important, and the conceptual differences between them matter less than the fact that both conceptions can underwrite similar social, moral, political, and legal claims.

To explain the two different ways of understanding autonomy, we will examine an exemplar of each.

2.4 Reconciling Psychological and Personal Autonomy

In the previous sections we described two categories of autonomy views. The first, which focuses on an individual’s special relationship to their values, intentions, and motivations, is exemplified by John Christman’s account. Christman’s view is procedural in that it does not index autonomy to the content of an individual’s values and motivations. The view is relational in that it understands the importance of a person’s history and social circumstances in supporting values and preferences from which a person would not be alienated. As noted in Section 2.3.1, there are numerous competing views of autonomy in this category, articulating different kinds of procedural conditions necessary for persons to be autonomous. The second category, exemplified by Marina Oshana’s view, understands psychological autonomy as an insufficient account of what matters morally, which is personal autonomy. Oshana’s view is that while elements of psychological autonomy are important, they are neither necessary nor sufficient for personal autonomy. Personal autonomy demands more, including substantive independence.

This split seems wide at first blush, one focusing on a person’s psychology and its history, the other focusing instead on facts about a person’s place in the social, material world (and in a weakly perfectionist way at that). However, for several reasons, we need not draw a conclusion about which family of views is better supported. To begin, our project is different from the projects of Christman, Oshana, Buss, Frankfurt, Dworkin, and others. We are not aiming at a new, comprehensive account of autonomy and its importance. Rather, we are beginning with the premise that autonomy is important, adopting a minimal, ecumenical approach to the existing accounts, and developing an understanding of the moral salience of automated decision systems from the account built from convergence among the other accounts. Hence, differences in the families of views matter here only to the extent that they affect an analysis of autonomy-based moral claims.

And when we look at that set of issues, the categories of accounts of autonomy (including Christman’s and Oshana’s) converge. Both types of account can agree about many of the conditions that are important for individuals’ autonomy. Consider Oshana’s example of the woman raised in oppressive circumstances. She argues that the woman in that case lacks autonomy because the social conditions necessary for her to act with substantive independence are lacking, regardless of whether her (current, local) values and beliefs are procedurally independent. Christman agrees that the woman lacks autonomy, but he argues that this is because, in all likelihood, she would experience alienation if she were to reflect on her values and preferences in light of the oppressive processes affecting their formation.Footnote ⁴³ What’s important for our account, though, is that Oshana and Christman agree that she lacks autonomy, it is the result of a moral wrong, and the lack of autonomy is a key part of the explanation for why her treatment is morally wrong.

Oshana’s would-be surrendered woman case is a bit more difficult. Oshana, recall, argues that this person is autonomous because she is procedurally and substantively independent. She has the social wherewithal to act differently than she does, regardless of whether her current actions are ones from which she would be alienated (if she were to critically reflect upon them). Christman’s view is that the would-be surrendered woman lacks autonomy precisely because her actions do not meet the authenticity conditions. However, as Oshana has constructed the case, Christman would not be able to point to a moral wrong undermining her autonomy. Instead, Christman could argue that a more thoroughgoing examination of a person’s history and the social structures in which a person’s values are formed are likely to uncover limitations rooted in competence conditions or authenticity conditions. However, if there are no such limitations, Christman would allow that she lacks autonomy.Footnote ⁴⁴

What is important, though, is that Christman and Oshana’s positions about what is morally relevant are compatible. What matters to each is that the would-be surrendered woman’s social and relational circumstances are not particularly constrained and that she does experience alienation. The difference is whether that alienation is incompatible with the best-supported conception of autonomy, not whether there is a moral infirmity, and not what the source of such an infirmity might be.

Moreover, even in the relatively narrow range of cases in which Oshana and Christman would disagree about whether a person is autonomous, they can agree that there are autonomy-based wrongs. Consider instead a case in which a person is raised in oppressive circumstances, internalizes facets of that oppression, and in which they come to have values, motivations, and preferences that reflect that oppression. And suppose that even with sufficient opportunity to critically reflect on those values, motivations, and preferences and their genealogy, they would experience no alienation. Christman would have to conclude that the person is autonomous (after all, they meet his competence and authenticity conditions). Oshana would conclude that they lack autonomy only if they lack sufficient social opportunity to change their views and act accordingly. She would disagree that the person is autonomous only if their current social circumstances reinforce the values, motivations, and preferences formed under limited conditions. But Christman and Oshana could agree that there is an autonomy-based wrong in their limited circumstances. Christman could argue that those circumstances are unjust because they tend to be wrongs that lead to alienation. Oshana’s view is that the wrong is based on the fact of social limitation per se. In other words, the precise explanation of the wrong will vary. However, the views overlap in the cases of moral wrongs and the type of wrongs, while differing in the explanation of them.

There is also the possibility of an account of autonomy that bridges the psychological authenticity, procedural accounts with social-relational accounts. Recently, Zi Lin has argued that instead of self-rule, autonomy should be understood as independence from other-rule.Footnote ⁴⁵ Her idea is that psychological authenticity requires something other than non-alienation or confluence with higher-order values. Those accounts index authenticity solely to a person’s psychology (and in Christman’s view, their diachronic psychology). Such views have some well-known conceptual problems, including the problem of regress and problem of problematic influencers. Both of these, Lin argues, can be resolved by building autonomy around independence from other-rule. Hence, on Lin’s view, psychological autonomy has a relational component that is constitutive of autonomy, rather than merely being causally relevant.

2.5 An Ecumenical View

There are plenty of disputes about the nature of autonomy: whether it describes a person’s relation to his or her values, motivations, and intentions or describes a person’s social and relational circumstances; whether competence and authenticity conditions are necessary or sufficient for a person to be autonomous; whether social conditions are constitutive of autonomy or merely causally relevant to whether one is autonomous. But, as we explain in the previous section, the different views can agree that certain things matter because of autonomy. That is the basis of our ecumenical view of autonomy in which we distill a number of key points about autonomy and its value from different conceptions.

To begin, any plausible view of autonomy will recognize the importance of procedural independence (though they may disagree about whether procedural independence is necessary for autonomy, sufficient for autonomy, or merely important for autonomy). Procedural independence requires several things. One is epistemic competence, which is to say one must be to some degree self-reflective, self-aware, and understand themself as able to actualize a life plan. Respecting people as autonomous demands fostering epistemic competence.

Notice, though, that affording people the circumstances in which they can exercise their epistemic competence is a facet of personal autonomy. Hence, severely or deliberately constrained information environments are ways in which an epistemically competent person will fail to have personal autonomy because they lack substantive independence.

Another component of psychological autonomy is rationality, or the ability to understand the world around oneself, to make close connections between facts and inferences, and to make decisions that line up with well-enough ordered assessments of facts and reasons. As noted, this is a low bar. Rationality is also related to personal autonomy. Constraints on a person’s ability to reason clearly can come from poor informational environments or from psychological stressors that thwart their ability to think clearly, hence inducing poor decisions.

Procedural independence also includes some degree of authenticity. Oshana argues that authenticity is not required, and this is a key difference between her and Christman. Our view, however, is not that authenticity is necessary or sufficient for autonomy. Rather, authenticity is important for two reasons. One is that authenticity is evidence of personal autonomy. The ability to reflect on one’s values and preferences and to recognize them as compatible with one’s sense of self and practical identity over time is an important test of the degree to which one’s values and preferences are one’s own, and (hence) that one self-governs. Likewise, lack of authenticity – that is, where a person would be alienated from their preferences and values upon reflection – is defeasible evidence that one’s personal or social autonomy is compromised. And fostering authenticity is itself morally important in that individuals are morally valuable in part because individuals are capable of determining for themselves what is of value in their lives, and they are (hence) the source of that value.Footnote ⁴⁶ In other words, structures that make it likely that individuals would experience alienation from their desires and values (in Christman’s conception, where structures make it likely that a person will have inauthentic preferences and motivations) are in all likelihood antithetical to personal autonomy. Those structures may be morally justifiable overall, but that would be in spite of their relationship to autonomy.

Beyond procedural independence and its competence and authenticity components, autonomy requires substantive independence. Whether one is personally autonomous turns on their circumstances and their environment.Footnote ⁴⁷ This includes social conditions such as choice among a range of attractive options (per Raz), control over meaningful facets of life, conditions of self-respect, relative financial independence, and so forth. In this we follow Oshana’s conception closely. There are, however, a few ways in which substantive independence is insufficient in ensuring autonomy overall. A person can be substantively independent (in terms of having financial wherewithal, not subject to others’ arbitrary power) but can have competence conditions undermined. Likewise, one can have local substantive independence but have been subject to conditions that make it difficult to have authentic preferences and desires (we will return to this in Chapters 4, 5, and 6). Of course, those limitations may be understood in terms of more global substantive independence.

We can summarize our view and provide a foundation for the arguments in the remainder of the book with a few primary propositions.

The first set of these concerns Feinberg’s distinctions between the various meanings of “autonomy.” First, autonomy as a capacity predominantly grafts onto psychological autonomy; it is entirely possible for a person to have a capacity for autonomy, but be prevented from exercising it (i.e., being denied personal autonomy). Second, successful exercise of autonomy in Feinberg’s sense demands substantive independence; however, the autonomy one exercises in that sense is psychological. Third, autonomy as ideal (which is to say that individual autonomy properly integrated into a larger matrix of values, including one’s responsibilities and community concerns) reflects persons’ obligations to respect others’ psychological autonomy and ensure the conditions of personal autonomy. Fourth, the idea that autonomy is a right should be understood in the sense that persons have claim-rights grounded in their autonomy.

The next propositions reflect the fact that both psychological and personal autonomy are important and (hence) that both procedural and substantive independence are valuable.

To begin, procedural independence is morally important, but limitations on a person’s procedural independence are important in large part because of conditions that impose such limitations (including limitations placed by other people, by organizations, by social structures, by natural causes, and so forth). Fostering conditions that promote procedural independence is the crucial part. It is similarly important that others respect the actions of those who lack procedural independence. They may have capacity autonomy, after all, even if it is unsuccessfully exercised.

Substantive independence is important beyond its role in fostering psychological autonomy. It does not require that one can literally do without others – business partners may need each other’s expertise and efforts, but that does not undermine their substantive independence. Rather, what matters is whether others make demands that are arbitrary, demand inauthentic or epistemically unjustifiable actions, or undermine one’s ability to act for one’s own reasons. That does not mean any kind of compromise is antithetical to respecting autonomy. What matters is whether agreements and dependence are such that one could agree to them as part of valuable social relationships and other goods. It also matters how global those contradictions are.

2.6 Objections

So far, we have considered a few different conceptions of autonomy and offered the account we will use to ground the arguments in the remainder of the book. There are, however, important critiques of autonomy, both as a concept and as a basic value. One family of criticism is that autonomy-based moral theories (and deontological theories generally) are simply mistaken about what matters. Consequentialists and virtue ethicists (among others) might argue that other values are the proper measure of moral value. As important as those criticisms are, we won’t offer a defense here. Rather, we will simply confirm that a rock-bottom assumption of this project is that autonomy is morally valuable, and it is an important enough (and rich enough) value that it can ground the arguments we offer throughout. If one disagrees with that assumption, this project probably won’t be persuasive.

Much more important in our view are criticisms levied against the very concept of autonomy and its value. Feminists, for instance, have critiqued autonomy-based theories as atomistic, divorced from social responsibilities, unmindful of the importance of relationships in identity formation, hyperrational, and disembodied. These criticisms have provided a compelling corrective to major strains of autonomy scholarship. The principal views that we have drawn on for our account are sensitive to these concerns and explicitly incorporate the critiques into their accounts. Indeed, the critiques are so clearly correct that any plausible contemporary account of autonomy will be explicitly relational. Nonetheless, it is worth canvassing a few of the critiques here.Footnote ⁴⁸

One set of critiques Mackenzie and Stoljar characterize as “metaphysical.” Specifically, such critiques reject autonomy views on the grounds that autonomy attributions assume that agents are atomistic. That, in turn, can mean several things. One possibility is that individuals are causally independent or isolated from others; that is, individuals are self-creating sources of values and desires. Of course, that is false; one can discern autonomy in some sense even while recognizing that people’s understandings are caused by their social milieu.Footnote ⁴⁹ A different possibility is that agents are atomistic in the sense that they are independent of social and family relationships or that persons have intrinsic properties that do not depend on how they relate to others. Again, individuals are not like this at all. First, as a matter of empirical fact humans are hyper-social. Second, people’s identities and values do not cohere with absent relationships with others. Nonetheless, autonomy remains conceptually coherent so long as we allow that there are important ways in which an individual can form their values under competence and authenticity conditions and has some claim to avoiding arbitrary interference.Footnote ⁵⁰ A final way of understanding atomistic individualism is that persons are metaphysically distinct. Mackenzie and Stoljar point out that this is not so much a critique as an obvious fact.Footnote ⁵¹

A related set of critiques are based on conceptions of care. The idea is that traditional understandings of autonomy have under- or devalued women’s perspectives and traditional, gendered social roles. Hence, relationships of care have been systematically excluded from autonomy conceptions. But those roles are of fundamental human value, and ignoring their centrality is a mark against any view. Mackenzie and Stoljar point out, though, that a number of feminist scholars, such as Jennifer Nedelsky and Evelyn Fox Keller, incorporate understandings of care into conceptions of autonomy. Acting as an agent in the world – a constitutive part of autonomy – should involve interdependence, nurturing, and care. And such capacities are indeed types of competence conditions for autonomy.Footnote ⁵²

The upshot of these critiques is that while traditional conceptions of autonomy are cramped and implausible, more capacious understandings can address some of those shortcomings. Specifically, any reasonable account of autonomy will recognize that the sources of self, meaning, and value will be intimately bound up with one’s social, community, and family relationships. We are not self-executing sources of value; rather, we are sources of value within social, relational contexts. Moreover, the competencies of autonomy will involve one’s ability to enter and foster those relations. And lastly, the procedural and substantive independence conditions (e.g., those articulated by Christman and Oshana, and upon which we draw heavily) provide an explanation for why oppressive conditions (based on gender, or race, or ethnicity, or class, or happenstance) conflict with autonomy.Footnote ⁵³

2.7 Conclusion: Related Concepts and Moral Salience of Autonomy

In this chapter we have made some basic distinctions about autonomy and canvassed two important, representative views (each exemplifying a different family of conceptions of autonomy). We have argued that those groups of views have substantial normative overlap, and we have used that overlap to advance an ecumenical view of autonomy. That conception provides a foundation for the arguments in the rest of this book. Before turning to those arguments, though, it is worth briefly describing why autonomy matters and how it grounds other concepts that will figure into the chapters that follow.

One way that autonomy matters morally is built into our account. People are capable of determining their values, desires, and preferences, and they can use those values, desires, and preferences to guide their decisions and steer their lives. They are, in other words, a source of value. To the extent that others severely constrain individuals’ ability to form their own sense of value (e.g., by limiting their procedural or substantive independence), they stifle the degree to which individuals’ values are their own. Moreover, thwarting persons’ abilities to act on their own values by coercion, deception, or severely constrained choice architecture is a way of circumventing autonomy and an affront to persons as choosers and self-governors.

However, the moral salience of autonomy goes well beyond limitations on persons’ abilities to act on their own desires and values. Consider again the distinctions we set out at the beginning of this chapter. The ability to act on one’s own desires and values is a matter of successfully exercising one’s capacity for autonomy. That assumes that people have the capacity to autonomous. It may not be enough, though, to simply assume people have that capacity. Rather, it is plausible that there is a social responsibility to promote that capacity. Indeed, that is an important justification for education.Footnote ⁵⁴

Throughout the book, we consider different kinds of moral concerns about automated systems, all of which are rooted in autonomy. In Chapter 3, we address broader social claims regarding use of algorithmic systems. The systems we discuss in that chapter – K-12 teacher evaluation systems and criminal justice risk assessment tools – do not directly limit autonomous individuals’ choices via deception, coercion, or manipulation, and they do not obviously undermine substantive independence. Rather, we argue that autonomy is a foundation for a claim that persons be subject only to social systems that they could reasonably endorse. In Chapter 4, we make the case that autonomy includes more than the ability to act on one’s values and preferences. It also has an important informational component. That informational component is vital regardless of whether a person is able to put information into practice. Autonomy demands, in our view, the ability to exercise both practical agency (the ability to function effectively in important arenas) and cognitive agency (the ability to exercise a kind of evaluative control and understand one’s place in the world, regardless of one’s ability to affect it).

Later, we will expand our discussion of the social conditions of freedom. We have already discussed how Oshana’s account incorporates facets of Philip Pettit’s understanding of republican freedom into her understanding of personal autonomy. Our task in Chapter 5 is to draw out the autonomy- and agency-based conditions of freedom. In Chapter 6, we argue that successful exercise of autonomy may in many cases demand a degree of epistemic paternalism. That is, given the competence and authenticity requirements for autonomy, certain kinds of media entities may be permitted (or even obligated) to exert a degree of editorial control over the content posted on their platforms. In Chapter 7 we explain the moral requirements of autonomous agents. That is, autonomous persons are capable of legislating and following moral principles. That capacity creates obligations to act responsibly. Obfuscating that responsibility will in many cases be a distinct kind of wrong. Finally, in Chapter 8, we will consider how autonomy plays a crucial role in underwriting political legitimacy.

Having introduced our polestar cases, situated concerns about algorithmic systems in a broader discourse, and explained our strategy in Chapter 1, and having provided our catholic conception of autonomy here, we can begin directly addressing the autonomy-grounded moral concerns in algorithmic systems. Next stop: Chapter 3.