3.4.1 Wagner v. Haslam

Recall from Chapter 1 that the plaintiffs in this case (Teresa Wagner and Jennifer Braeuner) were teachers who challenged the Tennessee Value-Added Assessment System (TVAAS), which is a proprietary system similar to IMPACT. Because TVAAS did not test the subjects Wagner and Braeuner taught, they were evaluated based on a school-wide composite score, combined with their (excellent) scores from in-person teaching observations. This composite score dragged their individual evaluations from the highest possible score (as they had received in previous years) to middling scores. As a result, Wagner did not receive a performance bonus and Braeuner was ineligible for consideration for tenure. Moreover, each “suffered harm to her professional reputation, and experienced diminished morale and emotional distress.”Footnote ³⁷

There is a deeper moral issue grounding the legal case. Wagner and Braeuner frame their case in terms of harms (losing a bonus, precluding tenure consideration, and so forth), but those harms matter only because they are wrongful. They are wrongful because TVAAS is an evaluation system that teachers could not reasonably endorse. Wagner and Braeuner’s scores did not reliably track their performances nor did the scores reflect factors for which they were responsible, as the scores were based on the performance in subjects Wagner and Braeuner did not teach. And the stakes in the case are fairly high (there were financial repercussions for Wagner and job security for Braeuner). So, per our account, they were wronged.

There may also be a relative burden issue with TVAAS, though it is not discussed explicitly in the Wagner opinion. A 2015 study of TVAAS found that mathematics teachers across Tennessee were, overall, found to be more effective by TVAAS than their colleagues in English/language arts.Footnote ³⁸ This finding supports two hypotheses: Tennessee’s math teachers are more effective than its English/language arts teachers, and TVAAS is systematically biased in favor of math teachers.Footnote ³⁹ If the latter hypothesis is true – and we suspect that it is – then, there are teachers, specifically Tennessee’s English/language arts teachers, who have an additional complaint against TVAAS: It imposes a higher relative burden on them because it arbitrarily returns lower scores for non-math teachers. They could not reasonably endorse this arrangement.

3.4.2 Houston Fed of Teachers v. Houston Ind Sch Dist

The Houston Schools case is superficially similar to Wagner in that it involves a similar proprietary VAM (EVAAS) to evaluate teachers. The school system used EVAAS scores as the sole basis for “exiting” teachers.Footnote ⁴⁰ The primary concern for our purposes is that Houston Schools did not have a mechanism for ensuring against basic coding and clerical errors. They refused to correct errors on the grounds that doing so would require them to rerun their analysis of the entire school district. That, in turn, would have two consequences. First, it would be costly; second, it would “change all other teachers’ reports.”Footnote ⁴¹

The moral foundations of the teachers’ complaints should by now be clear. The stakes here – i.e., losing one’s job and having one’s professional image tarnished – are high. EVAAS is unreliable, having what the court called a “house-of-cards fragility.”Footnote ⁴² And that unreliability is due to factors for which teachers are not responsible, “ranging from data-entry mistakes to glitches in the code itself.”Footnote ⁴³ Hence, teachers could not reasonably endorse being evaluated under such a system.

We can add a complaint about relative burden, at least for some teachers. EVAAS, like IMPACT, gives lower scores to teachers working in poorer schools. To see this, consider an analysis of EVAAS’s use in Ohio (in 2011–12) conducted by The Plain Dealer and State Impact Ohio, which found the following:

• Value-added scores were 2½ times higher on average for districts where the median family income is above $35,000 than for districts with income below that amount.

• For low-poverty school districts, two-thirds had positive value-added scores – scores indicating students made more than a year’s worth of progress.

• For high-poverty school districts, two-thirds had negative value-added scores – scores indicating that students made less than a year’s progress.

• Almost 40 percent of low-poverty schools scored “Above” the state’s value-added target, compared with 20 percent of high-poverty schools.

• At the same time, 25 percent of high-poverty schools scored “Below” state value-added targets while low-poverty schools were half as likely to score “Below.”Footnote ⁴⁴

In virtue of these findings, it is plausible that – in addition to complaints grounded in reliability, responsibility, and stakes – teachers in low-income schools have a complaint grounded in the uneven distribution of burdens.

And, hence, use of EVAAS is not something teachers subject to it could reasonably endorse.

3.4.3 Wisconsin v. Loomis

Our framework for understanding algorithmic systems and autonomy applies equally well to risk assessment tools like COMPAS.

To begin, COMPAS is moderately reliable. Researchers associated with Northpointe assessed COMPAS as being accurate in about 68 percent of cases.Footnote ⁴⁵ More important is that COMPAS incorporates numerous factors for which defendants are not responsible. Recall that among the data points that COMPAS takes into account in generating risk scores are prior arrests, residential stability, employment status, community ties, substance abuse, criminal associates, history of violence, problems in job or educational settings, and age at first arrest.Footnote ⁴⁶ Regardless of how well COMPAS’s big and little bars reliably reflect reoffense risk, defendants are not responsible for some of the factors that affect those bars. So, while Loomis did commit the underlying conduct and was convicted of prior crimes, COMPAS incorporates factors for which defendants are not responsible.Footnote ⁴⁷ For example, the questionnaire asks about the age at which one’s parents separated (if they did); whether one was raised by biological, adoptive, or foster parents; whether a parent or sibling was ever arrested, jailed, or imprisoned; whether a parent or parent-figure ever had a drug or alcohol problem; and whether one’s neighborhood friends or family have been crime victims.Footnote ⁴⁸ Moreover, even if some factors (e.g., residential stability, employment status, and community ties) are things over which individuals can exercise a degree of control, they are matters for which we shouldn’t attribute responsibility in determining sentencing for offenses.

Further, the use of COMPAS in Loomis is high stakes. Incarceration is the harshest form of punishment that the state of Wisconsin can impose. This is made vivid by comparing the use of COMPAS in Loomis with its specified purposes. COMPAS is built to be applied to decisions about the type of institution in which an offender will serve a sentence (e.g., lower or higher security), the degree of supervision (e.g., from probation officers or social workers), and what systems and resources are appropriate (e.g., drug and alcohol treatment, housing, and so forth). Indeed, Northpointe warns against using COMPAS for sentencing, and Loomis’s presentence investigation report specifically stated the COMPAS report should be used “to identify offenders who could benefit from interventions and to target risk factors that should be addressed during supervision.”Footnote ⁴⁹ When the system is used for its intended purposes – identifying ways to mitigate risk of reoffense of persons under state supervision – the stakes are much lower.Footnote ⁵⁰ Hence, it is more plausible that someone subject to its use could reasonably endorse it in those cases.

One of Loomis’s complaints about COMPAS is that it took his gender into account. The court found that COMPAS’s use of gender was not discriminatory because it served the purpose of promoting accuracy. So Loomis’s claim that he shouldered a higher relative burden under this system was undercut, and the court was – in our opinion – correct in their response to his claim. Because men do commit certain crimes more often than women, removing gender as a factor could result in the systematic overestimation of women’s risk scores.Footnote ⁵¹ This does not, however, mean that COMPAS has no issues with respect to the question of relative burden.

To introduce the relative burden issue, let’s turn to a related controversy surrounding COMPAS. In May 2016, ProPublica reported that COMPAS was biased against Black defendants.Footnote ⁵² Specifically, ProPublica found that COMPAS misidentified Black defendants as high risk twice as often as it did White defendants. Northpointe, the company that developed COMPAS, released a technical report that was critical of ProPublica’s reporting.Footnote ⁵³ They claimed that despite its misidentifying Black defendants as high risk at a higher rate, COMPAS was unbiased. This is because the defendants within risk categories reoffended at the same rates, regardless of whether they are Black or White.

The back and forth between Northpointe and ProPublica is at the center of a dispute over how to measure fairness in algorithmic systems. Northpointe’s standard of fairness is known as “calibration,” which requires that outcomes (in this case reoffense) are probabilistically independent of protected attributes (especially race and ethnicity), given one’s risk score (in this case high risk of reoffense and low risk of reoffense).Footnote ⁵⁴ In this context, calibration requires that knowing a defendant’s risk score and race should provide the same amount of information with respect to their chances of reoffending as just knowing their score. ProPublica’s standard of fairness, on the other hand, is “classification parity,” which requires that classification error is equal across groups, defined by protected attributes.Footnote ⁵⁵ As the dispute between ProPublica and Northpointe shows, you cannot always satisfy both standards of fairness.

This might be counterintuitive. ProPublica’s conclusions about COMPAS are a result of the fact that Black defendants and White defendants are arrested and rearrested at different rates, and hence Black and White defendants are counted as “re-offending” at different rates. To better understand how COMPAS can satisfy calibration but violate classification parity, it will be helpful to substitute a version of COMPAS with simplified numbers, which we will call “SIMPLE COMPAS.” Note that we choose the numbers here because they loosely approximate ProPublica’s analysis of COMPAS, which included larger numbers of Black defendants counted as reoffending.

Figure 3.1 SIMPLE COMPAS

Now consider three questions about SIMPLE COMPAS.

First, if you randomly select a defendant, not knowing whether they are from the high- or low-risk group, would learning their race warrant suspicion that their chance of reoffending is higher (or lower) than others from the risk group they are in? No. As we have stipulated, defendants within a risk group reoffend at similar rates regardless of race, and this is reflected in the bar chart, where the proportion of reoffenders to non-reoffenders in “Low – Black” is the same as “Low – White”: one in five. Similarly, the proportion of reoffenders to non-reoffenders in “High – Black” is the same as “High – White”: four in five. In virtue of this, SIMPLE COMPAS satisfies the calibration standard of fairness.

Second, if you randomly select a defendant, not knowing whether they are from the high- or low-risk group, would learning their race warrant increasing or decreasing your confidence that they are in the high-risk group? Yes. We have stipulated (tracking the analysis of COMPAS from ProPublica) that the high-risk group is predominantly Black and that the low-risk group is predominantly White (80 percent). If the randomly selected defendant is Black, you should increase your confidence that they are from the high-risk group, from about 55 percent (since five out nine total defendants – 250/450 – are high risk) to about 66 percent (since six out of nine Black defendants – 200/300 – are high risk). If, on the other hand, the randomly selected defendant is White, you should increase your confidence that they are from the low-risk group, from about 44 percent (since four out of nine defendants – 200/450 – are low risk) to about 66 percent (since six out of nine White defendants – 100/150 – are low risk).

Third, if you randomly select a defendant, not knowing whether they are from the high- or low-risk group, should learning their race affect your confidence that they are non-reoffending high-risk (i.e., misidentified as high-risk)? Yes. To see this, just look back to the previous question. If you learn the defendant is Black, you should increase your confidence that they are from the high-risk group. In virtue of this, your confidence that they are in the non-reoffending high-risk group should increase too, from about 11 percent (since one out of nine defendants – 50/450 – are non-reoffending high risk) to about 13 percent (since two out of fifteen Black defendants – 40/300 – are non-reoffending high risk). This may not seem like much, unless we appreciate that learning that a defendant is White should drive your confidence that they are non-reoffending high risk down to about 6.6 percent (since one out of fifteen White defendants – 10/150 – are non-reoffending high risk). This means that, in SIMPLE COMPAS, Black defendants are twice as likely as White defendants to be misidentified as high-risk, which violates classification parity. And this is what ProPublica found: COMPAS misidentifies Black defendants as high risk about twice as often as it does White defendants.Footnote ⁵⁷ As this should make clear, violating classification parity is one way that an algorithmic system can impose an undue relative burden.

Analyzing SIMPLE COMPAS further shows why this is a matter of relative burden and why this sort of issue is distinct from issues of reliability, responsibility, and stakes. Suppose that in the world of SIMPLE COMPAS, Black and White citizens use illicit drugs at similar rates. However, Black citizens are disproportionately charged with drug crimes because they are more likely to get stopped and searched. Because of this correlation, SIMPLE COMPAS (which does not directly take race into account) is more likely to identify Black defendants as high risk.

If we assume further that the justice system that SIMPLE COMPAS is embedded in has sensible penalties,Footnote ⁵⁸ then the shortcomings of SIMPLE COMPAS defy the categories of responsibility, reliability, and stakes. SIMPLE COMPAS does not hold defendants responsible for factors not under their control: The vast majority of those tagged as high risk are in fact likely to reoffend, and they are being held responsible for breaking laws that they in fact broke. Similarly, we can’t complain that SIMPLE COMPAS is unreliable; it is well calibrated and predicts future arrests very reliably. Finally, we cannot complain from the perspective of stakes, because – as we have stipulated – the justice system SIMPLE COMPAS is embedded in has sensible penalties. Yet something is wrong with the use of SIMPLE COMPAS, and the problem has to do with relative burden.

To see this, compare SIMPLE COMPAS with EVEN COMPAS.

Suppose that EVEN COMPAS, like SIMPLE COMPAS, is embedded in a system that has sensible penalties. The only difference is that the EVEN COMPAS police force does not discriminate, and so EVEN COMPAS does not learn to identify Black defendants as high risk more often than White defendants.

Let us now make two observations. First, note that EVEN COMPAS might not be problematic. That is, given the details of the case, it does not seem like there are obvious objections to its use that we can make: It is an accurate and equitable device. It does misclassify some non-reoffending defendants as high risk, but unless we abandon pretrial risk assessment altogether or achieve clairvoyance, this is unavoidable.

Second, note that SIMPLE COMPAS is intuitively problematic even though – from defendants’ points of view – EVEN COMPAS treats them worse on the whole (i.e., compared to SIMPLE COMPAS, EVEN COMPAS is worse for White defendants and it is not better for Black defendants). What could explain SIMPLE COMPAS being problematic while EVEN COMPAS is not? It cannot be an issue of reliability, responsibility, or stakes. Rather, it is the fact that SIMPLE COMPAS imposes its burdens unevenly: It is systematically worse for Black defendants. Hence, relative burden is a factor in whether people could (or could not) reasonably endorse a system that is distinct from reliability, responsibility, and stakes.

Let’s return to Loomis. COMPAS – like the fictional SIMPLE COMPAS – does have an issue with high relative burden. The burden does not happen to be one that negatively affects Loomis. He does not have an individual complaint that he has endured a burden that is relatively higher than other people subject to it. However, it does not follow that COMPAS is a system that any person subject to it can reasonably endorse. Rather, because it imposes a greater relative burden to Black defendants than to White defendants, it is one that at least some defendants cannot reasonably endorse.

4.1.1 Taylor, Arroyo, and the Reasonable Endorsement Test

One way to understand Taylor’s and the Arroyos’ situations is in the terms we spelled out in Chapter 3, namely whether the background reporting systems are ones that people subject to them can reasonably endorse. Both Taylor and Arroyo have experienced considerable material burdens based on algorithmically aided decision systems. Both were held to account by systems that are based on factors for which Taylor and Arroyo are not responsible, and the stakes in each case are high. Hence, one could make the case that the reporting systems are ones that individuals subject to them cannot reasonably endorse as comporting with their material interests. Such an analysis, while compelling, would not be complete.

Something has gone wrong in the Taylor and Arroyo cases beyond the fact that they were materially harmed. This separate consideration is an informational wrong. Taylor and Arroyo did not know (at least initially) what information in their files led to their background check results. Carmen Arroyo did not discover the basis for Mikhail’s check until it was too late to do anything meaningful about it. Taylor lost opportunities before she discovered the reason. Further, in Taylor’s case, several companies providing the misinformation would not fix their files upon learning that they had made a mistake. Finally, both Taylor and Arroyo were left in the dark as to how exactly the results came out the way they did; they were not afforded an understanding of the systems that cost them the opportunities they had sought.

Arroyo has an additional, distinctive complaint. When her son’s application was rejected, the apartment complex did not know the details of the disqualifying conduct because CoreLogic did not supply them. This means that Arroyo was not given enough information about Mikhail’s rejection to even contest the claim. Compare Arroyo’s case with Taylor’s. Taylor at least knew that her file had contained a false drug charge. Knowing what she had been accused of informed her that she had to prove what she had not done. Arroyo lacked even that.

We have mentioned that there is at least some regulation that attempts to address these sorts of issues and that there is plausibly a question as to whether CoreLogic complies with its legal obligations under FCRA (as stated earlier, companies do not always follow the notification requirement). Could full compliance with FCRA bring about practices that Taylor and Arroyo could reasonably endorse? Again, we think not. For one, FCRA does not specify when subjects are owed notification.Footnote ²⁸ So the notification requirement can be met without actually affording data subjects the underlying thing that really matters: time to effectively respond to any false or misleading information in their files and an understanding of where they stand with respect to decisions made about them. These are the claims we address in the following section.

4.2.1 Argument 1: Practical Agency

The first set of criteria for determining whether persons have claims to information about automated decision systems echoes the criteria we advanced in Chapter 3. Specifically, whether an individual has a claim to information about some algorithmic decision system that affects their life will be function of that system’s reliability, the degree to which it tracks actions for which they are responsible, and the stakes of the decision.

Assume for a moment that Taylor’s problems happen in the context of a reporting system that people cannot reasonably reject on grounds of reliability, responsibility, and stakes. Taylor nonetheless has a claim based on practical agency. To effectively cope with the loss of her opportunities for employment and credit, she needs to understand the source of her negative reports. To that extent, Taylor’s claims to information based on practical agency resemble those of anyone who is subject to credit reports and background checks. And, of course, Taylor did indeed have access to very general information about the nature of background checks and credit reporting. That might have been sufficient to understand that her background check was a factor in her lost opportunity.

We can capture this sense of Taylor’s claims with what we will call the Principle of Informed Practical Agency.

Surely this principle holds, but it cannot capture the degree to which Taylor’s practical agency was thwarted by ChoicePoint and other reporting agencies. Rather, a key limitation on Taylor’s practical agency is the fact that the reporting agencies systemically included misinformation in her reports. In other words, Taylor’s claims to information are particularly weighty because the background checks at once purport to be grounded in information for which she is responsible (including criminal conduct) and the reports were systemically wrong. Hence, to capture the strength of Taylor’s claims, we can add the following:

Knowing that the background checking system conflates the identities of people with similar names, knowing that her own record includes information pertaining to other people with criminal records, and knowing that the system relies on other background checking companies’ databases and thus repopulates her profile with mistaken information can provide Taylor with tools to address those mistakes. That is, she can better address the wrongs that have been visited upon her by having information about the system that makes those wrongs possible. To be clear, a greater flow of information to Taylor does not make the mistakes and harms to her any less wrongful. Even if it is true that a system is otherwise justifiable, respecting autonomy demands support for practical agency so that people may address the infirmities of that system.

What is key for understanding claims based on practical agency is the distinction we make in Chapter 2 between local autonomy (the ability to make decisions about relatively narrow facets of one’s life according to one’s values and preferences) and global autonomy (the ability to structure larger facets of one’s life according to one’s values and preferences). In many contexts, respect for autonomy is local. Informed consent for undergoing a medical procedure, participating as a subject in research, agreeing to licensing agreements, and the like have to do with whether a person can act in a narrow set of circumstances. Our principles of practical agency, in contrast, concern aspects of autonomy that are comparatively global. One rarely (if ever) provides meaningful consent to having one’s data collected, shared, and analyzed for the purposes of background checks and hence enjoys only a little local autonomy over that process.Footnote ³⁵

Individuals have little (if any) power to avoid credit and background checks and hence do not have global autonomy with respect to how they are treated. However, understanding how their information is used, whether there is incorrect information incorporated into background checks, and how that incorrect information precludes them from opportunities may be important (as in Taylor’s case) in order to prevent lack of local autonomy from becoming relatively more global. That is, mitigating the effects of algorithmic systems may allow one to claw back a degree of global autonomy. And that ability to potentially exercise more global autonomy underwrites a moral claim to information.

The two principles of informed practical agency only tell us so much. They cannot, for example, tell us precisely what information one needs. In Taylor’s case, practical agency requires understanding something about how the algorithmic systems deployed by ChoicePoint actually function, who uses them for what purposes, and how they absorb information (including false information) from a range of sources over which they exercise no control and minimal (if any) oversight. But other decision systems and other circumstances might require different kinds of information. The principles also cannot tell us exactly who needs to be afforded information. While the claim to information in this case is Taylor’s, it may be that her advocate, representative, fiduciary, or someone else should be the one who actually receives or accesses the relevant information. Taylor, for instance, might have a claim that her employer learn about the infirmities in ChoicePoint and Tenant Tracker’s algorithmic systems. The principles cannot tell us the conditions under which persons’ claims may be overridden.

The principles discussed so far only address the epistemic side of practical agency. But Taylor is owed more than just information. We can see this by considering one of the most deeply troubling facets of her case: the reluctance that the data controllers who are involved have toward fixing her mistaken data. One effect of their reluctance is that it undercuts her ability to realize her values, something to which she has a legitimate claim. To capture this, we need – in addition to the principles of informed agency – a principle that lays bare agents’ claim to control.

As before, we need a second principle specifying certain cases where this claim is stronger.

These principles demand of the systems used in the Taylor’s case that she not only is able to learn what information a system is based on, but that she is able to contest that information when it is inaccurate. The claim she has in this case is (just like the principles of informed practical agency) grounded in her agency, that is, her claim to decide what is valuable for herself and pursue those values so long as they are compatible with respect for the agency and autonomy of others.

Now, the principles of informed practical agency and informational control cannot tell us what a person’s informational claims are in cases where they are unable to exercise practical agency. We consider that next.

4.2.2 Argument 2: Cognitive Agency

Cognitive agency can also ground a claim to information. Consider a difference between the Taylor and Arroyo cases. Or, more specifically, a difference between Taylor’s case once she had experienced several iterations of problems with her background checks and Carmen Arroyo’s case after she had been denied housing with her son. Taylor at some point became aware of a system that treats her poorly and for which she bears no responsibility. Arroyo, in contrast, was precluded from moving her son into her apartment for reasons she was unable to ascertain, the basis for the decision was an error, and the result was odious. Denying tenancy to Arroyo’s son Mikhail is surely an injustice. But that wrong is compounded by its obscurity, which precluded Arroyo from interpreting it properly. That obscurity violates what we call the principle of informed cognitive agency.

As before – and for familiar reasons – we will add a second, stronger principle.

Arroyo is an agent capable of deciding for herself how to interpret the decision, and she deserves the opportunity to do so. Her ability to understand her situation is integral in her exercising cognitive agency, but the facts that are crucial for her understanding are that her ability to care for her son is a function of the vagaries of a background check system.

Cognitive agency is implicated in Arroyo’s case in part because her predicament is based on a system that bears on an important facet of her life (being able to secure a place to live and care for one’s child) and purports to be based on actions for which she is responsible (moving her son, who had been subject to criminal charges, into her apartment). The system, meanwhile, is such that it treats old charges as dispositive even though they were withdrawn and as remaining dispositive regardless of whether the person is at present in any position to commit such a crime at all. The reason such facts about the background check system are important is not because they will allow Arroyo to act more effectively to mitigate its effects. She was unable to act effectively when she was precluded from moving her son into her apartment. Rather, those facts are important for Arroyo to be able to act as a cognitive agent by exercising evaluative control over what to believe and how to interpret the incident.

Notice that the criteria for a claim to information based on cognitive agency appears less stringent than for practical agency. However, it does not follow that cognitive agency demands more information. Rather, cognitive agency demands different kinds of information. Because practical agency requires information sufficient to effectively act, it may require technical or operational information. Cognitive agency, in contrast, requires only enough information to exercise evaluative control. In the context of background checks, this might require only that one be able to learn that there is an algorithmic system underlying one’s score, that the system has important limitations, that it is relatively unregulated (as, say, compared to FICO credit score reporting), and the factors that are salient in determining outcomes.Footnote ³⁶

Of course, that leaves us with the question of what information is necessary to exercise evaluative control. Our answer is whatever information is most morally salient, and the claim to information increases as the moral salience of information increases. So, in the case of Arroyo’s background check, morally salient information includes the fact of an automated system conducting the check and the fact that her son’s current condition did not enter the assessment. It is true that there might be other morally salient information. For example, we can imagine a case where the future business plans of CoreLogic are peripherally morally salient to a case; however, a claim to that information is comparatively weaker and hence more easily counterbalanced by claims CoreLogic has to privacy in its plans.

4.2.3 Objections and Democratic Agency

There are several objections to the view we have set out so far that are important to address here. The first is that it proves too much. There are myriad and expanding ways that algorithmic systems affect our lives, and information about those systems bears upon our practical and cognitive agency in innumerable ways. Hence, the potential scope for individuals’ claims to information is vast.

It is certainly true that the principles of informed practical agency and of informed cognitive agency are expansive. However, the principles have limitations that prevent them from justifying just any old claim to information. To begin, the principles of practical agency require that an algorithmic system restrict an individual’s practical agency. How to determine what counts as a restriction, of course, is an interpretative difficulty. For example, does an algorithmic system that calculates one’s insurance premiums restrict one’s practical agency? What about a system that sets the prices one is quoted for airline tickets? Nonetheless, even on a capacious interpretation, it won’t be just any algorithmic system that affects one’s practical agency. Another significant hurdle is that the algorithmic system must affect significant facets of a person’s life. Perhaps insurance rates and airline prices clear that hurdle, but it is close. Other systems, such as what political ads one is served in election season, what music is recommended on Spotify, or which route Google maps suggests to your destination, do not impose restrictions on one’s practical agency.Footnote ³⁷ The requirement that information allows a person to correct or mitigate the effects of an algorithmic system, therefore, is a substantial hurdle for the information to clear. Claims to information that have no such effect would fall under cognitive agency (and as we explain later, information that respects cognitive agency is less onerous to provide).

A second, related, objection is that many people – probably most people – will not wish to use information to exercise practical or cognitive agency. It is cheap, so to speak, to posit a claim to information, but it is pricey for those who deploy algorithmic systems, and the actual payoff is limited. This criticism is true so far as it goes, but it is compatible with the principles we’ve offered. For one, the fact (if it is) that many people will not exercise practical agency does not say much in itself about the autonomy interests one might have in a piece of information. This is much the same as in the case of medical procedures: Few people opt out of care, but information about care remains necessary to respecting their autonomy interests. Moreover, the objection speaks mostly to the strength of individuals’ claims. All else equal, the higher the stakes involved, and the more information can advance practical agency, the stronger the claims. And the more unwieldy it is for entities using algorithmic systems to provide information, the greater are countervailing considerations.

A third objection is that the arguments prove too little. There is presumably a lot of information to which people have some sort of claim, but which does not advance individuals’ practical or cognitive agency. To introduce this objection, let’s start with a claim to information based on cognitive agency. Imagine a person (call him DJ) born into enormous advantage: wealth, social status, educational opportunities, political influence, and so forth. Suppose, however, that these advantages derive almost entirely from a range of execrable practices by DJ’s family and associates: child labor, knowingly inducing addiction to substances that harm individuals and hollow out communities, environmental degradation, and so forth. DJ’s parents, we might imagine, shield him from the sources of his advantage as he grows up, and when he reaches adulthood, he does not inherit any wealth (though of course he retains all the social, educational, and political benefits of his privileged upbringing). The degree to which his ignorance limits his practical agency is not clear, given his advantages.Footnote ³⁸ However, on the view we outline in the previous section, DJ’s parents certainly limit his cognitive agency by continuing to shield him from the sources of his advantage; he is precluded from understanding important facts about his life, as well as the chance to interpret his circumstances in light of those facts.

DJ is not the only person whose cognitive agency is a function of understanding the source of enormous wealth and advantage. Anyone who has an interest in their society’s social, political, financial, and educational circumstances has some claim to understand how DJ’s family’s and associates’ actions bear upon those circumstances. And that is true regardless of whether they are in any position to change things. In other words, it is the fact that DJ’s family’s actions have an important effect on the world that grounds others’ claims to information, not strictly how those actions affect each individual.Footnote ³⁹ But it is difficult to see how the importance of that information is a function of either practical or cognitive agency.

With that in mind, let’s return to algorithmic systems. In path-breaking work, Latanya Sweeney examined Google’s AdSense algorithm, which served different advertisements, and different types of advertisements, based on names of search subjects.Footnote ⁴⁰ Sweeney’s project began with the observation that some advertisements appearing on pages of Google search results for individuals’ names suggested that the individuals had arrest records. The project revealed that the ads suggesting arrest records were more or less likely to appear based on whether a name used in the search was associated with a racial group. That is, advertisements suggesting arrest records appeared to show up more often in Google ads served for searches that included names associated with Black people than in ads served for searches that included names associated with White people. This result was independent of whether the searched names actually had arrest records.Footnote ⁴¹ While Sweeney did not have access to the precise mechanism by which the AdSense algorithm learned to serve on the basis of race, as she explains, a machine learning system could achieve this result over time simply by some number of people clicking on ads suggesting arrest records that show up when they use Google to search for Black-identifying names.Footnote ⁴²

But what does this have to do with agency and information? After all, as Sweeney points out, the ads themselves may be well attuned to their audiences, and it might be that search engines have a responsibility to ensure that their targeted advertising does not reflect race simply on the basis of harm prevention. But our argument here is different. It is that people have claims to information about some kinds of algorithmic systems even where their individual stake is relatively small, even where the system is reliable, and even where the system makes no assumptions about responsibility. So while people who are White have relatively little personal stake in the issue of search engine advertising serving ads that suggest arrest records disproportionately to searches using Black-identifying names, they have an interest based on agency nonetheless. Specifically, they have an interest in exercising agency over areas of democratic concern.

For the moment we will call this democratic agency and define it as access to information that is important for persons to perform the legitimating function that is necessary to underwrite democratic authority. We will take up this facet of agency and autonomy in more detail in Chapter 8. The gist of the idea is this. Whether a democratic state, set of policies, actions, regulatory regimes, and so forth are justifiable (or legitimate) is an important part in the function of the autonomy of its citizens. Exercising the autonomy necessary to serve this legitimating function requires certain kinds of information. Google of course is not a state actor, but it serves an outsized role in modern life, and understanding how that interacts with basic rights (including treatment of people based on race) is important for people to understand.

4.3.1 The Right of Access and the Right to Rectification

Article 15 of the GDPR outlines the right of access, which is the (legal) right of data subjects who are citizens of the EU to obtain from data controllers confirmation as to whether or not their personal data are being processed, confirmation that personal data shared with third parties is safeguarded, and to obtain a copy of personal data undergoing processing.Footnote ⁴⁴ Article 16 outlines the right to rectification, which is “the [legal] right to obtain from the controller without undue delay the rectification of inaccurate personal data concerning him or her.”Footnote ⁴⁵ These legal rights can be underwritten by the same ideas that support the principles of practical and cognitive agency and the principles of informational control, and we can use the principles to underwrite them.

Begin with rectification. Where one’s data is being used to make decisions affecting significant facets of one’s life – such that the system restricts one’s agency – the principle of informational control tells us that there is a defeasible claim to correcting that information. Insofar as our data is being used to make decisions about us that will affect us, the right to rectification stands as a law that enjoys justification from this principle.

With these ideas in place, we can also offer a justification for the right of access. To know whether a controller has incorrect information about us or information that we do not want them to have or share, we need to know what information they in fact have about us. And so, if the right to rectification is to have value, we need a right of access. We can further support the right of access by reflection of the principles of practical and cognitive agency: Often we will need to know what information is being collected in order to improve our prospects or to simply make sense of decisions being made about us.

4.3.2 The Right to Explanation

Consider next the right to have significant automated decisions explained. The Arroyo case brings out the importance of this right. To respond to their predicament, Carmen and Mikhail need to understand it. We begin with a general discussion of the right.

Sandra Wachter, Brent Mittelstadt, and Luciano Floridi introduce two helpful distinctions for thinking about the right to explanation.Footnote ⁴⁶ The first of these distinctions disambiguates what is being explained. A system-functional explanation explains “the logic, significance, envisaged consequences and general functionality of an automated decision-making system.”Footnote ⁴⁷ In contrast, a specific decision explains “the rationale, reasons, and individual circumstances of a specific automated decision.”Footnote ⁴⁸ Note that if a system is deterministic a complete description of system functionality might entail an explanation of a specific decision. So, in at least some cases, the distinction between the two kinds of explanation is not exclusive.Footnote ⁴⁹

The second distinction disambiguates when the explanation is being given. An ex ante explanation occurs prior to when a decision has been made. An ex post explanation occurs after the decision has been made. Wachter et al. claim that ex ante explanations of specific decisions are not possible; a decision must be made before it is explained. As Andrew Selbst and Julia Powles point out, in the special case of a complete system-level explanation of a deterministic system, decisions are predictable and thus, ex ante explanations of those decisions are at least sometimes possible.Footnote ⁵⁰

Rather than stake a claim in this dispute, we will take a pragmatic approach. We can say all we need to say about the right to explanation by discussing the three categories that Wachter et al. admit of (i.e., ex ante system functional, ex post system functional, and ex post specific). If a subject has a right to an ex ante explanation of a specific decision, the arguments for such explanations will follow naturally from our arguments for specific explanations; the only issue that the right will turn on is whether such explanations are possible – an issue that we are not taking a stand on here. We think that, morally, the right to explanation could encompass any of the possibilities Wachter et al. outline. So we will understand the right to explanation as the right to explanations about ex post specific decisions, ex ante system function, or ex post system function.

Let us then work through some ideas about what our account says about the right to explanation.

Ex ante system-functional explanations: Subjects of decisions that have not yet been made often have good reason to know how decisions of that sort will be made in the future. The principles of practical agency delineate some of these conditions.

One way to see this is to return to Catherine Taylor. She now knows that because of her common name, systems that perform quick, automated searches are prone to making mistakes about her. Based on this, she has an interest in knowing how a given system might produce a report on her. If she knows a system is one that might produce a false report about her, she can save herself – and the purchaser of the report – quite a bit of trouble, either by insisting to ChoicePoint that more careful methods are used or by preempting the erroneous results by providing an independent, high-quality counter-report of her own.

Ex post system-functional explanations: Subjects of decisions that have been made often have good reason to know how those decisions of that sort were made. These claims can be grounded in practical or cognitive agency.

Consider Taylor again. If Taylor is denied a job and she learns that an automated background check was involved, she has reason to suspect that the automated check might have erroneously cost her the opportunity. For her, simply knowing the most general contours of how a system works is powerful information. This alone may be enough to allow her to get her application reviewed again, and she could not reasonably endorse a system where she is denied this minimal amount of information. But even if she cannot accomplish this – that is, even if the principle of informed practical agency is not activated because her situation is hopeless – she still has a claim, via the principle of informed cognitive agency, to gain an understanding of her situation.

Specific explanations: Finally, subjects of decisions often have good reason to know how those specific decisions were made. These claims can be grounded in practical agency.

Recall Arroyo’s denial of housing. Something is wrong with Arroyo’s report, yet his mother does not (and cannot) know what it is. This leaves her especially vulnerable in defending her son, since she does not know what to defend him against. As the principle of informed practical agency demands, subjects of decisions that have been made should at least know enough about those decisions to respond to them if they have been made in error.

We want to pause briefly to discuss a recent proposal pertaining to how specific explanations might be given, namely via counterfactual explanations, which have been detailed extensively in a recent article by Wachter et al. An example of a counterfactual explanation, applied to the Arroyo case, is as follows

Generalizing a bit, counterfactual explanations are explanations of the form “W occurred because X; Were Y to have been the case (instead of X), Z would have occurred (instead of W),” where W and Z are decisions and X and Y are two “close” states of affairs, identifying a small – perhaps the smallest – change that would have yielded Z as opposed to W.

Counterfactual explanations have several virtues qua specific explanations. For one, they are easy to understand.Footnote ⁵¹ They are efficient in communicating the important information users need to know to make sense of and respond to decisions that bear on them. Thus, such explanations are often sufficient for giving subjects what they are informationally owed. Another virtue is that they are relatively easy to compute, and so producing them at scale is not onerous: Algorithms can be written for identifying the smallest change that would have made a difference with respect to the decision.Footnote ⁵² Further, they communicate needed information without compromising the algorithms that underlie the decisions they explain; they offer explanations, as Wachter et al. put it, “without opening the black box.”Footnote ⁵³

Counterfactual explanations can serve as a useful tool for delivering what is demanded by the cognitive and practical agency of data subjects without running roughshod over the interests of their data controllers. Of course, such explanations will not always meet these demands; they will only work in contexts where specific explanations are called for. And even then, they might not always offer everything an agent needs; for instance, one could imagine counterfactual explanations that are too theory laden to be useful or that are only informative against knowledge of myriad background conditions. Nevertheless, this style of explanation can be a very useful tool in meeting agents’ needs. Thus, they serve as a good example of a realistic tool for giving data subjects what they are informationally owed.

Let’s take stock of what our account has to say about the right to an explanation. We take it that the right to explanation is a defeasible right to meaningful ex post, ex ante, system-level, and specific explanations of significant, automated decisions. Using our cases and principles, we have demonstrated how our account can underwrite a claim: As autonomous beings, we need to understand significant events in our lives in order to navigate the world so as to pursue our values; as autonomous beings, we have a duty to support each other’s autonomy; so, if we are in control of information pertaining to significant decisions affecting someone’s life, we often owe it to them to make that information available.

4.3.3 The Right to Object

In addition to rights of access and rights to rectification and explanation, the GDPR outlines the right to object, “the right not to be subject to a [significant] decision based solely on automated processing.”Footnote ⁵⁴ As stated earlier, our interest is in understanding whether there is a moral right to object. However, examining a version of a legal right can help us make sense of moral claims. There are two key features of the right to object as it is stated in the GDPR.

Note first that the right is vague. Specifically, the “based solely” condition, as well as the notion of significance, admits of vagueness. As Kaminski notes,

We will not focus too heavily on issues of vagueness here. However, it is important to note that the limiting condition of the right – as well as some of its content – is vague.

Second, the right to object is ambiguous.Footnote ⁵⁶ It could be understood broadly: as a broad prohibition on decisions that are based solely on automated processing. The same right could also be understood narrowly: as an individual right that data subjects can summon for the purposes of rejecting a particular algorithmic decision.Footnote ⁵⁷ Here, we won’t be interested in adjudicating which way to read Article 22 of the GDPR, because we regard both readings as supported by the same considerations that we cite in favor of the right to explanation.

Human oversight of an automated decision system requires that the system be functionally intelligible to at least some humans (perhaps upon acquiring the relevant expertise). So, in a world where the broad reading is observed, each significant automated decision is intelligible to some human overseers. What this means, in turn, is that the reasons for its decisions could be meaningfully explained to data subjects (or at the very least to their surrogates). The significance of this, from our point of view, is that it would help to secure the right to explanation, as it would require systems to be designed so that they are intelligible to humans. Further, in cases where a data subject cannot request an explanation, it serves to assure them that significant automated decisions made about them make sense. Similarly, in a world where the narrow right is observed, systems are designed to be intelligible so that, were their decisions meaningfully checked by a human decision maker, they would make sense. This, of course, means that they are designed so that they do make sense to humans (even if those humans are experts). Moreover, like the broad reading, it also affords data subjects the opportunity to have decisions checked when they themselves cannot check them (perhaps for reasons of trade secrecy). However, the narrow right might sound more plausible than the broad right because it means fewer human decision makers would have to be employed to satisfy it, allowing systems to operate more efficiently.

Now, unlike the previously mentioned rights, the right to object – particularly in its broad formulation – might sound onerous. However, abiding the rights to access, rectification, and explanation already requires that data controllers provide data subjects meaningful human oversight of decisions made about them, so perhaps the broad right isn’t as implausible as it may first seem. Further, the broad right has the advantage that it makes the exercise of the right to object less costly to those individuals who would otherwise have to explicitly exercise it. We can imagine data subjects worrying that they will face prejudice for exercising the right; for instance, a job applicant might worry that if she exercised the right, the potential employer will think that she is going to cause trouble.

What does the right to object add, then? Importantly, there are systems where inferences must be kept secret – either to prevent subjects from gaming it or because the system is simply too complicated – in these circumstances, the right to object plays the important role of ensuring that surrogates of data subjects understand whether high-stakes decisions made about those subjects make sense.

4.4.1 Loomis

One of Loomis’s primary complaints in his appeal is that COMPAS is proprietary and hence not transparent. Specifically, he argued that this violated his right to have his sentence based on accurate information. He bases the argument in part on Gardner v. Florida.Footnote ⁵⁸ In Gardner, a trial court failed to disclose a presentence investigation report that formed part of the basis for a death sentence. The U.S. Supreme Court determined that the failure to disclose the report meant that there was key information underwriting the sentence which the defendant “had no opportunity to deny or explain.” Loomis argued that the same is true of the report in his case. Because the COMPAS assessment is proprietaryFootnote ⁵⁹ and because there had not been a validation study of COMPAS’s accuracy in the state of Wisconsin (other states had conducted validation studies of the same system), Loomis argued that he was denied the opportunity to refute or explain his results.

The Wisconsin Supreme Court disagreed. It noted that Northpointe’s Practitioner’s Guide to COMPAS Core explained the information used to generate scores, and that most of the information is either static (e.g., criminal history) or in Loomis’s control (e.g., questionnaire responses). Hence, the court reasoned, Loomis had sufficient information and the ability to assess the information forming the basis for the report, despite the algorithm itself being proprietary.Footnote ⁶⁰ As for Loomis’s arguments that COMPAS was not validated in Wisconsin and that other studies criticize similar assessment tools, the court reasoned that cautionary notice was sufficient. Rather than prohibiting use of COMPAS outright, the court determined that presentence investigation reports using COMPAS should include some warnings about its limitations.

According to the principles of practical agency, Loomis has a defeasible claim to information about COMPAS if (a) information about COMPAS advances his practical agency, (b) COMPAS has restricted his practical agency, (c) COMPAS’s effects bear heavily on significant aspects of Loomis’s life, and (d) information about COMPAS allows Loomis to correct or mitigate the effects of COMPAS. If there is such a claim, it is strengthened (e) if COMPAS purports to be based on factors for which Loomis is responsible and (f) if COMPAS has errors.

It is certainly plausible that COMPAS limits Loomis’s practical agency insofar as it had some role in his sentence. Loomis faced a number of decisions about what to do in response to his sentence. One is whether he should appeal and on what grounds. Another is whether he should try to generate public support for curtailing the use of COMPAS. For Loomis, settling these questions about what to do depends on knowing how COMPAS generated his risk score. And there is much he doesn’t know. He doesn’t know whether the information fed into COMPAS was accurate. He doesn’t know whether, and in what sense, COMPAS is fair. And he doesn’t know whether the algorithm was properly applied to his case. That lack of information curtails his practical agency. The length of his criminal sentence certainly involved a significant facet of his life, and it is at least plausible that greater information would allow him to mitigate COMPAS’s effects. The strength of his claims increases in light of the fact that it is best understood as being based on factors for which he is responsible, viz., his propensity to reoffend.

So Loomis has a prima facie and defeasible claim to information about COMPAS. But that leaves open just what kind of information he has a claim to, what that claim entails, and whether there are countervailing considerations that supersede Loomis’s claim. It would seem that Loomis needs to know that the data fed into COMPAS was accurate, evidence that COMPAS is in fact valid for his case, and, finally, some kind of explanation – perhaps in the form of a counterfactual explanation – that makes clear why he received the score that he did. Such information would advance Loomis’s practical agency, either by giving him the information needed to put together an appeal or by demonstrating to his satisfaction that his COMPAS score was valid, allowing him to focus his efforts elsewhere.

Independent of the concerns based on practical agency, Loomis has a claim to information based on cognitive agency. Both factors in the principle of informed cognitive agency are present. COMPAS purports to be based on factors for which Loomis is responsible, and the stakes are high. Being imprisoned is among the most momentous things that may happen to a person and understanding the basis of a prison sentence is essential to one’s agency. That extends beyond the factors that matter in determining one’s sentence to include whether the process by which one is sentenced is fair. And as we have argued, agents have a claim to understand important facets of their situations. Hence, Loomis has a claim based on cognitive agency to better understand the grounds for his imprisonment.

While Loomis plausibly has claims to information based on both practical and cognitive agency, there are differences in what those claims entail. While practical agency will only underwrite information that can be used in advancing Loomis’s case – and hence, mostly supports information for Loomis’s legal representation – cognitive agency underwrites the provision of certain pieces of information to Loomis himself. It would involve providing him information about the fact that a proprietary algorithm is involved in the system, information about how well the system predicts reoffense, and information about the specific factors that led to Loomis’s sentence. There is no reason to think that it would advance Loomis’s cognitive agency to provide him with specific information about how COMPAS functions.

Moreover, the court did, in fact, respect Loomis’s cognitive agency. The Wisconsin Supreme Court upheld the circuit court’s decision in substantial part because the circuit court articulated its own reasons for sentencing Loomis as it did. In other words, it provided an account sufficient for Loomis to exercise evaluative control with respect to his reactive attitudes toward the decision and sentence.

4.4.2 Wagner and Houston Schools

The principles of informed practical agency and informed cognitive agency also aid our understanding of the K-12 teacher cases, especially Houston Schools. Recall that Houston Schools uses a VAM called EVAAS, which produces each individual teacher’s score by referencing data about all teachers.Footnote ⁶¹ This practice makes EVAAS’s scores highly interdependent. Recall also that Houston Schools was frank in admitting that it would not change faulty information because it would require a costly reanalysis for the whole school district and the potential to change all teachers’ scores. This was all despite warnings (as we note in Chapters 1 and 3) that value-added models have substantial standard errors.Footnote ⁶² So EVAAS’s scores are extremely fragile, produced without independent oversight, and cannot be corroborated by teachers (or the district or, recall, an expert who was unable to replicate them).

It seems clear enough that information about EVAAS is vital for teachers to exercise practical agency. Certainly, it is relevant to several significant aspects of teachers’ lives. For teachers who were fired or did not have their contracts renewed based on low performance, gaining an understanding of the system advances their practical agency in a couple of ways. It gives them (and their union leaders and lawyers) the bases of either an appeal (whether in court or to the public) of the firings or an appeal of the system altogether. It also gives teachers who are finding employment in other schools some context that could help them convince administrators that their departure from the Houston Independent School District (HISD) was not evidence of poor teaching. That is, affected teachers have a (defeasible) claim to information about EVAAS’s functioning, because it could allow them to correct or mitigate the system’s effects. Their claim is strengthened because EVAAS purports to be based on factors for which the teachers are responsible (viz., their work in the classroom), and yet (as HISD admits) EVAAS has errors. These claims also underwrite teachers’ claims to informational control, specifically their claim to have any inaccuracies reflected in their scores corrected.

The fact that EVAAS affects such important parts of teachers’ lives and purports to be based on factors for which they are responsible also gives them a claim to information based on cognitive agency. As in the COMPAS case, the type of information necessary for teachers to exercise evaluative control – that is, to assess their treatment at the hands of their school system – may be different from the information necessary for them to exercise practical agency. Cognitive agency may only require higher-level information about how EVAAS works, a frank assessment of its flaws, and a candid accounting of Houston Schools’ unwillingness to incur the cost of correcting errors rather than the more detailed information necessary for teachers to correct errors. To put a bookend on the importance of cognitive agency, we will return to an exemplary teacher’s public reaction to the VAM used by DC Schools: “I am baffled how I teach every day with talent, commitment, and vigor to surpass the standards set for me, yet this is not reflected in my final IMPACT score.”Footnote ⁶³ This would seem to be an appeal to exercise evaluative control.