1.2.1 The Interdependence of Individual and Group Profiling

An initial controversy surrounding algorithmic profiling based on large sets of digital footprints is whether the individual or the group is its real target and the potential object of manipulative practices. Whilst the language of personalisation and customisation suggests the individual is the focal point, in some ways ‘personalisation’ is a misnomer, as individual profiling is always a form of classification whereby the individual is assessed against group attributes (more on that below) and then put in a micro-category for the purpose of delivering the ‘personalised’ response or service. Thus although the individual is the target of the customised message, service or treatment, the outcome is based on group features and multiplied across the micro-group. Furthermore, the fact that individual profiling is premised on analysing data sets about populations – mined for correlations and leading to the construction of groups in the process – has led some to conclude that group profiling is the critical new phenomenon that challenges existing legal modalities:

This argument has some validity (given that privacy regimes envisage an individual victim, and harm to the group only derivatively), but the ability to micro-profile individuals is still at least as valuable to corporate and governmental actors as knowledge about the group, as borne out by the widespread emergence of personalisation practices. In any event, the individual-versus-group dichotomy may largely be misconceived because they reflexively interact with each other. Individual data feeds into population data sets and these sets produce, through correlations, knowledge about populations, that is patterns and groups within them (inductive), which in turn are instructive about the individual (deductive).

The close, yet varying, integration of individual and group profiles has been subject to some debate and conceptualised in the distinction between distributive and non-distributive group profiling.¹⁶ For distributive profiles (universal generalisations) attributes of the group are ‘actually and unconditionally manifested by all the members of that group’¹⁷ and thus group membership also allows for definitive inferences about the attributes of its members.¹⁸ Every member of university staff (the group) has an employment contract with the university and a salary (attributes). In contrast, non-distributive profiles (non-universal generalisations or stereotyping) refer to groups where a family resemblance unites members, but not every member shares every attribute.¹⁹ Here ‘a group is defined in terms of… significant deviances from other groups. They are based on comparisons of members of the group with each other and/or on comparisons of one particular group with other groups.’²⁰ The group boundaries in non-distributed profiles are inevitably fuzzy. Those with a high risk of cardiovascular disease (group) share a number of risk factors, for example, lifestyle, genes, age, weight, etc. (attributes),²¹ but membership does not allow for definitive inferences about the particular attribute of a particular member. The non-universal generalisation that ‘young men drive recklessly’ does not allow for a definitive inference about the driving of any particular young man but, as argued below, mistakes on the individual level are often legitimated by the benefits of identifying (empirically sound) tendential truths.

Whilst non-distributive profiling explicitly compares one group vis-à-vis other groups, ultimately the distinctiveness of a distributive group profile (university staff) can also only be understood against other groups, that is what it is not (police or hospital staff, or university students). Indeed, the difference between these two types of profiling may in practice (and theory) not be that clear cut (i.e. is the whiteness of swans ‘necessarily manifest’ or non-essential?) and becomes largely a function of the profiler’s knowledge, pre-conceptions and attendant construction of the group. This suggests that the certainty of (empirically based) distributive profiles may be illusory.²² The two types of profiling may simply reflect different philosophical traditions: distributive profiles adopt a Platonic top-down perspective on a concept or class that assumes and finds a common essence underlying all its manifestations, whilst non-distributive profiling builds on Wittgenstein’s bottom-up (and empiricist) notion of family resemblance whereby concepts or words just refer to clusters of similar or related phenomena:

So arguably distributive and non-distributing group profiling does not refer to different types of groups, but rather to different ways of looking at the same group, or, more precisely, to different ways of constructing groups.

As non-distributive profiling can capture a wider range of relevant, albeit non-essential, attributes (as opposed to seeking a group’s essence), it yields a much richer picture of groups and individuals, but also has blurry edges and is fallible in respect of making definitive inferences about its members.²⁴ This is significant for big-data individual profiling, or any form of statistical profiling: when individual profiles are inferred from comparison with the group (indirect profiling), it may be tempting to fill ‘gaps’ in an imperfect overlap with the missing group attributes. For example, in the policing context, a large aggregated criminal justice database with data on criminal activities mapped onto post codes, on criminal records and recidivism, social media activities and networks, education and employment histories of offenders, and personality traits may – based on strong correlations – predict for a particular offender a high risk of recidivism. The Harm Assessment Risk Tool, or the HART algorithm, used by Durham Constabulary makes such predictions based on 509 ‘votes’ by the system.²⁵ A digital footprint on social media may, in the absence of explicit evidence, be analysed to ‘reveal’ the missing attribute of a single person’s status, a left-wing political outlook or homosexuality. Based on the strength of the correlation, an unknown attribute may be ‘highly likely’ and in this respect fall somewhere between the distributive and non-distributive profiles – as neither necessarily manifest nor simply possible. However, the effect of the use of the predictive technology will often be such as to treat highly likely predictions as effectively established, along the lines of Plato’s essentialism. Yet, there may be rights-based reasons, such as the presumption of innocence or the right to privacy, why a particular inferred attribute should be treated as non-essential and its absence presumed, as, for example, when sensitive data may be inferred from a range of non-sensitive data points. (Chapter 5)

1.2.2 The Virtues of Stereotyping

One persistent objection to individual profiling based on comparisons to group data, including big data profiling, is that the resultant stereotyping (or non-universal generalisations) leads to the ‘deindividualisation of the person’ which occurs when ‘[p]ersons are judged and treated more and more as members of a group (i.e. the reference group that makes up the data or information subject) rather than as individuals with their own characteristics and merits.’²⁶ This critique is directed at indirect profiling that draws inferences about the individual from group data (invariably through non-universal generalisations), as opposed to direct profiling that is ostensibly based on data only about the particular individual and therefore arguably more accurate.²⁷ The objection to the ‘deindividualisation of the person’ or stereotyping based on comparisons with the group is flawed for a number of reasons. First, the argument that direct profiling delivers prima facie more legitimate profiles as it is solely focused on the digital footprint of the single individual assumes that someone’s past activities and preferences provide a valid yardstick for his future behaviour and preferences, and implicitly assumes that personhood is fixed in time. Such reasoning relies as much on stereotyping of the individual (and on denying agency) as indirect profiling, as it does not allow for the possibility of continual reinvention and development of individuality through repeated assertions of free choice. Indirect profiling has at least the virtue of squarely acknowledging that ‘no man is an island’ and that individuality is intimately tied up with social forces within which it develops and against which it may be understood. Still, all profiling used for predictive purposes is inherently irreconcilable with the notion of free will as underwriting moral and legal responsibility as well as autonomous participation in democratic processes.

Second, direct profiling is also necessarily comparative with the group, much like indirect profiling, and cannot but invoke the social dimension of human existence. Individuality can only be understood against an assumed ‘normality’ which contextualises individual divergence.²⁸ An individual’s social media digital footprint is entirely meaningless by itself, in a social vacuum. It can only signal depression or creditworthiness or criminogenic tendencies against data sets, drawn from the group, that display the whole spectrums of psychological, financial or criminogenic states.²⁹ The interdependence of the individual and the group, the particular and the general, uniqueness and commonality, may best be illustrated with reference to DNA profiling as the biological equivalent to behavioural profiling:

In parallel with the biological profile, where commonality far outweighs uniqueness, and individual genomic variations operate, and are identifiable, against genomic commonality, individual behavioural uniqueness can also only be conceptualised against the broad brush of collective humanity. The specific and the general are co-dependent. (See Chapter 5.)

Last, but not the least, even if direct and indirect profiling are, after all, not so fundamentally different from each other by being comparative and engaging in stereotyping, the crux of the problem may lie in stereotyping per se. The argument against stereotyping appears to have found legal recognition in antidiscrimination law: ‘Stereotyping, or the imposition of assumptions about a group on an individual, has been central to antidiscrimination law because of the prominence of individual autonomy as a juridical value.’³¹ This assertion, however, is misleading in its generality, considering that antidiscrimination law only addresses stereotyping based on a very limited range of factors, for example, race, gender or age. It does not outlaw stereotyping per se, nor could it. Human judgment and knowledge invariably, and necessarily, involves stereotyping – or non-universal generalisations – and this is neither irrational nor immoral, assuming it has a sound empirical basis and excepting certain historically disadvantaged groups, as protected by antidiscrimination law. Frederic Schauer in Profiles, Probabilities, and Stereotypes has argued for the ubiquity of stereotyping and its prima facie legitimacy:

In all these cases, stereotyping, or non-universal generalisation, is based on an acceptance of inaccurate results in particular cases (e.g. in a job, a particular student with poor grades might outperform the students with good grades; a plane from an airline with a good record may crash), but is still justifiable on the ground of efficiency. Shortcuts and proxies (e.g. the grade, the tattoo, the airline brand) reduce informational complexity and thereby facilitate decision-making that is faster and, if empirically sound, also tendentially correct, albeit not 100 per cent. In fact, as Schauer shows, such stereotyping often leads overall to fewer mistakes than a case-by-case approach where wide discretion and factual granularity and complexity introduce far more room for errors of judgement and inconsistencies.³³ In other words, from an efficiency perspective, empirically sound stereotyping allows overall for faster and better decisions. Having said that, the possibility of mistakes is problematic in serious contexts, such as criminal justice, which has traditionally been deeply individualistic and thus preoccupied with avoiding false positives.

Considering that the antithesis of stereotyping is a case-by-case or particularised approach, stereotyping is a form of rule-based decision-making: a proxy provides a simple instrumentalised yardstick against which individuals (or products or situations) are measured, and can be seen to be measured. So counterintuitively, stereotyping has the moral virtue, first, of facilitating – at least in principle – transparency and accountability, as judging by proxy creates simplicity (by removing informational noise) and makes it possible to understand how a judgment was reached. Given this transparency, the stereotyped individual is also often, not always, able to actively participate in his perception by others, for example, by working towards good grades or getting a tattoo. Stereotyping is then a two-way process whereby the individual deliberately sends relevant signals for the judgment of others, or contests the accuracy of the perceived stereotype in her particular case. Second, generalisation or stereotyping is also virtuous because applying a proxy equally to all, in disregard of individual differences, is aligned with the concept of formal equality, which in turn is fundamental to our understanding of fairness and justice. Resorting once more to Schauer, the value of formal equality lies in ‘understanding our common situation and our common plight as one in which there are limits to how much difference our own personal individual situations ought to make. It is no accident that Justice wears a blindfold. And she wears a blindfold not because she needs to steel herself against her own biases, prejudices, and mistakes, but because it is central to one conception of justice that equal treatment for its own sake – treating unlikes alike – serves important functions.’³⁴ Thus stereotyping applies a social framing to individual treatment, stressing commonality over differences, and thereby reinforces community and affinity.

Outside the legal or social context, the systematic use of proxies to make non-universal generalisations about the external environment (e.g. brown bears = danger; koalas ≠ danger) produces knowledge that enables choices about alternative courses of action and is thus fundamental to evolutionary success. Mireille Hildebrandt observed that ‘all living organisms, in order to survive, must continuously profile their environment to be able to adapt themselves and/or adapt the environment.’³⁵ Taking this perspective, profiling and stereotyping is an essential survival mechanism that consists of abstracting (life-critical) information from the environment in order to adapt to it, or vice versa. It is goal oriented and selective by focusing only on relevant features and by making generalisations that are tendentially true but not always – much like present day profiling by autonomous technology.

In short, whilst there may often be valid arguments about the empirical (un)soundness or other (il)legitimacy of a particular basis of stereotyping, in general, judging by proxy is not just intrinsic to the cognitive processing of external reality and fellow humanity, but also has solid moral foundations. Therefore, in so far as objections to direct and indirect profiling are centred on the intrinsic wrongfulness of stereotyping, they are based on a misunderstanding about its role in knowledge production and moral discourse. By the same token, big data-driven profiling and personalisation practices may be criticised for undue stereotyping, where the profiling is empirically unsound, or draws on proxies for groups that have suffered long-standing structural disadvantages. (Chapter 11) Furthermore, big data profiling does not facilitate stereotyping practices in which everyone engages equally, but is underwritten by power asymmetries which it consolidates and sharpens, to which the discussion will turn now.

1.3.1 Blind Knowledge of Data-driven Profiling: Correlations, Not Causation

Big data analytics, of which individual and group profiling is a prominent example, constitutes – in method, if not aim – the quintessential manifestation of the epistemological tradition of empiricism. For empiricists, in contradistinction to rationalists, the only reliable source of knowledge is information gathered through sense experience or observation. Knowledge follows experience rather than precedes it. For the early English philosopher of science and Enlightenment empiricist Francis Bacon, scientific knowledge had to be based on observations of nature and could only be produced through inductive reasoning – yet not without being guided by some form of hypothesis: ‘the true method of experience, on the contrary, first lights the candle, and then by means of the candle shows the way’.³⁶ David Hume, writing more than a century later, was equally persuaded of the need for observation in matters of fact (as opposed to relations of ideas, i.e. mathematics or logic):

As a philosopher, Hume was sceptical about inductive reasoning as a source of true knowledge. For him an observed ‘constant conjunction’ or correlation (i.e. the sun has risen in the past) does not give us an epistemic basis to know with certainty that the same will occur in the future (i.e. the sun will rise tomorrow) – even if observers generally jump to such conclusions and thereby imply an underlying scientific law about cause and effect, from which the future can demonstratively be known.

Hume’s sceptical empiricism is uncannily enacted by big data analytics in general, and individual and group profiling in particular – even if in a distorted fashion. Big data analytics produces ‘blind’ knowledge, or knowledge of ‘constant conjunctions’ or correlations, and often stops at that. Prima facie it is the correlation, rather than an explanation for the correlation or true knowledge, that matters. Methodologically, big data analytics makes a hypothesis drawing on a theory – as the foundation for trial-and-error scientific method – far less important than it is for traditional scientific discovery. Knowledge discovery in databases occurs by searching for relevant correlations or patterns with the help of algorithms, and ‘these algorithms can be supervised, that is they can start with a hypothesis that is tested on the data, or they can be unsupervised,… and just check for any patterns… [in line with a particular] mathematical function.’³⁸ Thus an unsupervised algorithm proceeds – without Bacon’s candle – to search itself for structures (or groups) in the input data towards a desired output. Having said that, there is a hybrid methodology whereby initial data results are used to generate the hypothesis for further statistical testing on other (validation) data.³⁹ Second and overlapping, in Big Data: A Revolution That Will Transform How We Live, Work, and Think,⁴⁰ Viktor Mayer-Schönberger and Kenneth Cukier commented that the move from a small-data to a big-data world has unleashed the power of correlations. The abundance of data and sophisticated analytical tools allow for faster and cheaper identification of correlations on a much wider range of subjects – with or without a posteriori explanations about cause and effect. Indeed, attempts to explain correlations are, according to the authors, often ‘caught in a web of competing causal hypotheses… [and] only make them cloudier. Correlations exist; we can show them mathematically. We can’t easily do the same for causal links. So we would do well to hold off from trying to explain the reason behind the correlations: the why instead of the what.’⁴¹ Thus they conclude, in a twisted nod to Hume, that ‘[c]ausality won’t be discarded, but it is being knocked off its pedestal as the primary fountain of meaning.’⁴² Similarly, Chris Anderson had previously – in ‘The End of Theory: The Data Deluge makes the Scientific Method Obsolete’⁴³ – argued that the abundance of data displaces every theory of human behaviour, from linguistics to sociology or psychology, given that it no longer matters knowing ‘why people do what they do[.] The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.’⁴⁴ These commentators effectively dissolve Hume’s intractable dilemma about the possibility of true knowledge (which Kant had already done rather effectively in 1781⁴⁵) by asserting that such knowledge is now redundant.

This argument for the virtues of instrumentalised blind ‘knowledge’ of big data analytics (if it can be called knowledge) presents a full inversion of Enlightenment ideals of rationality and scientific discovery, even if the claim of a paradigmatic shift in science is to some extent exaggerated. Some theory invariably drives and follows the search for correlations.⁴⁶ Mireille Hildebrandt has shown how wider theoretical assumptions are necessarily embedded, even within unsupervised learning algorithms, in the constructions of a model of reality by translating external reality into data sets, and ‘theory’ is also implicated in reviewing the validity of found correlations.⁴⁷ Furthermore, data simply never speaks for itself: ‘Making sense of data is always framed – data are examined through a particular lens that influences how they are interpreted.’⁴⁸ Still, whilst exaggerated, the core assertion about the new approach to discoveries stands, even if it over-promises results outside retail and marketing. (Chapter 15) Peter Coveney and others have argued – using precision or personalised medicine and human genomics projects as an example – that big data knowledge discoveries often fail beyond the big-yet-finite data sets on which they are trained precisely ‘because they are not designed to model the structure characteristics of the underlying system.’⁴⁹ In the human behavioural domain, data sets may prove to be ‘finite’ in terms of being time-specific or valid only for certain sections of the population. From a scientific perspective, there is also the rather more principled objection about discarding the quest for real knowledge in favour of the new instrumentalised ‘knowledge’: ‘In subjects where the level of theoretical understanding [i.e. physics and chemistry] is deep, it is deemed aberrant to ignore it all and resort to collecting data in a blind manner. Yet, this is precisely what is advocated in the less theoretically grounded disciplines of biology and medicine, let alone social sciences and economics.’⁵⁰ Notably, these latter disciplines provide the scientific backdrop within which individual and group profiling generally falls.

From an economic perspective, the blind ‘knowledge’ of big data analytics is wholly legitimated by efficiencies: if data-driven profiling generates revenue or saves scarce public resources, and effectively predicts (or moulds) choices, it matters not what the underlying reasons may be. On a more generic level, Andrew Feenberg has argued that efficiency and control are the inherent animating forces of ‘technology’ rather than values that offer an outside perspective on technology: ‘To judge an action as more or less efficient is already to have determined it to be technical and therefore an appropriate object of such a judgment. Similarly, the concept of control implied in technique is “technical” and so not a distinguishing criterion.’⁵¹ In other words, the very adoption of predictive algorithms in various social domains signals the entry of ‘efficiency and control’ as a dominant frame vis-à-vis alternative ways of engaging in the activity (e.g. research or understanding voters) and vis-à-vis alternative perspectives for evaluating the merit of the activity or outcome (e.g. ‘pure’ knowledge discovery, moral and political deliberation, human rights, equality, the rule of law or production of happiness). Feenberg’s observation also helps to explain why ‘efficiency’ should loom so large as justification for predictive and profiling technology, whilst ‘control’ or potential abuses of control feature heavily in its critique.⁵²

Given the ‘natural’ home of efficient technology in an economic conception of reality, it is perhaps not surprising that individual profiling and personalisation practices have been the most ‘successful’ in retail and marketing, that is, in terms of maximising efficiencies and control from a corporation’s perspective (although not necessarily from other perspectives, e.g., democratic deliberation). Famously, Amazon’s phenomenal rise was grounded in its recommender system that benefits from constant feedback loops for auto-correction, and can predict highly accurately the buying preferences of their customers and so helped to dramatically increase sales.⁵³ Such micro-targeting is now ubiquitously used by platforms, retailers and advertisers to order content and push products. Within the commercial realm, it is clear that the correlations themselves are valuable for the profilers, who would hold no further interest in the explanations behind the correlations.⁵⁴ If there were a correlation between a behavioural pattern on social media and excessive purchasing behaviour, it would be irrelevant to an online retailer for targeting adverts, that the common cause of the behavioural and purchasing patterns may be varying stages of depression. Being ignorant of the causes behind non-spurious correlations makes the extraction of economic value morally neutral, particularly as algorithmic value extraction generally draws on some weakness or susceptibility of the consumer. Empiricism’s apparently neutral focus on facts or corrections, not framed by explanations, provides the toolkit for de-moralising economic activity. Big data analytics realises that ambition more fully and forcefully than any previous metric or actuarial practice, as it has the strongest claim yet about the self-generated emergence of insights from data sets without human intervention.⁵⁵

This new ‘empiricism on steroids’⁵⁶ or the pursuit of data-driven blind knowledge has been equally attractive to governments that likewise chase efficiencies for their governing economies. Predictive algorithmic tools used for policing or sentencing are designed to use public recourses in a more targeted fashion and thereby deliver ‘better’ (more efficient) results, as in turn assessed by metrics. In the civil domain, smart city projects are driven by the prospect of efficiency gains by measuring traffic and footfall to enable live-management of traffic, parking, services, utilities, waste, etc. (Chapter 12) The continuous profiling of human activity aimed at facilitating efficient city life also provides ample opportunities for embedding policing activities, for example, speeding on smart motorways or unauthorised smoking,⁵⁷ whilst, at the same time, insidiously nudging populations – with a low-level awareness of an ever-present intelligent environment – into self-disciplining (control). (Chapter 3) Much like in the commercial context, the new algorithmic empiricism offers the added bonus of disassociating government from sensitive decisions. For the impersonal bureaucracies of governments – long familiar with cost-benefit analysis, evidence-based practice and impact metrics – big data algorithmic decision-making holds the promise of removing further residues of:

So the combination of ‘self-generated’ data and insights and automated decisions allows for standard normative evaluations to be shifted from political, legal or economic domains into the technical realm. Whilst the former require reasoning, judgment and communications that are necessarily contestable (Chapter 16), decisions made by technology appear to be incontestable and are, for that reason, as democratically problematic as tempting for bureaucracies. They appear incontestable not just because of the natural opacity of predictive analytics (recalling that unsupervised learning algorithms operate without a formal ‘theory’ within black boxes and are ‘adaptive’ to new data), but also because machine-generated decisions seem objective, neutral and rational, and not infested with human prejudice and fallibility. Notably, there is evidence that decisions based on experience and intuition are inferior to decisions based on simple rules implemented in weighted checklists, and the latter rival machine learning algorithms.⁵⁹ (Chapter 15) The justification for decisions by technology is thus arguably higher quality, but also, incidentally, it allows for responsibility to be shifted. Yet, ‘the choice of a technical rather than a political or moral solution to a social problem is politically and morally significant,’⁶⁰ and neither technology itself nor its decisions are apolitical or amoral.⁶¹

1.3.2 The ‘truth’ and ‘accuracy’ of Big Data Profiling

A related set of questions that flow from the empiricism of data-driven profiling and predictive practices centre around issues of truth and accuracy in light of the apparent impartiality and objectivity of statistical analysis: facts do not lie, they speak for themselves; numbers can be trusted. Implicit truth or accuracy claims bolster the authoritativeness of predictive algorithms, resulting in their apparently superior fairness. These claims can be unpacked on multiple levels as has already been comprehensively in existing scholarship.⁶² For the purposes of framing this collection, three brief reflections will suffice.

First, big data analytics, much like all statistical evaluation, is neither neutral nor objective. It inevitably embeds value judgments, and often perpetuates societal prejudices and biases. Just because unsupervised learning algorithms act ostensibly on their own does not mean their outcomes are free from biases. All algorithms are trained on data sets and replicate, in developing predictive models, the biases implicit in these sets, known inter alia as the ‘black data’ problem.⁶³ With traditional statistical modelling based on smaller and more structured data, it was possible to make conscious efforts to counteract biases against disadvantaged groups by deselecting obvious proxies of sensitive attributes (e.g. postcodes for race). Such editing is less feasible in respect of large sets of non-traditional, behavioural data, given that much apparently innocent data (e.g. cultural preferences or educational achievement) is correlated to sensitive attributes, as shown in research.⁶⁴ This means that big-data profiling based on innocent variables is still liable to be discriminatory in effect. Although the correlations are more easily identifiable, they are more difficult to eliminate as cleansing data of all proxies would leave little or no valuable data to be analysed. (Chapters 7 and 11) It also shows the depth of societal stratification; in Rieder’s words: ‘The problem, here, is not that data mining can be biased, but that, after centuries of inequality and discrimination, empirical reality is biased.’⁶⁵ Still, predictive analytics often exacerbates the problem by whitewashing those biases, whilst perpetuating inequalities under the cover of impartiality and objectivity. Of course, additional biases may also be built into the models. Furthermore, predictive algorithms are also not neutral or objective in so far as the goal-oriented perspective of the corporate or governmental profiler drives the choice of data and its interpretative framing. A government concerned with securing a maximum level of public safety is likely to define ‘recidivism’ or riskiness more expansively than one focused on rehabilitation, and will select and interpret data with that objective in mind. A corporation’s algorithmic model for hiring decisions may, at times of heightened competition, emphasise ruthlessness over competence. A predictive model for creditworthiness might prioritise profit over affordability. These perspectives provide ‘interested readings of reality’, rather than impartial descriptions.⁶⁶ They do not aspire to any ‘objective truth’ but to an instrumentalised vision of the past to make useful judgements for the future.

Second, even if not ‘true’ in any objective sense, algorithmic predictions may or may not be accurate. If the model increases profits or reduces expenditure, it is valid; it works. The accuracy of data-driven profiling and attendant predictive practices is measured against subsequent individual consumption or behavioural choices. Did the user click on the recommended news story, film or advertisement, did she default on the loan, defraud the insurance company or buy the product with the increased personalised price? Did the accused reoffend; and did the employee become the rain-maker? One objection is that mistakes can be costly especially in grave contexts, such as sentencing of offenders, creditworthiness assessments or employment decisions. This is not unique to predictive algorithms (and present in all predictions about future human actions drawn from non-universal generalisations) but arguably aggravated here as the outcomes are often not contestable (see below). Still, the more accurate the algorithmic prediction, the greater its legitimacy against alternatives. A more profound objection against predictive algorithms and their proliferation across societal domains – justified by their (assumed, for the time being) superior accuracy – relates to the underlying assumption that they do no more than actualise a future that would have occurred in any event. If they turn out to be largely accurate, interference with reality is marginal. Yet, is this really the case? Predictive practices do not simply ‘follow’ expected futures, but in the course of implementing them intervene, encroach, manipulate and make these futures. As early as 1988, Jonathan Simon commented that actuarial practices ‘cannot be dismissed as merely forms of knowing, when to be known is to be subject to significant alterations in life opportunities.’⁶⁷ By the same token, predictive algorithmic practices are not passive, but active in nudging the profilee towards the profiler’s choice of pre-destinies,⁶⁸ whilst foreclosing other opportunities. An offender with a predicted high risk of recidivism is deprived of the chance to enact a different future of law-abiding behaviour; a social media user targeted with particular political messages is deprived of alternative perspectives on the subject-matter. The effect of personalisation practices is to ‘normalise’ populations within sub-groups. Lawrence Lessig’s take on data-driven ‘normalisation’ (a Foucauldian concept associated with disciplinary regimes, e.g. schools or prisons, to institutionalise populations) is this: ‘The observing will affect the observed. The system watches what you do; it fits you into a pattern; the pattern is then fed back to you in the form of options set by the pattern; the options reinforce the patterns; the cycle begins again.’⁶⁹ Mireille Hildebrandt thus observes that although customisation ‘may seem the opposite to normalisation, in fact has a similar effect.’⁷⁰ It is not the profiler that adjusts to individual uniqueness, but rather individuals are – after an initial impetus of data – fitted into the profiler’s instrumentalised patterns. Uniqueness is supressed; everyone is like others in crucial respects. It is not the profile that is ‘accurate’, but predictive analytics cajoles the individual into becoming ‘accurate’ to fit the pattern, into becoming compliant. (Chapters 2 and 3)

That having been said, actuarial practices present a subtler and more insidious form of normalisation than that exerted by the disciplinary institutions Foucault had in mind. Simon argued for the strategic distinctiveness of actuarial practices: ‘Rather than seeking to change people (“normalize” them…), an actuarial regime seeks to manage them in place… While the disciplinary regime attempts to alter individual behavior and motivation, the actuarial regime alters the physical and social structures within which individuals behave,’⁷¹ and thereby increases the efficiency of power through making populations more docile and manageable.⁷² (Chapter 3) Still, predictive practices impact on, and shape, individual identity, including subjectivities about (group) belonging, or the lack of belonging: algorithmic or actuarial groups have ‘no experienced meaning for the members, and therefore lack the capacity to realise common goals or purposes.’⁷³ Whilst stereotyping in the analogue world generally entails reflexive communications between profiler, profilee, the stereotyped group and the community, which generate a sense of belonging and internal solidarity within the group, algorithmic groups are often artificial, seemingly arbitrary, opaque and transient aggregates.⁷⁴ (Chapter 2) This ‘makes it more difficult for group subjecthood to develop (or reproduce itself)’⁷⁵ and undermines the possibility of collective practices of resistance (see below) which in turn also reinforces their apparent accuracy.

Third, accuracy claims legitimising algorithmic profiling also have repercussions for moral agency, free will and normative domains. Predictive practices are in substance, if not in name, enabled by assumptions of character essentialism. (See Front Cover of the Book) Nicola Lacey commented: ‘new technologies in fields such as neuroscience and genetics, and computer programs that identify crime “hot spots”… offer, or perhaps threaten, yet more sophisticated mechanisms of responsibility attribution based on notions of character essentialism combined with assessments of character-based risk, just as the emerging sciences of the mind, the brain, and statistics did in the late nineteenth century.’⁷⁶ If individual preferences and behaviour can be foretold ‘accurately’, it is tempting to conclude that the human actor is after all not the autonomous decision-maker the Enlightenment ideals of human rationality and freedom envisaged. The implications of this are stark, albeit not unprecedented. In criminal justice, forms of character essentialism have long grounded predictive practices that draw deterministic inferences from the delinquent’s past to her future behaviour, in an uncomfortable tension with moral agency as the foundation of criminal responsibility and punishment. (Chapter 8). Liberal democracy and the market economy are equally dependant on the presence of autonomous citizens and consumers that can make rational choices between competing political parties and manifestos, or competing goods and services. Whilst in natural sciences, an overarching theory facilitates and constitutes the jump from ‘is’ (sun rose yesterday and today) to ‘ought’ (sun rise will rise tomorrow), within the legal and other normative domains, the notion of free choice is a moral imperative that prevents precisely such deterministic reasoning. Free choice entails the possibility of future difference, unexpectedness and new-ness. Importantly, the presumptive existence of personal autonomy (and an underlying understanding of human individuality as open-ended) allows for the possibility of normative demands in the first place. As such, moral agency is a necessary political requirement, even a necessary political fiction, not amenable to empirical disproof by algorithms or otherwise, no matter how great their apparent ‘accuracy’ is. Within a political and moral community, the open-endedness of human individuality is also essential for the possibility of collective betterment, premised on an understanding that the future is not, need not be, a replication of the past. Such betterment tends to lie in ‘unknowable and unpredictable outlier-events’ which are wholly outside ‘accurate’ statistical framings of reality.⁷⁷

1.3.3 Contesting Big Data Profiling within Spaces of Contestation

The contestability of algorithmic processes and outcomes, and their reflexive impact on existing spaces of contestation, are likely to emerge as an overall touchstone for their differential legitimacies.⁷⁸ Contestation takes different forms in different social domains, but goes some way towards enlisting reasoning and explanations of and for the profiling, and could in principle discipline the uses and nature of predictive analytics. The source of the imperative for contestability varies across social spheres. In the scientific community, contestability is an integral part of the scientific method, and can be found explicitly in Karl Popper’s concept of falsification that provides for the existence of scientific knowledge or truth, but always on a provisional basis, as long as it has not been falsified. In fact, according to Popper, the possibility of falsification is the hallmark of science and scientific knowledge as opposed to non-scientific ones, such as psychoanalysis.⁷⁹ Against this reading, insights of big data analytics are not, as such, falsifiable (and in that sense not ‘scientific’), considering that algorithms deliver probabilities based on past patterns (facts) rather than (dis)proving universal propositions (normative claims). Even where the predicted probabilities are not borne out against new data sets, for example, Google’s flu predictions,⁸⁰ and thus false, in the absence of a universal and grounded claim there is no theory that could be ‘falsified’ in the Popperian sense. This does not mean that these insights cannot be informative or useful (including in the development of theories), but rather that their standing as scientific knowledge or truths needs to be treated with caution. This holds significance beyond science in that data-driven ‘scientific’ knowledge and its methods routinely inform, for example, legal and political reasoning, propped up with the claim of ‘scientific authority’.

In the legal sphere, the imperative of the contestability of governmental decisions flows from the State’s unique power over individual life and liberty (classically expressed as the state’s monopoly on the legitimate use of violence) and oversight in administering public life, and is designed to stem abuses of that power. The contestability (or reviewability) of governmental decisions is integral to the rule of law and actualised in a host of constitutional conventions, rights and guarantees that create spaces for individual or collective contestation. The right to judicial review of administrative decisions, the right of defence,⁸¹ as well as data subject access rights are prime examples of individual rights to contest governmental decisions that affect the individual, underwritten by the understanding that such contestation makes for better decisions with greater legitimacy. Public decision-making based on black-box technology is thus, by its very nature, an anomaly. Its problematic is put in sharp relief within criminal justice, where data-driven profiling of delinquents falls uncomfortably between the countervailing forces of the efficient pursuit of public safety (economic perspective), on the one hand, and the presumption of innocence and moral agency as a basis for criminal responsibility, on the other hand. (Chapter 8) In the US case of State of Wisconsin v. Loomis (2016)⁸² (Chapter 11 and 15) the Wisconsin Supreme Court upheld the denial of parole and consequential six-year prison term (for a drive-by shooting) which was based, in parts, on the ‘Correctional Offender Management Profiling for Alternative Sanctions’ black-box algorithm, otherwise known as COMPAS, that had classified Loomis as high risk to the community and of recidivism. For the court, it was not fatal to the defendant’s constitutional due-process right, that neither the sentencing court nor Loomis had any insight into the workings of COMPAS to review the risk assessment for ‘accuracy and scientific validity’ – in light of the facts that the tool was only used to corroborate the judge’s opinion and that Loomis had known the personal information that fed into the algorithm (which arguably also gave him some derivative insight about the algorithm’s variables.)⁸³ The court’s attempt to trivialise the tool suggests a keen awareness of its poor fit with the demands of a fair trial, which at the very minimum would require a reasoned decision (even ‘black-box’ jury decisions are foregrounded by testing of opposite narratives in the adversarial trial). Still, it approved the legitimacy of COMPAS, and thus, in the end, efficiency and blind knowledge trumped due process and articulated reasoning. Collective contestation to these algorithmic tools is unlikely by a public that is increasingly habituated to criminal justice as instrumentalist, preventative risk-management administration towards ‘law and order’ as opposed to the traditional retrospective, individualised model focused on moral accountability and just punishment, enacted through a trial that serves as a communicative process between defendant, state and community.⁸⁴

In the political domain, contestability lies at the heart of democracy and democratic accountability, most prominently manifested in representative government, free elections and free speech protection. John Stuart Mill in reflecting on representative government insisted on the ‘function of Antagonism’ which, if unfulfilled, would condemn a ‘government to infallible degeneracy and decay’.⁸⁵ Antagonism serves progress: ‘No community has ever long continued progressive, but while a conflict was going on between the strongest power in the community and some rival power… When the victory on either side was so complete as to put an end to the strife, and no other conflict took its place, first stagnation followed, and then decay.’⁸⁶ Similarly and overlapping, the marketplace of ideas rationale for free speech protection is animated by the notion that the best or most truthful ideas will emerge victoriously in the free and robust competition (or contestation) of ideas and opinions.⁸⁷ Although the reverse may well be the case (i.e. the marketplace of ideas produces rather than reveals ‘truths’), in approach it resonates with the concept of falsification in the scientific community. And yet again, personalisation or data-driven political micro-targeting provides for an uneasy fit with the public sphere and political deliberation, and at best disrupts and at worst diminishes existing practices of contestation. First and most importantly, unlike in the legal context, political micro-targeting taps into user preferences, and so almost inevitably incapacitates resistance to itself. Individuals have no or little incentive to question the accuracy or appropriateness of the profiling or personalisation practice, that is continuously adjusted in light of feedback loops. Second, although the overall effect of micro-targeting on the public sphere as a shared space for political deliberation remains ambivalent (Chapter 13), the problematisation of filter bubbles and echo chambers arises from a concern that micro-targeting profoundly undermines the practice of testing and contesting individual political standpoints through exposure to alternative narratives – with strong self-reinforcing dynamics.⁸⁸ (Chapter 17) The lack of exposure to such alternative narratives due to personalised context arguably generates more insular, polarised and extremist political perspectives, making them even less amenable to challenge. Thus third, increased political homogeneity and polarisation within echo chambers translates, at the collective, heterogeneous level, into a weakened willingness and capacity to communicate across political sub-communities, and thus tends to diminish social cohesion and conflict resolution, both of which traditional mass media fostered as a prerequisite for effective democratic governance.⁸⁹ Thus Cass Sunstein has argued that citizens in a well-functioning democracy ought to be exposed to chance encounters and to shared experiences which act as social glue, stressing affinity over difference, necessary for solving social problems.⁹⁰ Both demands are diametrically opposed to targeted personalised experiences, and up against a powerful alliance of commercial, political and individual interests in the practice of personalisation.

In the commercial sphere, contestability is implicit in the idea of the market which is, in its very conception, a space of contestation between providers, enacted by consumers through consumption choices freely and autonomously made and who thereby allocate resources to the ‘best’ providers. So, much like in the scientific, governmental and political arenas, market contestation also performs a type of ‘quality control’ with consumers as final arbiters. Whilst data-driven profiling and personalisation are apparently wholly in tune with such market contestation, in fact they undermine the market in significant ways – and again with the blessing of users. First, as Nick O’Donovan explores in Chapter 4, data-driven personalisation exerts strong secondary network effects that have led to further online concentration with a diminishing choice for consumers, especially amongst non-profiling providers. Second, consumers exercise their autonomy in favour of data-driven personalisation in an environment that is deeply hostile to alternative (and contesting) choices and nudges them towards personalisation, often in non-compliance with the General Data Protection Regulation⁹¹ (Chapter 5) but not necessarily so. Against the immediate personal benefits gained (e.g. access to the site or free use of a service), the distant and often collective harms caused by mass algorithmic profiling, for example, filter bubbles, user-pay models, surveillance, manipulation, pale into insignificance. Facebook’s ‘emotional contagions study’ of 2014 involved 689,003 users and showed how easily Facebook could manipulate the emotions of its subscribers.⁹² Although this study was ‘corporate research’ in response to popular narratives,⁹³ in fact Facebook’s core business lies in continually adjusting its algorithm to increase user time spent on the platform (‘stickiness’) and decrease buying resistance. The research caused outrage at the time, but did not lead to a mass exodus of its subscribers, as the collective and distant harms failed to trump the immediate personal gains. In that sense algorithmic profiling presents a classic tragedy of the commons, or collective action, problem. Even on a personal level, the collection of personal micro-data points drawn from online behaviour that are generally by themselves neither sensitive nor significant, does not easily translate – in the mind of users – into highly sophisticated individual profiles that may be used across a range of purposes from insurance to pricing to credit risks, and become virtual alter-egos which are hard, if not impossible, to escape. For privileged users, these virtual alter egos may give them significant leeway. Once in place, personalisation, which is designed to break down consumer resistance (i.e. create zero buying resistance⁹⁴) as continuously perfected through feedback loops, becomes fully self-perpetuating, and thus pre-empts contestation. (See Chapter 17) The sheer profitability of profiling also makes corporate ‘AI ethics’ an unlikely candidate for (self)disciplining these practices. (Chapter 14) Third, a ‘personalised’ market is a highly fragmented market and in fact no marketplace at all; it fails as a communication network within which consumers can get (price) signals that would enable rational decision making and collective action that are defining features of a market. (Chapter 6) Typically, personalised pricing (or price discrimination) disempowers consumers as market participants, given the absence of stable reference points (i.e. the standard price offered to all consumers by a provider) against which buying choices may be made.⁹⁵ (Chapter 10)

Against these dynamics, any legal attempt to empower consumers and citizens individually, as the General Data Protection Regulation does as a primary strategy, is immediately met by the resistance-breaking features underlying personalisation. In the commercial context, individual contestation of profiles, similar to the challenge in Loomis, is likely only in respect of more significant transactions with an obviously negative outcome, for example, a rejected credit, employment or insurance application. In these very limited contexts (considering the overall ubiquity of personalisation), Article 22 of the GDPR might be useful as it gives profiled subjects the right to human oversight over ‘significant’ automated decisions,⁹⁶ including ‘meaningful information about the logic involved [in the algorithm], as well as the significance and the envisaged consequences [of the profile] for the data subject.’⁹⁷ This entitlement does not extend to an explanation for the actual decision, or normative justification, but would probably entail information about variables governing the predictive algorithms and their relevance to the decision, thus amounting to something vaguely approaching causation, or normative justification.

In summary, the use of predictive analytics has, in different social domains, encountered and, more or less successfully, disabled resistance to itself and thereby also chipped away at the broader spaces of contestation, which traditionally have served core public values, such as promoting knowledge, accountable government and functioning markets. Yet, the possibility of contestation also enfranchises the individual by allowing her to become an active participant or critical listener in processes and decisions that have an impact on her life. The inscrutability of profiling algorithms in criminal justice is blatantly problematic as its attendant decisions have an immediate and significant impact on individuals. The systematic exposure to mundane algorithmic personalised outcomes shrouded in darkness is perhaps even more challenging precisely because their all-embracing effects are more insidious, diffuse and evasive. Daniel Solove has argued for the inscrutability of personal data usage as a core harm that privacy is, or should be, concerned about. Using Kafka’s The Trial, he has shown how – beyond the harms caused by a surveillance environment when personal data is continuously collected – the inscrutability of the storage of data, its analysis and use creates a ‘suffocating powerlessness and vulnerability’ that repositions the individual vis-à-vis the state: ‘a bureaucracy with inscrutable purposes… uses people’s information to make important decisions about them, yet denies the people the ability to participate in how their information is used.’⁹⁸ His commentary was addressed to government but arguably extends to private actors whose inscrutable data practices are in their exertion of impersonal domination equally harmful,⁹⁹ particularly where those actors control the infrastructure of collective social, economic and political life. (Chapter 4) In the private sphere, the collection of non-traditional non-volunteered data, its aggregation and open-ended secondary uses (e.g. ‘All data is credit data’ Chapter 7) magnifies inscrutability and, by implication, the power asymmetries between profiler and profilee. It is this profound asymmetry that undermines the legitimacy of stereotyping which, as argued above, is prima facie necessary and useful. This asymmetry also means that whilst stereotyped individuals are in the analogue world, at least in some circumstances, able to participate in the creation of their own stereotype or contest it subsequently, the inscrutability of algorithmic decisions largely prevents active participation in, or contestation of, stereotyping. The individual is kept out of the loop in the processes and decisions that affect his life; or, in Rieder’s words, kept ‘in the conditions of paranoid meritocracy, constantly wondering whether their practices and preferences signal their adherence to “economic morality.”’¹⁰⁰ Corporate actors – and derivatively governmental actors, considering the amount of data sharing – know us intimately and much more closely than we know them. Such intimate knowledge is neither harmless nor prima facie legitimate. It amplifies corporate and governmental power over consumers and citizens in a reversal of existing understandings of effective democracy and markets, which, even absent abuses, reshapes social relationships. The emergent judicial recognition of the non-domination principle, as advocated by academics as a new privacy standard, is a small tentative step towards acknowledging these paradigmatic shifts in power differentials and redressing them.¹⁰¹ However, much wider systematic thinking about profiling and personalisation practices will be needed to retain and regain the foundations of a humanist digital society, grounded in individual and institutional responsibility, reason and articulated reasoning; individuality, community and solidarity; and communication as a means for expressing conflicts and work towards their resolution.