1.1 Introduction

Facial recognition technology (FRT) is fast becoming a defining technology of our times. The prospect of widespread automated facial recognition is currently provoking a range of polarised responses – from fears over the rise of authoritarian control through to enthusiasm over the individual conveniences that might arise from being instantly recognised by machines. In this sense, FRT is a much talked about, but poorly understood, topic of contemporary social, political, and legal importance. As such, we need to think carefully about exactly what ‘facial recognition’ is, what facial recognition does, and, most importantly, what we as a society want facial recognition to become.

Before this chapter progresses further into the claims and controversies surrounding FRT, a few basic definitions and distinctions are required. While various forms of technology fall under the broad aegis of ‘facial recognition’, we are essentially talking about technology that can detect and extract a human face from a digital image and then match this face against a database of pre-identified faces. Beyond this, it is useful to distinguish three distinct forms of facial technologies that are currently being developed and implemented. First, and most widespread to date, are relatively constrained forms of FRT that work to match a human face extracted from a digital image against one pre-identified face. This ‘one-to-one’ matching will be familiar to the many smartphone users who have opted for the ‘Face-ID’ feature. The goal of one-to-one matching (sometimes termed ‘verification’ or ‘authentication’) is to verify that someone is who they purport to be. A smartphone, for example, is programmed to ascertain if a face in front of the camera belongs to its registered user (or not) and then unlock itself accordingly (or not).

In this manner, one-to-one facial recognition makes no further judgements beyond these repeated one-off acts of attempted identification. Crucially, the software is not capable of identifying who else might be attempting to unlock the device. In contrast, a second ‘one-to-many’ form of FRT is capable of picking a face out of a crowd and matching it to an identity by comparing the captured face to a database containing thousands (or even millions) of faces. This form of isolating any face from a crowd and making an identification has more scope for mass surveillance and tracking. Alongside these forms of facial recognition technologies designed to either verify or ascertain who someone is, is a third form of ‘facial processing’ technologies, ones that seek to infer what someone is like, or even how someone is feeling. This is technology that extracts faces from digital images and looks for matches against databases of facial expressions and specific characteristics associated with gender, race, and age, or in some cases even emotional state, personality type, and behavioural intention. This form of facial scanning has prompted much interest of late, leading to all manner of applications. During the height of the COVID-19 pandemic, for example, we saw the development of facial processing technology designed to recognise high body temperature and thus infer symptoms of virality through the medium of the face.

All told, considerable time, investment, and effort is now being directed towards these different areas of facial research and development. For computer scientists and software developers working in the fields of computer vision and pattern matching, developing a system that can scan and map the contours and landmarks of a human face is seen as a significant computational challenge. From this technical perspective, facial recognition is conceived as a complex exercise in object recognition, with the face just one of many different real-life objects that computer systems are being trained to identify (such as stop signs on freeways and boxes in warehouses). However, from a broader point of view, the capacity to remotely identify faces en masse is obviously of considerable social significance. For example, from a personal standpoint, most people would consider the process of being seen and scrutinised by another to be a deeply intimate act. Similarly, the promise of knowing who anyone is at any time has an understandable appeal to a large number of social actors and authorities for a range of different reasons. A society where one is always recognised might be seen as a convenience by some, but as a threat by others. While some people might welcome the end of obscurity, others might rightfully bemoan the death of privacy. In all these ways, then, the social, cultural, and political questions that surround FRT should be seen as even more complex and contestable than the algorithms, geometric models, and image enhancement techniques that drive them.

1.2 The Increasing Capabilities and Controversies of Facial Recognition Technology

Facial recognition has come a long way since the initial breakthroughs made by Woody Bledsoe’s Panoramic Research lab in Palo Alto nearly sixty years ago. By 1967 Bledsoe’s team had already developed advanced pointillistic methods that could assign scores to faces and make matches with a mugshot database of what was described as 400 ‘adult male Caucasians’. Despite steady subsequent technical advances throughout the 1970s and onwards, FRT became practicable on a genuinely large scale only during the 2010s, with official testing by the US National Institute of Standards and Technology, reporting accuracy rates for mass-installed systems in excess of 99 per cent by 2018.

As with all forms of AI and automated decision-making, FRT development over the past ten years has benefited from general advances in computational processing power, especially deep learning techniques, and the data storage capabilities required to develop and train large-scale machine learning models. However, more specifically, the forms of FRT that we are now seeing in the 2020s have also benefited from advances in cheap and powerful camera hardware throughout the 2010s (with high-definition cameras installed in public places, objects, and personal devices), alongside the collation of massive sets of pre-labelled photographed faces harvested from publicly accessible social media accounts.

Thus, while the technical ‘proof of concept’ for FRT has been long established, the society-wide acceleration of this technology during the 2020s has been spurred primarily by recent ‘visual turns’ in consumer digital electronics and popular culture towards video and photo content creation, and the rising popularity of self-documenting everyday life. But, equally, it has been stimulated by the desire of organisations to find automated solutions for managing the problem of distancing and anonymity that networked digital technologies have effected, as well as by vendors who market the virtues of the technology as a means to improve security, convenience, and efficiency, while eliminating the perceived fallibilities of human-mediated recognition systems. Thus, a combination of cultural factors, alongside exceptional societal events such as the COVID-19 pandemic, and the wider political economic will to propose and embrace techno-solutions for redressing social issues and to increasingly automate access to various spaces and services, has fashioned receptive conditions for an expansion in FRT and its concurrent normalisation.

And yet the recent rise to prominence of FRT has also led to a fast-growing and forceful counter-commentary around the possible social harms of this technology being further developed and implemented. Growing numbers of critics contend that this is technology that is profoundly discriminatory and biased, and is something that inevitably will be used to reinforce power asymmetries and leverage unfair ends. Such push-back is grounded in a litany of controversies and misuses of FRT over the past few years. For example, the United States has seen regular instances of FRT-driven racialised discrimination by law enforcement and security agencies – not least repeated instances of US police using facial recognition to initiate unwarranted arrests, false imprisonment, and other miscarriages of justice towards minoritised social groups. Similar concerns have been raised over FRT eroding civil liberties and human rights – constituting what Knutson describes as conditions of ‘suspicionless surveillance’, with state authorities emboldened to embark on delimited ‘fishing expeditions’ for all kinds of information about individuals.Footnote ¹

Elsewhere, FRT has proven a key element of Chinese authorities’ suppression of Muslim Uyghur populations, as well as in illegal targeting of political protesters by authorities in Myanmar and Russia. Moreover, for others, FRT represents a further stage in the body’s progressive colonisation by capital, as the technology has enabled the capture of increasingly detailed information about individuals’ activities as they move through public and shared spaces. This data can be used to sort and manipulate consumers according to commercial imperatives, tailoring the provision of products and services so that consumption behaviours are maximised. All told, many commentators contend that there have already been sufficient examples of egregious, discriminatory, and harmful uses of FRT in everyday contexts to warrant the cessation of its future development.

Indeed, as far as many critics are concerned, there is already ample justification for the outright banning of facial recognition technologies. According to Hartzog and Selinger, ‘the future of human flourishing depends on facial recognition technology being banned before the systems become too entrenched in our lives’.Footnote ² Similarly, Luke Stark’s thesis that ‘facial recognition is the plutonium of AI’ advocates the shutdown of FRT applications in all but the most controlled circumstances.Footnote ³ In Stark’s view, the potential harms of using FRT for any purpose in public settings are sufficient reason to render its use too risky – akin to using a nuclear weapon to demolish a building. Such calls for the total suppression of FRTs have been growing in prominence. As noted scholar-activist Albert Fox Cahn recently put it: ‘Facial recognition is biased, broken, and antithetical to democracy. … Banning facial recognition won’t just protect civil rights: it’s a matter of life and death.’Footnote ⁴

1.3 Justifications for Facial Recognition as Part of Everyday Life

While some readers might well feel sympathetic to such arguments, there are also many practical reasons to raise doubts that such bans could ever be practically feasible, even with sufficient political and public support. Proponents of FRT counter that it is not possible to simply ‘dis-invent’ this technology. They argue that FRTs are now deeply woven throughout the fabric of our digital ecosystems and that commercial imperatives for the information technology and surveillance industries to continue developing FRT products remain too lucrative to give up. Indeed, the technology is already becoming a standard option for closed-circuit television (CCTV) equipment and is regularly used by police even in jurisdictions without any formal rules governing its deployment. The industry-led and practitioner-backed promissory discourse that propagates the various virtues of FRT is already so deeply entrenched in organisational thinking and practice that it would seem highly unlikely for systems and applications to be withdrawn in the various social contexts where they now operate. In this sense, we perhaps need to look beyond polarised discussions over the fundamental need (or not) for the existence of such technology and instead pay closer attention to the everyday implications of FRT as it gets increasingly rolled out across various domains of everyday life to transform how people, things, and processes are governed.

Proponents of FRT – especially those with a commercial interest in encouraging public and political acceptance of the technology – will often point to a number of compelling ‘use cases’ that even the staunchest opponents of FRT will find difficult to refute. One common example is the use of FRT to reunite kidnapped, lost, or otherwise missing children with their families. The controversial face recognition company Clearview AI, which has scraped billions of face images from online sources, has highlighted the use of the app to identify victims and perpetrators of child sexual abuse.Footnote ⁵ Other pro-social use cases include the use of face recognition to identify people whose documentation has been lost or destroyed during natural disasters, as well as the development of specialised facial recognition software to identify the victims of war and disaster, providing some sense of closure to loved ones and avoiding the time and cost of alternative methods (such as DNA analysis or dental records). Even critics such as Luke Stark concede that FRT might have merit as a specialised accessibility tool for visually impaired people. Indeed, given the fundamental human need to know who other people are, it is always possible to think of potential applications of this technology that seemingly make intuitive or empathetic sense.

Of course, were FRT to remain restricted to such exceptional ‘potential limited use cases’,Footnote ⁶ then most people would rarely – if ever – come into contact with the technology, and therefore the concerns raised earlier over society-wide discrimination, biases, and harms would be of little significance. Nevertheless, we already live in times where a much wider range of actual applications of FRT have proven to be largely ignored or presumed uncontentious by a majority of the general public. These ‘everyday’ uses of FRT, we would argue, already mark the normalisation of a technology that is elsewhere perceived as controversial when in the hands of police, security services, the military, and other authorities.

These ‘pro-social’ uses span a diverse range of everyday contexts and settings. Perhaps one of the most established installations of facial recognition can be found at airports. FRT is a key component of ‘paperless boarding’ procedures, allowing airline travellers to use this one-to-one biometric matching capacity between their e-passport photo and their physical face to check in, register their bag-drop, and then proceed through the departure and arrival gates. A major rationale for this automated infrastructure is that it makes travel processes more seamless, lessening queues and cutting costs, while also enhancing the recognition capacities (and thus organisational efficiency) of the airport authority. For instance, various studies on recognition have illustrated that the technology outperforms human recognisers, in this case, the security officials and airline clerks stationed at passport control or check-in counters. Another public setting with a long history of FRT is the casino industry. Most large casinos now operate some form of FRT. For example, the technology is strategically used to enforce blocklists of banned patrons, to enforce ‘responsible gaming’ by identifying under-age and ‘impaired’ players, and to support the exclusion of self-identified problem gamblers, as well as for recognising VIP guests and other high spending customers at the door who can then be quickly escorted to private areas and given preferential treatment.

Various forms of facial recognition and facial processing technology are also being deployed in retail settings. The most obvious application is to augment retail stores’ use of CCTV to identify known shoplifters or troublemakers before they gain entry to the premises. Yet, as is the case with casinos, a range of other retail uses have also come to the fore – such as using FRT to recognise repeat customers; target screen-based advertising to particular demographics; collect information on how different customers use retail space and engage with particular arrangements of goods; and gauge satisfaction levels by monitoring the facial expressions of shoppers waiting in checkout lines or engaging with particular advertisements. Another major retail development is the use of ‘facial authentication’ technology to facilitate payment for goods – replacing the need to present a card and then tap in a four-digit PIN with so-called ‘Pay By Face’ systems, and thus lessening the ‘friction’ that stems from a customer forgetting or wrongly entering their code on the EFTPOS terminal, while also reducing opportunities for fraudulent activity to occur.

Alongside these cases, there are other instances of FRT being used in the realms of work, education, and healthcare. For example, the growth of FRT in schools, universities, and other educational settings encompasses a growing range of activities, including students using ‘face ID’ to pay for canteen meals and to check out library books; the detection of unauthorised campus incursions; the automated proctoring of online exams; and even gauging students’ emotions, moods, and levels of concentration as they engage with content from the curriculum and different modes of teaching delivery. Similarly, FRT is finding a place in various work settings – often for ‘facial access control’ into buildings and for governing the floors and areas that employees and contractors can (and cannot) enter, as well as for registering who is in the building and where people are in the case of emergency. Other facial recognition applications also allow factory and construction employees to clock in for work via contactless ‘facial time attendance’ applications, and – in a more disciplinary sense – can be utilised to monitor the productivity and activities of office staff who are working from home. Similarly, in healthcare contexts, FRT is being used for multiple purposes, from more efficient recognition of patients’ identities as they enter clinical facilities so that the need for documentation is reduced (a handy administrative feature in the case of a medical emergency or to support those suffering from mental conditions such as dementia or psychosis), to improving knowledge on wait times and thus better targeting resources and services. FRT is also used to enhance facility security by controlling access to clinical facilities and identifying visitors who have previously caused trouble, as well as for patient monitoring and diagnosis, even to the point of purportedly being able to ‘detect pain, monitor patients’ health status, or even identify symptoms of some illnesses’.Footnote ⁷

These workplace technologies are complemented by the rise of domestic forms of FRT – with various products now being sold to homeowners and landlords. One growing market is home security, with various manufacturers producing low-cost security systems with facial recognition capabilities. For example, homeowners are now using Wi-Fi-enabled, high-definition camera systems that can send ‘familiar face alerts’ when a person arrives on their doorstep. Anyone with an inclination towards low-cost total surveillance can run up to a dozen separate facial recognition cameras inside a house and its surrounding outside spaces. Facial recognition capabilities are also being enrolled into other ‘smart living’ products, such as the rise of in-car facial processing. Here, some high-end models are beginning to feature in-car cameras and facial analysis technology to infer driver fatigue and trigger ‘drowsiness alerts’. Some systems also promise to recognise the faces of different drivers and adjust seating, mirror, lighting, and in-car temperatures to fit the personal preferences of whoever is sitting behind the wheel.

1.4 The Limits of Facial Recognition ‘for Good’: Emerging Concerns

Each of these ‘everyday’ forms of FRT might appear innocuous enough, but taken as a whole, they mark a societal turn towards facial technologies underpinned by a growing ecosystem of FRT, perhaps even biometric consciousness, that is becoming woven into the infrastructural fabric of our urban environments, our social relations, and our everyday lives. Most importantly, it could be argued that these growing everyday uses of FRT distract from the various latent and more overt harms that many people consider this technology to perpetuate, specifically in a landscape where the technology and its diverse applications remain either under-regulated or not regulated at all. Thus, in contrast to the seemingly steady acceptance and practical take-up of FRT throughout our public spaces, public institutions, and private lives, there is a pressing need to pay renewed attention to the everyday implications of these technologies in situ, especially to temper some of the political rhetoric and industry hyperbole being pushed by various proponents of these systems.

1.5 Future Directions and Concerns

The development of FRT to date clearly raises a host of important and challenging issues for regulators and legislators to address. Before we consider the prospects for what this handbook describes as ‘possible future directions in regulating governments’ use of FRT at national, regional and international levels’, it is also worth considering the broader logics and emerging forms of FRT and facial processing that have been put into train by the development of FRT to date, and the further issues, concerns, and imperatives that this raises.

One obvious emerging application of concern is the growing use of facial processing to attempt to discern internal mental states. Thus, for example, face recognition has been used by job screeners to evaluate the stress levels and even the veracity of interviewees. While these inferences are without scientific basis, this does not necessarily stop them from being put to use in ways that affect people’s life chances. This raises the human rights issue of protecting the so-called forum internum – that is, control over the disclosure of one’s thoughts, attitudes, and beliefs. Inferential technologies seek to bypass the ability of individuals to control the disclosure of the innermost sentiments and thoughts by reading these directly from visible external signs. We are familiar with the attempt to ‘read’ sentiment through non-verbal cues during the course of interpersonal interactions, but automated systems provide these hunches with the patina of (false) scientific accuracy and machinic neutrality in potentially dangerous and misleading ways. The inferential use of this type of automated inference for any type of decision making that affects people’s life chances should be strictly limited.

Second is the prospect of the remote, continuous, passive collection of facial biometric data at scale, and across all public, semi-public and private spaces. At stake is not simply the diminishment of individual privacy, but also the space for democratic participation and deliberation. Unleashed on the world, such technology has a very high potential for a host of new forms of social sorting and stalking. Marketers would like to be able to identify individuals in order to target and manipulate them more effectively, and to implement customised offers and pricing. Employers, health insurers, and security officials would be interested in using it for the purposes of background checking and forensic investigations. With such technology in hand, a range of entities could create their own proprietary databases of big spenders, poor tippers, potential troublemakers, and a proliferating array of more and less desirable customers, patients, employees, tenants, clients, students, and more.

Indeed, the continued integration of facial processing capabilities into urban CCTV systems with automated facial recognition also marks a fundamental shift in how surveillance in public space operates. Standard ‘dumb’ forms of CCTV see the same thing and record what people already see in public and shared space – but do not add extra information. The ability to add face detection and recognition enables new strategies of surveillance and control that are familiar from the online world. For example, with facial recognition, the target of CCTV surveillance can shift from particular individuals or groups to overall patterns. Cameras that track all the individuals within their reach enable so-called pattern of life analysis, looking for different patterns of activity that facilitate social sorting and predictive analytics. For example, the system might learn that particular patterns of movement or interaction with others correlate with the likelihood of an individual making a purchase, getting into a fight, or committing a crime. This type of analysis does not necessarily require identifying individuals, merely recognising and tracking them over time and across space.

Finally, then, there are concerns over how FRT is part of an increasing turn towards surveillance as a replacement of trust. As the philosopher Byung-Chul Han puts it, ‘Whenever information is very easy to obtain, as is the case today, the social system switches from trust to control.’Footnote ²⁰ No amount of surveillance can ever fully replace trust, but it can undermine it, leading to an unfillable gap that serves as an alibi for ever more comprehensive and ubiquitous data collection. Han describes a resulting imperative to collect data about everything, all the time, in terms of the rise of ‘the society of transparency’. It is not hard to trace the symptoms of this society across the realms of social practice: the collection of increasingly comprehensive data in the workplace, the home, the marketing realm, and public spaces. As sensors and network connections along with data storage and processing become cheaper and more powerful, more data can be collected with respect to everything and anything. Face recognition makes it possible to link data collected about our activities in shared and public spaces to our specific identities – and thus to link it with all the other data troves that have been accumulating both online and offline. All told, the concern here is that the technology addresses broader tendencies towards the automated forms of control that characterise social acceleration and the crisis of social trust associated with the changing information environment.Footnote ²¹

1.6 The Need for (and Prospects of) Regulation and Oversight

With all these issues in mind, it seems reasonable to conclude that FRT requires to be subject to heightened scrutiny and accountability. For many commentators, this scrutiny should involve increased regulatory control, government oversight, and increased public understanding of the issues arising from what is set to be a defining technology of the next decade and beyond. That said, as this chapter’s brief overview of the sociotechnical complexity of the technology suggests, any efforts to regulate and hold FRT to account will not be easy. We therefore conclude by briefly considering a number of important concerns regarding the philosophical and regulatory implications of FRT, issues that will be developed and refined further in the remainder of the book.

As with most discussions of technology and society, many of the main concerns over FRT relate to issues of power. Of course, it is possible to imagine uses of FRT that redress existing power imbalances, and provide otherwise marginalised and disempowered populations with a means of resisting authoritarian control and to hold power accountable. For example, during the 2020 Black Lives Matter protests, activists in Portland developed FRT to allow street protesters to identify and expose violent police officers. Nevertheless, while it can be used for sousveillance, the mainstream roll-out of FRT across society looks set to deepen asymmetry of power in favour of institutions. Indeed, there is an inherent asymmetry in both power and knowledge associated with these processes of datafication. Only those with access to the databases and the processing power can collect, store, and put this information to use. In practice, therefore, face recognition is likely to become one more tool used primarily by well-resourced organisations and agencies that can afford the necessary processing power and monitoring infrastructure.

As such, any efforts to regulate FRT need to focus on issues of civil rights and democracy, the potential misuse of institutional power, and resulting harms to marginalised and minoritised groups. In this sense, one of the profound shifts envisioned by the widespread use of automated facial recognition is the loss of the ability to opt-out. When public spaces we need to access for the conduct of our daily lives – such as the shops where we get our food, or the sidewalks and streets we travel – become equipped with face recognition, we do not have a meaningful choice of whether to consent to the use of the technology. In many cases we may have no idea that the technology is in place, since it can operate passively at a distance. The prevalence of existing CCTV networks makes it possible to implement facial recognition in many spaces without significantly transforming the visible physical infrastructure.

Following this logic, then, it is likely that automated face recognition in the near future will become a standard feature of existing CCTV surveillance systems. Regulatory regimes that rely on public notification are ineffective if they do not offer genuine opt-out provisions – and such provisions are all but impossible in shared and public spaces that people need to access. When face recognition is installed in public parks or squares – or in commercial locations such as shopping centres, the only choice will be to submit to their monitoring gaze or avoid those spaces. Under such conditions, their decision to use those spaces cannot be construed as a meaningful form of consent. In many cities CCTV has become so ubiquitous that its use passes without public notification. Without specific restrictions on its use, facial recognition is likely to follow the same trajectory. Seen in this light, there are many reasons why regulation and other attempts to hold FRT to account faces an uphill battle (if not the prospect of being thwarted altogether). This is not to say that regulation is not possible. For example, more than two dozen municipalities in the United States banned government use of one-to-many face recognition during the first few years of the 2020s, and the European Union continues to moot strict regulation of its use in public spaces. Nevertheless, the use of the technology by private entities for security and marketing and by government agencies for policing continues apace.

All our future discussions of possible FRT regulation and legislation therefore need to remain mindful of the strong factors driving continued demand for FRT and its uptake. For example, the promise of convenience and security combined with increasing accuracy and lower cost all serve as strong drivers for the uptake of the technology. There are also sustained commercial imperatives to continue this technology – not least the emergence of a $5 billion FRT industry that is estimated to grow to $50 billion by 2030. At the same time, we are living in a world where there are a number of powerful authoritarian drivers to continue the uptake of FRT regardless of pushback from civil society. As discussed earlier in this chapter, universal automated access comes at the expense of perpetual tracking and identification. In addition to the pathologies of bias and the danger of data breaches and hacking, there is also the threat of authoritarian levels of control. Widespread facial recognition creates the prospect of a tool that could, in the wrong hands, be used to stifle political opposition and chill speech and legitimate forms of protest. It can also be used to extract detailed information about people’s private lives, further shifting control over personal information into the hands of those who own and control the monitoring infrastructure.

Regardless of such impediments and adversaries, many people contend that the time to develop clear regulations in keeping with commitments to democracy and human rights is now. Building support for such regulation will require concerted public education programmes that focus on the capabilities and potential harms of the technology. At the moment, its potential uses and capabilities are not understood widely and are often framed in terms of personal privacy invasion rather than its potentially deleterious effects on democracy and civic life. Developing appropriate regulation will also require negotiating the tension between the commercial pressures of the data-driven surveillance economy, the security imperatives of law enforcement, and civic values of freedom of expression, movement, and personal autonomy. The outcome we need to avoid is the one towards which we seem to be headed: a situation in which the widespread deployment of the technology takes place in a regulatory vacuum without public scrutiny or accountability.

The legal challenge of FRT lies in the fact that the consent scheme is not the best approach to protect individual rights as discussed earlier. And in some contexts, preventing its uses based on individual rights’ argument may not be in the interest of the general public. In this complex situation, we should not be forced into making a choice between protecting the individuals and protecting the society at large (an argument that Chinese lawmakers are now working on through the introduction of a revised data protection law effective in 2021). Instead, we need to develop laws that will not obscure self-governance (individual rights protection) in relation to the promotion of the application of FRT as public interests. The boundaries of legal application of FRT need to be established. In it, the liability of those who are collecting, collating, and analysing facial data should be a key consideration. For example, if the use of FRT is permitted, the re-use of such information without individual authorisation should be prohibited. The emphasis should also be about how to prevent harms resulting from public interest exceptions.

1.7 Conclusions

These are just a few opening observations and points in what needs to be a prolonged society-wide discussion over the next decade and beyond. While it is unlikely that a consensus will ever be reached, it is possible to develop a clear sense of the boundaries that we want to see established around this fast-changing set of technologies. That said, such is the pace of change within biometrics and AI, it might well be that facial recognition technology is only a passing phase – researchers and developers are already getting enthused over the potential scanning of various other bodily features as a route to individual identification and inference. Yet many of the logics highlighted in this chapter apply to whatever other part of the human body this technology’s gaze is next trained on – be it gait, voice, heartbeat, or other.

Of course, many of the issues raised in this chapter are not unique to FRT per se – as McQuillan reminds us, every instance of ‘socially applied AI has a tendency to punch down: that is, the collateral damage that comes from its statistical fragility ends up hurting the less privileged’.Footnote ²² Nevertheless, it is worth spending time unpacking what is peculiar about the computational processing of one’s face as the focal point for this punching down and cascading harm. This chapter has therefore presented a selection of issues that we identify from the perspective of sociology as well as culture, media, and surveillance studies. There are many other disciplines also scrutinising these issues from across the humanities and social sciences – all of which are worth engaging with as bringing a valuable context to legal discussions of FRT. Yet we hope that the law and legal disciplines can bring an important and distinctive set of insights in taking these issues and conversations forward. Legal discussions of technology bring a valuable pragmatism to the otherwise ambiguous social science portrayals of problematic technologies such as FRT – striving to develop ‘a legitimate and pragmatic agenda for channelling technology in the public interest’.Footnote ²³ We look forward to these conversations continuing across the rest of this handbook and beyond.

2.1 Introduction

The best way to anticipate the risks and concerns about the trustworthiness of facial recognition technologies (FRT) is to understand the way they operate and how such decision-making algorithms differ from other conventional information technology (IT) systems. This chapter presents a gentle introduction to characteristics, building blocks, and some of the techniques used in artificial intelligence (AI) and FRT solutions that are enabled by AI. Owing to simplification and limitation, this is by no means a complete or precise representation of such technologies. However, it is enough to better understand some of the available choices, the implications that might come with them, and considerations to help minimise some of the unwanted impacts.

When talking about facial recognition technologies, usually the first thing that comes to mind is identifying a person from their photo. However, when analysing an image that includes a face, quite a few processes can be done. Apart from the initial general image preparation and enhancement steps, everything starts with a face detection process. This is the process to find the location of all of the faces within an image, which usually follows by extracting that part of the image and applying some alignments to prepare it for the next steps.

Face recognition that follows the detection step deals with assessing the identity of the person in the extracted face image and can be either an identification or a verification process. Face identification is when a 1:N, or one-to-many, search happens and the target face image is compared with a database of many known facial images. If the search is successful, the identity of the person in the image is found. For example, when doing a police check, a newly taken photo of the person might be checked against a database of criminal mugshots to find if that person had any past records. In the verification process, by performing a 1:1, or one-to-one check, we are actually trying to confirm an assumed identity by comparing a new facial image with a previously confirmed photo. A good example for this can be when a newly taken photo at a border checkpoint is compared with the photo on the passport to confirm it is the same person.

Although it is not always categorised under the facial recognition topic, another form of facial image processing is face categorisation or analysis. Here, rather than the identity of the person in the image, other characteristics and specifications are important. Detecting some demographic information such as gender, age, or ethnicity, facial expression detection, and emotion recognition are a few examples with applications such as sentiment analysis, targeted advertisement, attention detection, or driver fatigue identification. However, this sub-category is not the focus in this text.

All of the above-mentioned processes on facial images fall under the computer vision field of research, which is about techniques and methods that enable computers to understand images and extract various information from them. This closely relates to image processing, which can, for example, modify and enhance medical images but not necessarily extract information or automatically make decisions based on them. Eventually, if we go one step further, along with computer vision and image processing, any other unstructured data processing such as speech processing or natural language processing falls under the umbrella of AI. The importance of this recognition is that facial recognition technologies inherit a lot of their characteristics from AI, and in the next section we take a closer look at some of these specifications to better understand some of the underlying complexities and challenges of FRT.

2.2 What Is AI?

Although there have been many debates around the definition of AI, we do not yet have one universally accepted version. The definition by the Organisation for Economic Co-operation and Development (OECD) is among one of the more commonly referenced ones: ‘Artificial Intelligence (AI) refers to computer systems that can perform tasks or make predictions, recommendations or decisions that usually require human intelligence. AI systems can perform these tasks and make these decisions based on objectives set by humans but without explicit human instructions.’Footnote ¹

2.3 AI Life Cycle and Success Factor Considerations

Considering the foregoing, the life cycle of AI systems also differs slightly from the common software development life cycle. Figure 2.1 is a simple view of these life cycle steps.

Figure 2.1 AI system life cycle

2.4 Under the Hood of AI

At a very simplistic level and in a classic view, an AI system consists of a form of representation of knowledge, an inference engine, and an optional learn or retrain mechanism, as illustrated in the Figure 2.2.

Figure 2.2 AI system key components

Knowledge in an AI system may be encoded and represented in different forms including and not limited to rules, graphs, statistical distributions, mathematical equations and their parameters, or a combination of these. The knowledge base represents facts, information, skills or experiences from human knowledge or existing relationships, associations, or other relevant information in the environment that can help in achieving the main objective of the AI system. For example, in an FRT system the knowledge might define what shapes, colours, or patterns can indicate the location of a human face in the input image. Or it can suggest what areas and measurements on the face would be the most discriminating factors between two different human faces. However, it is not always as explicit and explainable as these examples.

Inference engine consists of the algorithms, mechanisms, and processes that allow the AI system to apply knowledge to the input facts and observations and to come up with the solutions for achieving its objective, making a prediction or a decision. The type of inference engine depends on the knowledge representation model to be able to apply that specific type of model, and usually they come as a pair. However, these two components are not always necessarily separable. For example, in AI systems based on artificial neural networks (ANNs), the knowledge is stored as the trained parameters and weights of the network. In such cases we can consider that the inference engine and knowledge base are combined as an ANN algorithm together with its parameters after training.

Learn or retrain, as already mentioned, is an optional component of the AI system. Many AI systems after being fully trained and put into operation remain static and do not receive any feedback from the environment. However, when the ‘learn’ component exists, after making a decision or prediction, the AI system receives feedback that indicates the correct output. The learning mechanism compares the predicted output with the feedback and, in case of any deviation or error, it tries to readjust the knowledge to gradually minimise the overall error rate of the system. For example, every time that your mobile phone Face ID fails to identify your face and you immediately unlock the phone using your passcode, it can be used as a feedback signal to improve your face model on the phone by using the most recently captured image. While this is a great feature for improving AI models, it also has the risk of changing their behaviour in an unexpected or unwanted manner. In the example just given, if with each failure your mobile phone keeps expanding the scope of acceptable facial features that unlock your phone, it may end up accepting other people whose faces are only similar to yours.

2.4.1 The Source of Knowledge

We have just mentioned how the knowledge base might be updated and improved based on the feedback received during the operation. But what is the source of the knowledge and how that knowledge base is created in the first place? Generally speaking, during the initial build of an AI system the knowledge base can be created either manually by the experts or automatically using suitable data. You might have previously seen illustrations similar to Figure 2.3, which tries to explain the relation between AI and machine learning (ML). However, before getting to the details of ML, it might be good to consider what is AI outside the ML subset.

Figure 2.3 AI versus ML

The AI techniques outside the ML subset are called Symbolic AI or sometimes referred to as Good Old-Fashioned AI. This is mostly based on the human expert knowledge in that specific domain, and the knowledge base here is being manually curated and encoded by the AI developers. As a result of that, it is mostly human readable (hence symbolic) and usually separable from the inference part of the system as described in the building blocks of AI earlier. Expert Systems are one of the well-known and more successful examples of symbolic AI, where their knowledge is mainly stored as ‘if-then’ rules.Footnote ⁵

Symbolic AI systems are relatively reliable, predictable, and more explainable owing to their transparency and the readability of their knowledge base. However, the manual curation of the knowledge base makes it less generalisable and more importantly converts the knowledge acquisition or updating step into a bottleneck owing to the limited availability of the domain experts to collaborate with the developers. Symbolic AI solutions have therefore had limited success, and we have not heard much about them recently.

To obtain knowledge without experts dictating it, another approach is to observe and automatically learn from the relevant examples, which is the basis of computational learning theory and ML techniques. There is a wide range of ML techniques starting from statistical models and mathematical regression analysis to more algorithmic methods such as decision trees, support vector machines, and ANNs, which are one of the most well-known subsets of ML in the past couple of years, thanks to the huge success stories of deep neural networks.Footnote ⁶ When enough sample data is provided, these algorithms are capable of training models with automatically encoded knowledge that is required to achieve their objectives when put into operation. The table in Figure 2.4 summarises some of the key differences between these two groups of AI techniques.

Figure 2.4 Symbolic AI versus ML

2.5 Facial Recognition Approaches

Similar to the AI techniques, facial recognition approaches were initially more similar to Symbolic AI. They were naturally more inclined towards the way humans might approach the problem and were inspired by anthropometry.Footnote ⁷ Owing to the difficulty of extracting all important facial features and accurate measurements that could be easily impacted by small variation in the images, there was limited success in such works until more data driven approaches were introduced; these were based on mathematical and statistical methods and had a holistic approach to face recognition, an example being Eigenfaces,Footnote ⁸ which is basically the eigenvectors of the training grayscale face images (An eigenvector of a matrix is a non-zero vector that, when multiplied by the matrix, results in a scaled version of itself.). This shift towards ML techniques got more mature and successful by combining the two approaches through other ideas such as neural networks in DeepFace,Footnote ⁹ and many other similar works. More in-depth review of the history of FRT is discussed in Chapter 3, so here we just look at technical characteristics and differences of these approaches.

Feature analysis approaches rely on the detection of facial features and their measurements. Here, each face image is converted to a numeric vector in a multi-dimensional space and the face recognition challenge is simplified to more common classification or regression problems. Similar to symbolic AI, the majority of the knowledge, if not all, is manually encoded in the form of rules that instruct how to detect the face within an image and identify each of its components to be measured accurately. These rules may rely on basic image and signal processing techniques such as edge detection and segmentation. This makes the implementation easier and, as mentioned earlier when discussing symbolic AI, the process and its decision-making is more transparent and explainable. However, intrinsic to these approaches is the limited generalisability challenge of symbolic AI. In ideal and controlled conditions these methods can be quite accurate, but changes in the imaging condition can dramatically impact the performance. This is because in the new conditions, including different angles, resolution, or shadows and partial coverage, the prescribed rules might not apply any more, and it would not be practical to manually find all these variations and customise new rules for them.

Holistic approaches became popular after the introduction of Eigenfaces in the early 1990s.Footnote ¹⁰ Rather than trying to detect facial features based on human definition of a face, these approaches consider the image in its pixel form as a vector in a high dimensional space and apply dimensionality reduction techniques combined with other mathematical and statistical approaches that do not rely on what is inside the image. This largely simplifies the problem by avoiding the facial feature extraction and measurement step, together with its sensitivities. This shifts the face recognition approach towards the classic ML techniques and changes the training to a data-driven problem rather than manual rule development. Unfortunately, purely holistic approaches still suffer from a few challenges, including statistical distribution assumptions behind the method that do not always apply, and any deviation from the controlled imaging condition makes it worse.

Deep neural networks made a leap in the advancement and success of face recognition approaches. After Eigenfaces and its variations, there were many other small improvements made to the holistic approaches by adding some generic feature extraction steps such as Gabor prior to the main classifier,Footnote ¹¹ followed by some neural network-based ML approaches. However, it was not as successful until the introduction of deep learning for image processing,Footnote ¹² and applying it for face recognition.Footnote ¹³ Convolutional neural networks convert the feature extraction and selection from the images to an unsupervised process, so it is not as challenging as manually defined facial features and not too generic like the Gabor filters used prior to some of the holistic approaches. The increasingly complex and important features that are automatically selected are used in a supervised learning layer to deliver the classification or recognition function.Footnote ¹⁴ This is the key in the success of object and face recognition of deep neural networks.

2.6 The Gift and the Curse of Complexity

Many variations of ANNs have been used in ML applications including face recognition. However, the so called shallow neural networks were not as successful owing to their limited learning capacity. Advancements in hardware, use of graphical processing units, and cloud computing to increase processing power along with access to more training data (big data) made the introduction of deep learning possible. In addition to novel network structures and the use of more sophisticated nodes such as convolutional functions, another important factor in the increased capacity of learning of DNNs is the overall complexity and scale of the network parameters to train. For example, the first experimental DNN used in FaceNet includes a total of 140 million parameters to train.

While the complexity of DNNs increases their success in learning to solve challenging problems such as face recognition, these new algorithms become increasingly data hungry. Without going into too much detail, if the number of training samples are too small compared with the number of parameters of the model, rather than learning a generalised solution for solving the problem it overfits the model and memorises the answer only for that specific subset. This causes the model to perform very well for the training samples, as it has memorised the correct answers for the training set, but fail when it comes to test and unseen samples, owing to a lack of generalisation and fitting the model only to the previously seen examples. Therefore, such successful face recognition systems based on DNNs or a variation of them are actually trained on large training facial datasets, which can be the source of new risks and concerns.

Privacy and security concerns are one of the first to pay attention to. It is difficult and expensive to create new and large face datasets with all appropriate consents in place. Many of these large datasets are collected from the web and from a few different sources where copyright and privacy statements raise problems from both legal and ethics perspectives. Additionally, after collection, such datasets could be potentially a good target for cyber-attacks, especially if the images can be correlated to other information that may be publicly available about the same person.

Data labelling is the next challenge after collection of the suitable dataset. It is labour intensive to manually label such large datasets to be used as a supervised learning signal for the models. As discussed earlier, self-supervised learning is one of the next best choices for data-heavy algorithms such as DNNs. However, this introduces the risk of incorrect assumptions in the self-supervised logic and the missing of some problems in the training process even when performance measures seem to be adequate.

Hidden data quality issues might be the key to most of the well-known face recognition failures. Usually, a lot of automation or crowdsourcing is involved in the preparation of such large face datasets. This can prevent thorough quality checks across the samples and labels, which can lead to flawed models and cause unexpected behaviour in special cases despite high performance results during the test and evaluation. Bias and discrimination are among the most common misbehaviours of FRT models, which can be either due to such hidden data quality issues or simply the difficulty of obtaining a well-balanced large sample across all cohorts.

2.7 Conclusion

Face recognition is one of the complex applications of AI and inherits many of its limitations and challenges. We have made a quick review of some of the important considerations, choices, and potential pitfalls of AI techniques and more specifically FRT systems. Given this is a relatively new technology being used in our daily lives, it is crucial to increase the awareness and literacy of such technologies and their potential implications from a multi-disciplinary angle for all its stakeholders, from its developers and providers to the operators, regulators, and the end users.

Now that with DNNs the reported performance of FR models is reaching or surpassing human performance,Footnote ¹⁵ a critical question is why we still hear so many examples of failure and find FR models insufficiently reliable in practice. Among many reasons, such as data quality discussed earlier, the difference between development and operation conditions can be one of the common factors. The dataset that the model is trained and tested on may not be a good representation of what the model will receive when put into operation. Such differences can be due to imaging conditions, demographic distribution, or other factors. Additionally, we should not forget that the performance tests are usually done directly on the FRT model. However, an FRT-based solution has a lot of other software components and configurable decision-making logic that will be applied to the facial image similarity scores. For example, such surrounding configurable logic can easily introduce human bias to a FRT solution with a good performing model at core. Finally, it is worth reminding that like many other software and digital solutions, FRT systems can be subject to adversarial attacks. It might be a lot easier to fool a DNN-based FR model using adversarial samples or patches compared with the human potential for identifying such attempts.Footnote ¹⁶

Hence, considering all such intentional and unintentional risks, are the benefits of FRT worth it? Rather than giving a blanket yes/no answer, it should be concluded that this depends on the application and impact levels. However, making a conscious decision based on a realistic understanding of potentials and limitations of technology, along with having humans in the loop, can significantly help to minimise these risks.

3.1 Introduction

On 10 September 2020, Pace Gallery in London held an exhibition by the artist Trevor Paglen examining the visual products from artificial intelligence and digital data systems.Footnote ¹ Titled ‘Bloom’, the exhibition featured an over-sized sculpture of a human head. Bald, white, and possibly male, this eerily symmetrical ‘standard head’ had been modelled on measurements from canonical experiments in facial recognition history by Woody Wilson Bledsoe, Charles Bisson, and Helen Chan Wolf occuring at Panoramic Research Laboratory in 1964.Footnote ²

Centring this ‘standard head’ in the space, Paglen surrounded it with photographic prints of leaves and flowers re-composed from RAW camera files by computer vision algorithms. These machine visualisations of nature encircled the ‘standard head’ illustrating how digital imaging using autonomous toolsets can achieve significantly different graphical outcomes. The exhibit foregrounded face recognition technology yet provoked viewers to consider the cross-practice connections between computing and data classification, humans and nature, and how image-making is becoming technically autonomous.Footnote ³ Another take-away is how these systems require multi-faceted elements to work and the ‘mushrooming and blossoming from all kinds of datasets’.Footnote ⁴

As a form of networked visual surveillance, facial recognition technology (FRT) works from the extent to which it operates in larger information infrastructures, FRT ‘is not a single technology but an umbrella term for a set of technologies’.Footnote ⁵ These digitally networked systems allow imaging data to transform from one state to another, and transfer from one site to another. Recent improvements in FRT, as a remote identification system, has reached a point to be technically possible to capture biometric images and data from subjects in public, private, and personal spaces, or interactions online, without their consent or awareness, or adequate regulatory oversight. This includes a distribution of sensitive and personal user information between state and private-sector organisations, while contributing to training machine learning tools using honeypots of data, and enabling ‘ever more sophisticated and effective forms of social control’.Footnote ⁶

Unlike the suggestion of Paglen’s exhibition, the origins of FRT cannot be reduced to the experiments in 1964. We need to widen the lens as the technical operations Stakeholders inside these systems are globally distributed and as Chair of Electronic Frontiers Australia’s Policy Committee, Angus Murray iterated require ‘bargains of trust’.Footnote ⁷ For example, Domestic and federal police agencies use systems that rely on huge amounts of data aggregation in private cloud servers and proprietary hardware that store and transmit data from online platforms, smart devices, foreign owned closed-circuit television (CCTV) companies and creators of wearable body cameras.Footnote ⁸ In Australia, retail outlets such as Bunnings use FRT and identity data to extract information from social media, where most people have images of themselves uploaded. They perform analysis based on the specific visits and transactions for certain shoppers.Footnote ⁹ Similarly images captured in public spaces, of crowds or of protesters, can be matched to social media posts or online forums managed by global technology firms, such as Facebook and Google, or transnational intelligence agencies such as the NSA and GCHQ. In the United Kingdom, Daragh Murray witnessed FRT software draw rectangles around the faces of people in public streets from a live CCTV feed. The system then extracted key features and compared these with stored features of criminal suspects in a watch list.Footnote ¹⁰ Matching an image to a watchlist is not the only function to consider here, but a need to query the distribution and ownership of data in the system being collectively assembled by the Tokyo-based technology giant NEC, in the example provided above.Footnote ¹¹ Other examples of this diffuse and operational data flow include how China’s Zhejiang Dahua Technology Co. Ltd sold thermal imaging cameras, armed with facial recognition software, to scan workers entering Amazon factories during COVID-19, that is despite them being black-trade listed in the United States.Footnote ¹²

FRT and its computer procedures are therefore systems and ‘technologies in the making’, not artefacts with singularly defined origins and easy to regulate outcomes.Footnote ¹³ While an abundance of research looks at the use of FRT in border security and biometric surveillance,Footnote ¹⁴ retail shopping or school aged education,Footnote ¹⁵ and the gendering and racial divide between datasets with calls to ban these systems,Footnote ¹⁶ other elements also require scholarly, legislative, and regulatory attention.

This chapter considers how large-scale technical systems such as FRT have bloomed yet build on the echnical roots of multiple systems and the provenance of data sources that remain under considered. Tracing the genealogical origins and provenance of such datasets and statistical toolsets plays an important role in framing current uses for regulatory challenges. In this regard, this chapter presents empirical findings from research on early Indian statistical measures, the convergence of Chinese and Western technology companies, and the increase in computer vision experiments including those conducted on animals for bio security identification purposes. This chapter argues these diverse material innovations and information domains not only act as testbeds for FRT systems, but encompass some of the globalised products contained in FRT infrastructure.Footnote ¹⁷

3.2 FRT Does Not Have a Singular Origin, They Are ‘Systems in Motion’

Bledsoe’s ‘standard head’ algorithm didn’t remain at the University of Texas nor in the domain of artificial intelligence history. Owing to funding by the RAND Corporation, the algorithm worked its way into informational models for law enforcement purposes. In the development of the New York State Intelligence and Identification System (NYSIIS), Bledsoe was recruited to develop his algorithm to computationally solve ‘the mug-file problem’.Footnote ¹⁸ By contributing to the world’s first computerised criminal-justice information-sharing system,Footnote ¹⁹ as Stephanie Dick posits, Bledsoe’s algorithm and its ideas travelled with his over-simplifications and data assumptions in tow.Footnote ²⁰ This influenced not only law enforcement databases and decisions on criminal targets in the United States, but also FRT developments that followed.Footnote ²¹ In its final state the algorithm was not used to automatically detect faces – as FRT does now – but contributed to a standardisation of ‘mug shot’ photos for computer filing systems. Bledsoe, who was later the president of the Association for the Advancement of Artificial Intelligence, used 2,000 images of police mug shots as his ‘database’ for making comparisons with a new set of photographs to detect any similarity. This American National Standards Institute Database, whose archives of mug shots featured convicted criminals (and those just accused), was the predominant source of visual information for Bledsoe’s facial-recognition technology (a role now filled by social media).Footnote ²² To this end, Bledsoe and his Panoramic Research collaborators manually drew over human facial features with a device that resembled an iPad called a GRAFACON or RAND tablet. By using a stylus, images were rotated and re-drawn onto the tablet and recorded as coordinates on a grid. This produced a relatively high-resolution computer readable image. A list of distances were calculated and recorded as a person’s identification code for locations such as the mouth, nose, or eyes.Footnote ²³ Facial recognition (at this time) was a mathematical code of distances between features, drastically reducing individual and social nuances between them, and largely informed by Bayesian decision theory to use ‘22 measurements to make an educated guess about the whole’.Footnote ²⁴

In essence, Bledsoe had computerised the mug shot into a ‘fully automated Bertillon system for the face’.Footnote ²⁵ This system, invented by French criminologists Cesare Lombroso and Alphonse Bertillon in 1879, gained wide acceptance as a reliable and scientific method for criminal investigation, despite problematic eighteenth-century anthropometric experiments. The mug shot was invented to recognise criminal suspects who were repeatedly arrested: portraits were drawn and statistically labelled on common morphological characteristics.Footnote ²⁶ The resulted ‘mug shots’ were standardised and collected by police departments and accepted as evidence in courts. Photo IDs modelled on the mug shot not only became an official format for policing, but have become standard issue in nation-state passports presented at airports and for driver’s licence photographs. The first ever US photo driver’s licence, issued in 1958, was created by French security company IDEMIA – a world leader in biometric security. Founded in 1922 as the defence contractor SAGEM, it then became SAGEM-Morpho in the 1980s, and parts of IDEMIA go back even further, and they have effectively led to every shift in the photo identity issuance and credentialling in the US since.Footnote ²⁷

Bledsoe’s 1960s laboratory experiments thus relied on two separate building blocks invented in France. Hampered by the technology of his era, Bledsoe’s ideas for FRT were not truly operationalised until the 1990s – driven by a technological wave of mobile phone and personal computer sales, online networked wireless video systems, and digital cameras.Footnote ²⁸ Yet the experimental use of FRT is still being conducted in a way largely never done before.Footnote ²⁹ Clare Garvie contends that forms of automated imaging for policing actions remain unregulated and represent a ‘forensic science without rules’:

In the next section, I explore how the automation of FRT relies not only on a diverse manufacturing of ‘images’ – products of reduction, appropriation, transformation, or digital manipulation – and situated instances of exploitation conducted in South America, the United States, France, Russia, Japan, and China to name a few different jurisdictions, but also how modern FRT resurrects a century old vision of ‘statistical surveillance’.Footnote ³¹ To do so, I consider how a 100 year old mathematical experiment in British India has aided the probabilistic functionality of autonomous FRT systems.

3.3 The ‘Mind Boggling Systems’ Where Everyone Only Ever Has One ID

In 1991 Turk and Pentland produced the first real-time automated face recognition.Footnote ³² Famously, this was deployed at the crowded USA Super Bowl in 2001. This experimental trial was called ‘Facefinder’. The system captured surveillance images of the crowd and compared them with a database of digital mug shots held by Tampa police, the Florida Department of Law Enforcement and the FBI.Footnote ³³ The experiment not only demonstrated the potential for remote surveillance of crowds, but also led to the National Institute of Standards creating a Face Recognition Vendor Test (FRVT) to evaluate this emerging FRT market.

A quick look at the ongoing FRVT of 1: N facial algorithms reveals a globalised picture: ‘The report lists accuracy results alongside developer names as a useful comparison of facial recognition algorithms and assessment of absolute capability. The developer totals constitute a substantial majority of the face recognition industry.’Footnote ³⁴ This includes performance figures for 203 prototype algorithms from the research laboratories of over fifty commercial developers and one university. Similar to Beldsoe’s 1960s experiments for NYSIIS, this evaluative test scenario also uses frontal mug shots and profile view mug shots alongside desktop webcam photos, visa application photos, immigration lane photos, and traveller kiosk photos.

A brief survey of this report illustrates the scale and scope of a global FRT market. To name a few vendors, the developers and their places of origin include NEC (Tokyo); Microsoft (United States); Veritas (Spain); Herta Security (Spain); AnyVision (Israel); IDEMIA (France), utilised in Kenya and in Turkey; Daon (Ireland); Dahua (China); Moonwalk (China); Sensetime (China); Hyperverge (California); Cognitec (Germany); QNAP (Taiwan); Tevian (Russia); VisionLabs (Russia/Netherlands); Clearview AI (United States); DeepGlint (China) and finally Neurotechnology (Lithuania), which is a provider of deep-learning-based solutions for high-precision biometric identification and object recognition technology.

Importantly, the Lithuania based Neurotechnology recently partnered with Tata Consultancy Services as one of three biometric service providers for the largest biometric ID system in the world, Aadhaar.Footnote ³⁵ Co-ordinated by The Unique Identification Authority of India, the system registers people and compares their facial biometric with the existing records of 1.3 billion people to verify applicants have not registered under a different name. Aadhaar is ‘a mind-boggling system’, says Anil Jain, a computer scientist who consulted on the scheme, ‘and the beauty is that it ensures one person has only one ID’.Footnote ³⁶

India has a rich history of producing material and statistical innovations to identify individuals based on their physical characteristics.Footnote ³⁷ In 2020, Google posted an online tribute to Professor Prasanta Chandra Mahalanobis (1893–1972) as part of its ‘Arts and Culture’ series.Footnote ³⁸ Mahalanobis is famous for creating new statistical and biometric functions as key technologies he advocated to the world through his Indian Statistical Institute.Footnote ³⁹ The global celebration of his work was recognised in part after his creation of a similarity distance metric in 1936. This was produced from his specific interest in racial classification.Footnote ⁴⁰ He developed a biometric function to analyse and identify people based on physical and racial similarity. To do so he compared data collected from the Chittagong Hill Tract area (modern Bangladesh) with international race data sets collected from Swedish and Chinese records.Footnote ⁴¹ He then set about learning how to create an identification of race based on statistical measurements of facial features and their similarity, which he could apply in India. The aim was to help identify exotic and ethnic caste groups to be classified in the British colonial administration.Footnote ⁴²

Significantly, he also innovated by using facial photographs of living subjects to compare the accuracy of his biometric measurements, compared with analysing skulls in the era’s practice of phrenology.Footnote ⁴³ By testing his distance function with the invention of an experimental imaging device in 1937, Mahalanobis was a central figure in pushing ‘part of a biometric nationalism in which the face provided a form of data’.Footnote ⁴⁴ His metric, commonly known as a Mahalanobis Distance Function, despite being created eighty-six years ago, is consistently used in modern FRT.

Even the most sophisticated and large-scale FRT systems necessitate this basic approach of comparing images on facial features by using scores that compare a match of the similarity.Footnote ⁴⁵

In technical terms, the selection of a decision metric – such as the Mahalanobis Distance Function – ‘[h]elps to measure distances between specific facial features and generate a unique representation (as a ‘facial signature’) for each human face.Footnote ⁴⁶ Similar to Bledsoe’s code, this is then compared with a database of stored images in order to match a face to similar images.

In this regard, similarity measure functions operationalise the matching process as a critical decision-making module. Selection of the proper similarity measure is thus an important determination for the accuracy of the matching result. Such measures include Minkowski distances, Mahalanobis distances, Hansdorff distances, Euclidean, and cosine-based distances.Footnote ⁴⁷ Yet the Mahalanobis distance is the best at structuring data for unknown targets. This is critical to criminal subject investigations for matching suspects from surveillance images in supermarkets, stadiums, or of protest crowds. The similarity measure enables high-speed cluster analysis – critical to a speed of decision-making – especially for faces with a high-number of variables and in relation to fitting an unknown person into a known database. FRT can then determine if an unknown image (taken from a web profile or a surveillance camera) matches a person in the database (compared with drivers’ licences or mug shots). This approach is also suitable for machine learning and is a prominent approach for training systems on person re-identification by ‘improving classification through exploiting structures in the data’.Footnote ⁴⁸

As Adriana Dongus suggests, ‘[t]he large datasets produced by science and law enforcement at the turn of the nineteenth century continue to form the material backbone and precedent to current machine learning.’Footnote ⁴⁹ By examining the critical and ubiquitous distribution and embedment of early decision classifiers, we establish the importance of selecting certain rule functions in ‘a statistical layer’ of FRT systems.

When applied to machine learning, this includes assigning weights to autonomously identify the importance in probable matches. This is used in image-labelled data sets,Footnote ⁵⁰ to estimating facial position poses from video,Footnote ⁵¹ to automatically locating an unproductive worker on a factory floor,Footnote ⁵² or identifying ethnic minority faces in a crowd, as is occurring in China with the Uyghur (Uighur) population. While much important work on facial recognition is salient to the United States,Footnote ⁵³ there is a need to examine how FRT is conditioned on a globalised supply chain. This includes the ‘production, schematization, maintenance, inflection, and reproduction of certain [decision] rules’ and how they replicate use of problematic standards in public surveillance.Footnote ⁵⁴

Indeed, there has been a ‘tendency to gloss over the amount of effort that goes into developing and integrating new technologies and systems with older technologies’.Footnote ⁵⁵ Computation moves fast – yet many lessons remain and are yet to be learned.

From legislative, ethical, and regulatory standpoints, it is worth noting that biometric systems and data (including use of statistical functions and facial images) are constructed on complex and interoperable supply chains involving third-party vendors needed to make these systems work. Yet there is potential incentives built within these globalised computing systems to exploit regulatory gaps and vulnerabilities that could be used against various human populations at a later date.Footnote ⁵⁶ The final section examines how Mahalanobis’s 100 year old experiment is relevant not only to our digital identity systems today, such as the United Nations High Commission for Refugees (UNHCR) Population Registration and Identity Management Eco-SystemFootnote ⁵⁷ but builds on different use-cases. These include not only nation-state surveillance, such as the identification and detection of ethnic minorities in China, but the increasing datafication of animals and computerisation of biosecurity measures in agriculture that can be transferrable to human populations.Footnote ⁵⁸

3.4 Dynamic Matching Strategies in FRT Extend Beyond Recognising Human Beings

To securely identify forcibly displaced persons seeking UNHCR repatriation assistance at refugee processing centres the UNHCR records biometrics such as iris, fingerprints, and facial metrics.Footnote ⁵⁹ Driven in part by a Biometric Matching Engine developed by Accenture, this Population Registration and Identity Management Eco-System (PRIMES) employs a patented ‘dynamic matching strategy’ comprising at least two sets of biometric modalities.Footnote ⁶⁰ With the advent of new, technologically advanced modes of biometric data gathering and analysis, some of the current ‘international legal thought, doctrine, and practice are, in the main, poorly equipped to deal with them’, especially in situations of forced migration.Footnote ⁶¹ One reason is the lack of manual processing options and how the introduction of machine learning can lift the collection of sensitive and personally identifiable information outside the scope of pre-existing legal methods. In grappling with new forms of quantification and statistics these systems do not just contain hundred-year old statistical decision functions but the pairing of imaging, data aggregation, and machine learning at scale. The autonomy granted to machine learning may remove abilities to interrogate the validity of the earlier datasets and matching results a system relies on to achieve a result. Such logic clusters ever increasing data collections into new ‘probabilistic dependencies’.Footnote ⁶² Yet what this curtails are reasonable efforts to disentangle bias from standardised classifications, and how the natural divergences that occur between humans, different social groups, and their situated actions, are erased in deference to the calculative inferences instead. In the use of FRT there is always ‘politics attached’. Avi Marciano illustrated this in the context of Israel where biometric standards establish hierarchies for decision making by defining particular bodies as ‘ineligible’ to access.Footnote ⁶³

Some FRTs are directly complicit in human rights abuses, including a reported detention of up to 1.5 million Uyghur Muslims in Xinjiang.Footnote ⁶⁴ Owing to the increasing scale of an inescapable surveillance that the Chinese Communist Party has funded, ubiquitous CCTV systems and facial recognition are operationalised in public spaces alongside the monitoring of online communications and patterns-of-life data from mobile phones. Idealised as an all-seeing pervasive surveillance network enabled by a state manufacturing of computer vision technology, digital platforms, and data aggregation centres,Footnote ⁶⁵ the simplified idea that Chinese technology and its authoritarian state surveillance system are indigenous is significantly flawed. Before China started using CCTV systems and facial pattern-matching techniques to identify ethnic minorities in Xinjiang Province, Bledsoe proposed to the Defence Department Advanced Research Projects Agency (then known as ARPA) that it should support Panoramic Research Laboratory in studying the feasibility of using facial characteristics to determine a person’s racial background.Footnote ⁶⁶ This is another instance of the politics and the power of FRT recurring and returning and re-playing into new uses, new places, and new eras, yet with similar purposes.

Western companies were involved in the creation of these systems at the start. The export of surveillance technologies from the Global North to China started in the 1970s. It is only now that Chinese technology companies are found competing with and replacing those suppliers in a globalised market.Footnote ⁶⁷ The current status of FRT developed in China with known human rights and privacy violations is not adequately restricted by regulatory frameworks in Europe and the United States.Footnote ⁶⁸ To better disentangle use-cases requires not only a more through mapping of globally entangled and technical supply-chains, whether through critical research or in the building of oversight capabilities such as independent risk assessments, compliance audits, or technical red-teaming in the light of such swiftly evolving material properties.

A contemporary focus on understanding FRT must therefore be concerned not only with the implementation and implications for nation and state-bound privacy law, but to make transparent infrastructural supply chains and situated origins of datasets and technical domains they were created in. This should not simply be restricted to law enforcement and public organisations being required to undertake better procurement strategies – often limited to purchasing orders or responses to requests for information – but to identify the exact sources of the FRT hardware, software, decision functions, and datasets.Footnote ⁶⁹

Indeed, there are circumstances in which we may need to look further afield. This includes so-called dual-use systems that are adopted not just from domains in nation state and military operations but those trained on animals within precision agriculture.Footnote ⁷⁰ In the shift from classical identification methods to computer vision tools, the future of farming lies in the paddock-to-plate digital identification of each product. Whether for cross-border bio-security purposes or the optimisation of meat traceability FRT is seen as a viable investment to remotely track animals. These systems commonly utilize open-source software architectures, machine learning and modular camera systems.Footnote ⁷¹ Yet in the computational transference between animal bodies, digital and data visualisation, and informational materials, we collapse into the heart of Trevor Paglen’s art project titled in ‘Bloom’. The visualisation and classification of all images and all bodies helps to establish the adoption of autonomous methods. This includes initiatives from the global accounting firm KPMG and Meat and Livestock Australia to collect data that translate into efforts to strengthen computer vision market positions. Agribusinesses are not yet treated as handling any sensitive data or training bodily surveillance systems nor are they subjected to regulatory approaches that can throw their data practices into question.Footnote ⁷²

As Mark Maguire suggests, a genealogical and infrastructural approach to FRT ‘demands we consider how technologies are an assemblage of different elements delivered from specific contexts’ yet re-made, aggregated, customised, adapted, and re-purposed for newly defined, profit-driven, and yet often speculative objectives.Footnote ⁷³

3.5 Conclusion

At the time of Bledsoe’s experiments there was a meeting between the administrative management of the NYSIIS law enforcement data bases and the computer design company Systems Development Corporation (SDC) of Santa Monica, California, in September 1964.Footnote ⁷⁴ The aim was to decide in what manner to proceed with the implementation of the system, and what techniques to commission for deployment. In summary, the critical inflexion point centred on: ‘First buy the computer and decide what to put on it; (2) Or do an extensive feasibility analysis and, as a result of that study, decide on the computer (how large and powerful) and the functions to be performed.’Footnote ⁷⁵

As the technical capacity of computing systems in the 1960s was nascent, SDC lacked capability to deliver the required system at scale. Yet this allowed a pause for discussion, consideration, and to recognise that computing capabilities must be defined for a particular purpose, and there should be a thorough vetting of the modular building blocks the system would contain.Footnote ⁷⁶ The title of that report was ‘A System in Motion’, and it recognised that multiple capabilities – from query and search functions onto image recognition – could not be adequately managed and regulated when developed at once. The NYSIIS report stated the application of computers to solve recognition problems for law enforcement was a foregone conclusion. Yet the question remained whether social institutions and organisations should allow for deploying use of complete automation, especially as they function as a sum of moving, and largely unknown ‘experimental parts’?Footnote ⁷⁷

Although most state departments and law enforcement undertake basic steps to adhere to industry best practices, such as compliance, testing, and legal obligations to avoid public scrutiny, these approaches often lack consistency. FRT is an experimental practice constituted by practices and elements that can be hidden from view, trialed and tested in domains unsuitable to be deemed fit-for-purpose. Whether being trained on exploitative data captured from refugees, prisoners, or operationalised on farm animals, this is called ‘the deploy and comply problem’ and requires public consultation and impact considerations before being put into action.Footnote ⁷⁸ A prime example is the use of Clearview AI facial algorithms by New Zealand Police in 2020 without consulting the Privacy Commissioner or considering the impacts to vulnerable Indigenous groups.Footnote ⁷⁹ This is indicative of multiple instances of harm, error, oppression, and inequality that have been caused by autonomous decision and surveillance systems.Footnote ⁸⁰ What is needed are efforts to trace, assess, and determine if the modular ‘elements’ of an FRT system are legitimate, credible, feasible, and reasonable. This challenge seeks to ringfence the ‘lineage of intent’ – yet can FRT systems be restricted by ethical, legal and technical guardrails to specific, deliberate, and predefined purposes?Footnote ⁸¹ This is what this book is seeking to address.

4.1 Introduction

Facial recognition technology (FRT) is being increasingly used by border authorities, law enforcement, and other government institutions around the world. Research shows that among the 100 most populated countries in the world, seven out of ten governments are using FRT on a large-scale basis.Footnote ¹ One of the major challenges related to this technology is the lack of transparency and explainability surrounding it. Numerous reports have indicated that there is insufficient transparency and explainability around the use of artificial intelligence (AI), including FRT, in the government sector.Footnote ² There are still no clear rules, guidelines, or frameworks as to the level and kind of transparency and explainability that should be expected from government institutions when using AI more generally, and FRT in particular.Footnote ³ The EU General Data Protection Regulation (GDPR) is among the first instruments to establish a right of explanation in relation to automated decisions,Footnote ⁴ but its scope is very limited.Footnote ⁵ The proposed EU Artificial Intelligence Act (Draft EU AI Act) sets minimum transparency standards to high-risk AI technologies that include FRT.Footnote ⁶ However, these transparency obligations are generic to all high-risk AI technologies and do not detail transparency requirements for FRT specifically.

Transparency and explainability are arguably essential to ensuring the accountability of government institutions using FRT; empowering supervisory authorities to detect, investigate, and punish breaches of laws or fundamental rights obligations; allowing individuals affected by an AI system’s outcome to challenge the decision generated using AI systems;Footnote ⁷ and enabling AI developers to evaluate the quality of the AI system.Footnote ⁸ According to the proposed EU AI Act, ‘transparency is particularly important to avoid adverse impacts, retain public trust and ensure accountability and effective redress’.Footnote ⁹

At the same time, one should note that transparency and explainability of FRT alone would not help remedy essential problems associated with FRT use, and might further contribute to its negative impacts in some cases. For instance, if an individual learns about the government use of FRT in public spaces where public gatherings take place, this might discourage her from participating in such gatherings and thus have a ‘chilling effect’ on the exercise of her human rights, such as freedom of speech and freedom of association.Footnote ¹⁰ These considerations have to be kept in mind when determining the desirable levels of FRT transparency and explainability.

While there is extensive technical literature on transparency and explainability of AI in general,Footnote ¹¹ and of FRT more specifically,Footnote ¹² there is very limited legal academic discussion about the requisite extent of transparency and explainability of FRT technologies, and challenges in ensuring it, such as trade secrets. The goal of this chapter is to examine to what extent trade secrets create a barrier in ensuring transparent and explainable FRT and whether current trade secret laws provide any solutions to this problem.

This chapter first identifies the extent to which transparency and explainability is needed in relation to FRT among different stakeholders. Second, after briefly examining which types of information about AI could be potentially protected as trade secrets, it identifies situations in which trade secret protection may inhibit transparent and explainable FRT. It then analyses whether the current trade secret law, in particular the ‘public interest’ exception, is capable of addressing the conflict between the proprietary interests of trade secret owners and AI transparency needs of certain stakeholders. This chapter focusses on FRT in law enforcement, with a greater emphasis on real-time biometric identification technologies that are considered the highest risk.Footnote ¹³

Apart from the critical literature analysis, this chapter relies on empirical data collected through thirty-two interviews with experts in AI technology. The interviews were conducted with representatives from five stakeholder groups: police officers, government representatives, non-governmental organisation (NGO) representatives, IT experts (in academia and private sector), and legal experts (in academia and private sector) from Europe, the United States, and Asia-Pacific (October 2021–March 2022, online). The data collected from these interviews is especially useful when identifying the transparency and explainability needs of different stakeholders (Section 4.2).

Keeping in mind the lack of consensus on the terms ‘AI transparency’ and ‘AI explainability’, for the purpose of this chapter we define the concepts as follows. First, we understand the ‘AI transparency’ principle as a requirement to provide information about the AI model, its algorithm, and its data. The AI transparency principle could require disclosing very general information, such as ‘when AI is being used’,Footnote ¹⁴ or more specific information about the AI module – for example, its algorithmic parameters, training, validation, and testing information. While this concept of transparency might require providing very different levels of information for different stakeholders, it does not include information about how AI decisions are being generated. The latter is covered by the principle of ‘AI explainability’, which we define in a narrow technical way; that is, as an explanation of how an AI module functions, and how it generates a particular output. Such explanations are normally provided using so called Explainable AI (XAI) techniques.Footnote ¹⁵ Generally speaking, XAI techniques might be ‘global’, explaining the features of the entire module; or ‘local’, which explain how a specific output has been generated.Footnote ¹⁶ While this chapter largely focusses on FRT transparency and its possible conflict with trade secret protection, it also briefly reflects upon the need for FRT to be explainable.

In the following sections, we discuss the scope of explainability and transparency that different stakeholders need in relation to FRT in law enforcement (Section 4.2), in which situations trade secrets may conflict with these transparency and explainability needs (Section 4.3), and whether the ‘public interest’ defence under trade secrets law is capable of addressing this conflict (Section 4.4).

4.2 FRT Transparency and Explainability: Who Needs It and How Much?

Before examining whether trade secrets conflict with FRT transparency and explainability principles, we need to clearly identify the level of transparency and explainability that different stakeholders require in relation to FRT. We demonstrate that different stakeholders need very different types of information, some of which is – and some is not – protected by trade secrets.

For the purpose of this analysis, we identified six categories of stakeholders who have legitimate interests in certain levels of transparency and/or explainability around FRT technologies: (1) individuals exposed to FRT; (2) police officers who directly use the technology; (3) police authorities that acquire/procure the technology and need to ensure its quality; (4) court participants, especially court experts, who need access to technical information to assess whether the technology is of sufficient quality; (5) certification and auditing bodies examining whether the FRT meets the required standards; and finally (6) public interest organisations (NGOs and public research institutions) whose purpose is to ensure, in general terms, that the technology is high quality, ethical, legal, and is used for the overall public benefit.

As could be expected, our interviews with stakeholders have shown that different stakeholders have different explainability and transparency needs in relation to FRT.

4.3 In Which Situations Might Trade Secrets Inhibit Transparency of FRT?

There are a number of challenges in ensuring transparency around FRT.Footnote ³³ One of them is trade secrets, which can arguably create barriers to ensuring transparency of AI technologies in general and FRT technologies in particular. The example often used is the State v. Loomis case decided by a US court, in which the defendant was denied access to the parameters of the risk assessment algorithm COMPAS owing to trade secrets.Footnote ³⁴ In this section, we demonstrate that the answer is more nuanced: while trade secrets might create barriers to transparent FRT in some situations (‘actual conflict’ situations), they are unlikely to interfere with transparency needs in other situations (‘no conflict’ and ‘nominal conflict’ situations).

4.4 Does Trade Secret Law Provide Adequate Solutions?

Trade secret law provides certain limitations that are meant to serve the interests of the public. Namely, in common law jurisdictions, when a breach of confidentiality is claimed, the defendant could raise a so-called public interest defence. In short, it allows defendants to avoid liability for disclosing a trade secret if they can prove the disclosure was in the public interest.Footnote ⁵⁵ As explained by the House of Lords, protection of confidential information is based on the public interest in maintaining confidences, but the public interest sometimes favours disclosure rather than secrecy.Footnote ⁵⁶ However, this public interest defence is of limited, if any, use in addressing the conflict between trade secrets and the legitimate transparency needs of identified stakeholders in an FRT scenario.

First, the scope of this defence is unclear.Footnote ⁵⁷ Some judicial sources suggest the existence of a broad public interest defence, which is based upon freedom of the press and the public’s right to know the truth.Footnote ⁵⁸ Other court judgments suggest that the defence should encompass no more than an application of the general equitable defence of clean hands, namely the information that exposes a serious wrongdoing of the plaintiff should not be classified as confidential in any case (iniquity rule).Footnote ⁵⁹ For instance, Australian courts have confirmed that disclosure in the public interest should be construed narrowly; it should limited to information affecting national security, concerning breach of law, fraud, or otherwise destructive to the public, and must be more than simply the public’s interest in the truth being told.Footnote ⁶⁰

Most importantly, the defence does not provide interested stakeholders with an active right to request information about the FRT technology and its parameters. It is merely a passive defence that could be invoked by a defendant only after they have disclosed the information (or where there is an imminent threat of such a disclosure). In order to disclose the information, the defendant should already have access to the information, which is not the situation of law enforcement authorities or public interest organisations seeking information about the FRT.

The public interest defence could be possibly useful in some exceptional situations. For instance, the employee/contractor of an FRT developer might disclose certain confidential technical information about the FRT system with the public or a specific stakeholder (public authority, NGO, etc.) in order to demonstrate that the AI developer did not comply with legal requirements when developing the FRT system and/or misled the public and/or the government authority as to the accuracy of the FRT technology, for example. If breach of confidence is claimed against this person, they could argue that the disclosure served the public interest: the use of an FRT system that is of low quality or biased may lead to incorrect identification of individuals, especially ethnic or gender minorities, which may further result in the arrest of innocent people and violation of their human rights. The defendant could argue that the disclosure of technical information about such an FRT system would thus help prevent harm from occurring.

Even then, the ability of a defendant to rely on the public interest defence is questionable. For instance, the court might accept the defence if the information is disclosed to government authorities responsible for prosecuting breaches of law or fraud, as ‘proper authorities’ for public disclosure purposes,Footnote ⁶¹ but not to public interest organisations or the public generally.Footnote ⁶² While the law enforcement authority (which is also the user of FRT in this case) might qualify as a ‘proper authority’, a public interest organisation is unlikely to meet this criterion.

Furthermore, if a narrow interpretation of the public interest defence is applied, the defendant would have to prove that the disclosed information relates to ‘misdeeds of a serious nature and importance to the country’.Footnote ⁶³ It is questionable whether a low quality or biased FRT, or the AI developer hiding information about this, would qualify as a misdeed of such serious nature. More problematically, the defendant might not know whether the FRT does not meet certain industry or legal standards until the technical information is disclosed and an independent examination is carried out.

4.5 Conclusions

It is without doubt that transparency is needed around the development, functioning, and use of FRT in the law enforcement sector. The analysis here has shown that in some cases trade secrets do not impede the transparency around FRT needed by some stakeholders (e.g., affected individuals or direct users of FRT) and some possible conflicts could be resolved through existing arrangements and laws (e.g., with relation to the transparency needs of certification and auditing organisations, and court participants). However, trade secrets might conflict with the transparency needs of some stakeholders, especially law enforcement authorities (after acquiring the technology) and public interest organisations that might want access to confidential technical information to assess the quality of the FRT system. Unfortunately, trade secret law, with its unclear and limited public interest exception, is unable to address this conflict. Further research is needed as to how the balance between the proprietary interests of AI developers and transparency needs of other stakeholders (law enforcement authorities and public interest organisations) could be established.

5.1 Introduction

‘Privacy’ has long been central to understanding the impacts of facial recognition and related technologies. Privacy informs the intuitions, harms, and legal regimes that frame these technological systems. Privacy and data protection law already have a ready-at-hand toolkit for related practices such as closed-circuit television (CCTV) in public space, surreptitious photography, and biometric data processing. These regimes measure facial recognition applications against familiar privacy and data protection categories such as proportionality, necessity, and legality, as well as identifiability and consent. But as facial recognition becomes more widespread and diverse, and the tools, ecosystems, and supply chains for facial recognition become more visible and better understood, these privacy and data protection concepts are becoming more difficult to consistently apply.

For as long as privacy has been deployed to constrain facial recognition, analysts have been decrying its inadequacy. This research typically identifies some novel dimension of harm associated with facial recognition that evades existing regulatory strategies. This chapter proposes an alternate diagnosis for why privacy fails to deliver premised on the nature of facial recognition as a broader socio-technical system. The jurisprudence shows that privacy and data protection function as intended at the level of ‘applications’ such as one-to-one and one-to-many identification and identity verification systems. But emerging cases show how privacy concepts become awkward and even incoherent when addressing different dimensions of the facial recognition ecosystem – at the level of ‘tools’ and supply chains, such as biometric image search engines and the production of facial image datasets. Inconsistencies in how law connects to this part of the facial recognition ecosystem challenge the suitability of regulatory concepts like identifiability and consent, the nature of harm being addressed, and perhaps most fundamentally, how privacy conceptualises the nature of online images. New rules for facial recognition products and applications are being included in the in the risk-based regulatory regimes for artificial intelligence (AI) in development around the world. In the EU, these include prohibitions on untargeted scraping of facial images from the internet or CCTV footage to create facial recognition databases. But as described below, the industrial organisation of the facial dataset business will continue to thwart these regulatory efforts, and privacy and data protection will continue to be legal bases for litigation against companies using facial recognition today and in the future.

This chapter offers an account as to why privacy concepts lose traction in this arena. It argues that existing regulatory approaches reflect an understanding of images as primarily ‘representational’, whereas facial recognition demonstrates that online images are better understood as ‘operational’ or ‘operative’. The operational image does not simply represent a referent but actively enables and participates in a sequence of automated operations. These operations take place at the level of facial recognition supply chains, where existing law struggles to find traction. Law’s inability to come to terms with the operational image pushes existing legal categories to the limits of their utility.

5.2 Facial Analysis and Identification

Privacy law and emerging AI regulations have effectively addressed the ‘watch list’ type facial recognition applications that come up in human rights litigation. For instance, the 2020 Bridges v. South Wales Police decision found the South Wales Police (SWP) force’s use of facial recognition in public to identify individuals on a watch list was a violation of Article 8 of the European Convention on Human Rights (ECHR).Footnote ¹ SWP deployed their surveillance system at large public events, using CCTV towers that collected footage of individuals in public, and performed real-time facial recognition against a database of persons of interest. Despite legislation allowing for the creation of that watchlist, the exact parameters for inclusion were not clear. The practice violated the ECHR Article 8 because, while proportionate and strictly necessary for the law enforcement purpose for which it was deployed, it failed to be ‘in accordance with the law’ in certain respects. Specifically, the enabling legislation and applicable Codes of Conduct failed to adequately specify rules around who could be the subject of surveillance (i.e., who could be placed on a watch list in the first place), or where facial recognition systems could be deployed. The enabling law thus gave police too much discretion. These issues have also clearly informed the regulation of biometric identification by law enforcement in the EU AI Act.

But the Bridges case also highlighted some conceptual issues of interest to the argument made in this chapter. In particular, the court’s conceptualisation of facial recognition as something different from both (1) police taking photographs of people in public and (2) the collection of biometric data such as fingerprints.Footnote ² Facial recognition occupied a place somewhere between the two in terms of level of intrusion, generating some conceptual discomfort for privacy. And while this was ultimately of little consequence to the court’s decision, with facial recognition easily enough absorbed into a human rights proportionality analysis without having to delve deeper into facial recognition’s ‘in-between’ character, the inability to analogise with existing police techniques for this in-betweenness was not merely a matter of novelty. This type of watch list surveillance and associated photography, including real-time (non-automated) identification, has been practised by police for decades. But facial recognition’s in-between character reflected something more fundamental about the media system that automates the identification task – its operationalism.

The argument made here is that facial recognition and related techniques are a function of the operational image.Footnote ³ The central insight of operationalism is that the ontology of images has shifted from one of representation to that of an element in a sequence of operations that are typically machine executed. Mark Andrejevic and Zala Volcic, for instance, describe the ‘operational enclosure’ through which the operational image includes automated identification, social sorting, decision-making, and responses that enable the governance of space.Footnote ⁴ Their basic example is facial recognition in retail stores that, when identifying a person on a watch list, not only calls security, but also actively locks the doors. This example also exemplifies Trevor Paglen’s emphasis that the audience for (operational) images is no longer humans but rather machines.Footnote ⁵

The operational image reconfigures images as the communicative instruments of automated non-human visuality. Images consumed by humans are increasingly the output of machines staging what they ‘see’ as a derivative function. But the primary audience of an image is a complex network of machines, with human-legibility a trivial or arbitrary secondary process. As Andrejevic and Volcic note, ‘In the case of facial recognition technology, there is, still, a camera with a lens, but for the purposes of recognition and response no image need be produced.’Footnote ⁶ The operational function of an image in the facial recognition context is, on the one hand, its capacity to communicate biometric information to other machines, which can then trigger various actions as described by Andrejevic and Volcic. On the other hand, facial images themselves have become operational through their absorption into an ecosystem and economy of image databases, search engines, and AI model training and benchmarking. In other words, online images are operationalised by the biometric supply chain. This additional operational character is revealed through the existence of companies and tools like Clearview AI, as well as the proliferating number of massive image datasets built from web-scraping and surreptitious public photography.Footnote ⁷

Privacy and data protection law struggle to accommodate this theorisation of images and this domain of economic activity. For instance, the operational image ontology suggests images are always already enrolled in a biometric recognition process. Privacy and data protection law, however, understand images as ‘representations’ of a referent, amenable to subsequent human interpretation and inference. Under the GDPR, for example, images are only considered biometric data after ‘specific technical processing’ that renders it comprehensible to a machine.Footnote ⁸ In other words, privacy and data protection law insist on the separation of images and any biometric information that can be derived from them.Footnote ⁹ This means images alone cannot be biometric data. Various authors have pointed out that this is contrary to technical understandings of biometrics,Footnote ¹⁰ which would conceptualise every image as also a biometric sample, and the beginning of a biometric ‘operation’. And as discussed Section 5.4.1.1, companies such as Clearview AI are exposing that a degree of processing of images, even if simply for aggregation in datasets, is already the default status of images online.

The following sections describe the different treatment of image and biometric data in existing law, with a focus on how the operational character of images expresses itself as conceptual confusion in how privacy addresses the tools and supply chains that make up the facial recognition ecosystem.

5.3 What Kind of Data Is That?

5.4 Representationalism versus Operationalism in the Case Law

5.5 Conclusion

The way privacy and data protection are configured may make sense if online images are representations of individuals, browsed by humans, at risk of certain autonomy effects; but it makes much less sense if images are already part of a socio-technical ecosystem, viewed primarily by machines, used to train and benchmark facial recognition algorithms in order to produce economic value. Privacy and data protection’s representationalism struggles to grasp the mobilisation of images as supply chain components in a dynamic biometric ecology. This chapter has argued that the issues in this ‘back end’ of the facial recognition ecosystem are very different from those that have been typically raised in privacy discussions. Here, regulatory questions intersect with what has become a new frontier of value creation in the digital economy – facial recognition model training. The concern is no longer exclusively losing anonymity in public, but also information being captured from public spaces, not for the sake of identifying you, but for the sake of generating an archive of images of you in the wild in order to train facial recognition models and extract economic value. Once we pay attention to how facial recognition systems are built and function, privacy and data protection start to lose their grip.

6.1 Introduction

Facial recognition technology (FRT) is one of several data-based technologies contributing to a shift in the criminal justice system, and society more broadly, towards ‘automated’ decision-making processes. Related technologies include other forms of biometric identification and predictive policing tools. These technology-based applications can potentially improve investigative efficiency but raise questions about bias and discrimination.Footnote ¹ It is important for designers of these systems to understand the potential for technology to operate as a tool that may can discriminate, furthering biases that are already entrenched in the criminal justice system.

This chapter examines how FRT contributes to racial discrimination in the criminal justice system, potentially exacerbating existing over-representation of racial minorities. From one perspective, this technology may be viewed by some as a value neutral, objective, decision-making tool, free from human prejudice and error. However, it is also recognised that FRT, and the associated algorithms, are dependent on datasets that influence its performance and accuracy.Footnote ² If the input data is biased, so too is the algorithm, and consequently the eventual decisions and outputs. Moreover, this discriminatory potential inherent in the technology is compounded by existing discrimination and over-representation of minority groups.

The chapter is divided into four parts: the first discusses FRT, including current applications. The second discusses the potential for bias and discrimination in the criminal justice system in relation to FRT. The third moves away from a focus on technology and considers social and structural discrimination, integrating the relevant critical literature into our argument. Finally, we conclude that even if the technology could be designed in a way that was completely free from discrimination in a techno-determinist sense, it may still be used to discriminate, given, for example, the long-standing over-policing and disproportionate representation of marginalised groups in the criminal justice system. This should be considered by governments when regulating FRT and by law enforcement and judicial officers making decisions that are informed by it.

6.2 Facial Recognition Applications and Issues

The face is central to an individual’s identity and, consequently, to identifying suspects in criminal investigations. The analysis of faces by law enforcement has progressed from descriptions and sketches of suspects to the contemporary biometric integrated closed-circuit television (CCTV) technology widely used around the world in both the public and private sectors today.Footnote ³ Although there are many applications of FRT, its fundamental process remains the same. FRT involves the automated extraction, digitisation, and comparison of the geometric distribution of facial features in a way that can identify individuals. It begins with a digital image of a subject’s face, from which a contour map of their features is created and then converted into a digital template. An algorithm compares digital templates of facial images and ranks them according to similarity.Footnote ⁴

There are two ways in which FRT is used. The first, and less controversial, is one-to-one matching. It is used to verify the identity of a person; for example, in a security feature granting access to a smartphone or to compare a person at an international border. The use of FRT expanded rapidly following the 9/11 terrorist attacks in 2001, when it was widely integrated into passports and international border control security systems, allowing the comparison of a facial template with a live image created using SmartGate technology.Footnote ⁵

The second way it can be used is one-to-many searching: the focus of this chapter. One-to-many searching seeks to identify an unknown person, for example by scanning CCTV footage of a crowd or images gathered from social media sites or more widely on the internet. Police could search based on a photograph of an unknown suspect to identify them or search for a known person in a crowd in real time. The integration of FRT with CCTV to identify unknown persons in public spaces is a major change that has taken place progressively over the past twenty years, to the point where it is normalised and widely used today. Examples of this type of application include not only fixed cameras, but also cameras on vehicles, body worn cameras, and drones, to search public spaces for persons of interest using integrated FRT.Footnote ⁶

More recently, FRT has been used to search images from the internet, including images uploaded to social media, from sites such as Twitter, Instagram, LinkedIn, Google, and Facebook. Facebook alone has over 250 billion images uploaded.Footnote ⁷ The use of Clearview AI by law enforcement agencies around the world came to light in 2020, and the company has been the subject of public debate and controversy, not least from social media and other internet companies that commenced legal action over the right to use these images. They claim its business model is in contravention of the terms of service of the websites the images were harvested from. In addition to the widespread use of the Clearview AI application by law enforcement agencies, the company also provides its services to the private sector, raising broader concerns. Clients that use the company’s services for security purposes include the National Basketball Association, Bank of America, and Best Buy.Footnote ⁸ The use of images from the internet demonstrates how facial templates can be collected and used in ways that individuals may not be aware of and has the potential to connect many sources of data. It also provides insights into the scale of use of FRT, adding to the significance of racial discrimination and other pertinent issues in this context.Footnote ⁹

There are inherent limitations in the use of FRT when deployed for the purposes of one-to-many identification that extend beyond bias and discrimination. Accuracy is impacted by factors such as the quality of images and cameras used, and the background and lighting conditions when the images were taken. Individual changes can impact on accuracy, including plastic surgery, ageing, weight gain, and facial coverings, such as the surgical masks that became commonplace during the COVID-19 pandemic.Footnote ¹⁰ In 2020, technology companies including IBM, Amazon, and Microsoft announced they would pause (or cease altogether), sales of their FRT to law enforcement and border security agencies owing to concerns around accuracy and privacy (Clearview AI was a notable exception to this position).Footnote ¹¹ There have also been bans by some local governments in the United States – Somerville, Massachusetts, and San Francisco, California – which have outlawed any city department, including law enforcement, from using FRT.Footnote ¹²

FRT has been found to be less accurate when used for the purposes of identifying people with darker skin tones, meaning that police deployment of FRT in criminal investigations can increase the likelihood that ethnic minorities will be wrongfully identified and prosecuted for crimes that they have not committed.Footnote ¹³ If this is not considered and addressed, it will likely increase the interaction of these individuals with police and compound their existing over-representation in the criminal justice system.

The issues we have raised in relation to racial discrimination cannot be viewed in isolation. In liberal democracies, there is ongoing tension between security, individual privacy, autonomy, and democratic accountability. The rapid growth and application of FRT in both the private and public sectors creates a power imbalance between individuals and the state (and corporations) and should be limited to specific and justified purposes (i.e., where the use of FRT is deemed to be both necessary and proportionate), with associated data and images carefully protected. As far as FRT being justified for security purposes, and privacy concerns mitigated, it must be subject to accountability mechanisms to prevent it being misused. Moreover, citizens should be informed about the potential use of their images for facial recognition and should have meaningfully consented to their use. Whether these systems are operated by public or private sector agencies or law enforcement, regulatory options should be publicly debated, and their use governed by legislation and subject to judicial review.

6.3 Data, Bias, and Racial Discrimination

In 2020, a police investigation in Detroit involving Robert Williams received attention in the national press in the United States. Williams, an African American man, was arrested for shoplifting based on facial recognition identification. He was held for thirty hours before posting bail; but it was later established to be a false match based on his driver’s licence photograph and distorted crime scene surveillance footage. The police department provided an apology and instigated a review of the use of FRT. Williams commenced litigation against the police department seeking compensation for his treatment.Footnote ¹⁴ The incident highlights the risks of inaccurate technology being used to identify suspects and relied upon in an arrest. Williams’s case is one of several similar examples from across the United States that has drawn attention to the potential for racial bias to occur in relation to facial recognition, and for this to exacerbate the over-representation of minorities.

These incidents took place around the same time as the murder of George Floyd by a police officer, and the subsequent attention on the issue of racial discrimination through the Black Lives Matter movement.

The existing over-representation of minority groups in police databases will mean that they are more likely to be identified using facial recognition. Brian Jefferson notes that in the United States more than three-quarters of the black male population is listed in criminal justice databases.Footnote ¹⁵ Because facial images are included in these databases, they can also be used by analysis by FRT. Depending on the specific use cases (i.e., how the technology is deployed and the watchlists used), it is reasonable to suggest that FRT directs police towards those individuals who are already known to them.

There are also data-based reasons why minority groups may be subjected to mis-identification, or over-identification, in relation to FRT, as established by empirical studies on the issue of racial bias associated with FRT. In 2019, a National Institute of Standards and Technology (NIST) report indicated that the technology achieved significantly lower rates of accuracy in African American and Asian faces – in fact, it found that faces of these races were between 10 and 100 times more likely to be mis-identified, when compared with white male faces.Footnote ¹⁶ This is supported by other research which has found that the mis-identification rate for dark-skinned women is about 35 per cent, fifty times higher than white males.Footnote ¹⁷

The reason for this rate of mis-identification is the data inputs that the algorithms undertaking the matching rely upon. It has been established that, on average, the datasets used to train the algorithms comprise approximately 80 per cent ‘lighter skinned’ subjects.Footnote ¹⁸ The issues with accuracy are therefore likely to be caused by ethnic representation in datasets used to create and train the matching algorithms. Designers of the technology need to consider the racial representation in the datasets used to train facial recognition algorithms. Failing to rectify this issue, by not proactively taking steps to include representative representation in the FRT datasets, could constitute a form of racism, whether that is intended or an oversight.Footnote ¹⁹

This is especially concerning given that ethnic minorities are already disproportionately scrutinised by law enforcement and over-represented in the criminal justice system. Increased error rates and mis-identification by facial recognition and other new technologies may compound this serious existing problem. This should be a focus for those building facial recognition systems – designing out the potential for racial discrimination by embedding racial equality in the data used to train the algorithms. Beyond this issue, any form of identification technology should not be relied upon in isolation, but only ever used in the context of other circumstantial evidence in an investigation. However, addressing the technology will only ever be part of the solution. As Damien Patrick Williams notes, ‘merely putting more Black faces in the training data will not change the fact that, at base, these systems themselves will be most often deployed within a framework of racialised and gendered carceral justice’.Footnote ²⁰

6.4 Social and Structural Discrimination

Police attention is not equally applied across the population; racial minorities are subject to disproportionate criminal justice system intervention. The consequences of this are most clearly seen in the disproportionate over-representation of minority groups in prisons around the world. This context is a necessary consideration when thinking about FRT and discrimination, because as we have described, technology can potentially perpetuate racial inequality. The following part of this chapter moves on from technical or technologically deterministic sources of bias and discrimination introduced above (i.e., those within the data or algorithms under-pinning the technology) and adopts a broader structural and social view. It considers facial recognition as a socio-technical phenomenon and argues there is a need to dis-aggregate the technical and social dimensions to discrimination, as well as understand their interaction, and to do so it is necessary to clearly define and evaluate the use cases of technology vis-à-vis specific social and institutional contexts.

It has been recently argued that ‘assisted’ (rather than ‘automated’) facial recognition is a more suitable descriptor for the technology given the way that it is used to inform and direct police activities and operations (rather than truly ‘automate’ them).Footnote ²¹ Pete Fussey and colleagues’ research examines a range of organisational, system, and operator factors, including the processes of human–computer interaction, and demonstrates how technical and environmental influences impact on the operation of facial recognition systems deployed by police. Fussey argues that ‘while practitioners shape and condition the application and potential of their technological instruments, these practices, forms of action and ways of thinking are simultaneously shaped and conditioned by these technologies and the affordances they bring’.Footnote ²² They conclude that ‘operator decision-making activities involving discretionary and suspicious judgements over who should be stopped once a possible identification has been articulated by the algorithm’ and that ‘technological capability is conditioned by police discretion, but police discretion itself is also contingent on the operational and technical environment’.Footnote ²³ These are important considerations, because the roots of discrimination in policing do not stem entirely from the use of new technology in and of itself, but rather the institutions of policing and the actions of police officers in discretionary and discriminatory enforcement of the law.

Work by Simon Egbert and Monique Mann on discrimination and predictive policing technologies also draws attention to the socio-technical interactions between the inputs/outputs of predictive technologies and the street level decisions made by police.Footnote ²⁴ Egbert and Mann argue that predictive policing is ‘a socio-technical assemblage, encompassing not only the technical predictions themselves, but also the enactment of the predictions on the street level police – which can also have serious ramifications including discrimination’.Footnote ²⁵ Connecting this argument to the work by Fussey, we argue that like predictive policing technologies, facial recognition technologies operate within a wider socio-technical assemblage that is shaped by the technology and wider social and structural factors such as police discretion and long-standing discrimination by police and criminal justice institutions. We contend that more attention needs to be directed to the social and structural contexts of technologies to understand their discriminatory potential when examining discrimination in policing, including in the application and use of facial recognition technologies.

Even if FRT could be designed to be perfectly ‘bias free’ from a technological perspective, it may still be targeted specifically against racial minorities or deployed in contexts that control and oppress them. An example of the relevance of such contextual considerations in which technology is deployed with discriminatory potential and impacts are the Smart City developments in Darwin, Australia. Pat O’Malley and Gavin Smith examine the deployment of this programme to improve public safety and public spaces, which involved the deployment of an extensive network of CCTV cameras.Footnote ²⁶ While administrators assert that the video analytics do not include facial recognition software, there is nothing to prevent police from using facial recognition software on the CCTV footage collected. This is significant given the stark over-representation of Indigenous people in the criminal justice system in this part of Australia. For example, in 2016–2017, Indigenous people comprised 84 per cent of the prison population, and Indigenous youth comprised almost 95 per cent of those in youth detention, in addition to many other forms of disadvantage demonstrating social-economic inequality and injustice.

O’Malley and Smith argue that the Smart City technologies deployed in Darwin are ‘directed at the monitoring and control of [Indigenous] people in public places’ and draw attention to the ‘very real prospect of the system being used to sharpen a criminalising gaze on the predominantly marginalised and excluded bodies of the Indigenous people living in and around the city’.Footnote ²⁷ The risk is that the surveillant capabilities of the Smart City in Darwin will create negative and disproportionate impacts for Indigenous people, not only because they are already the focus on a racialized criminal justice system, but also by virtue of their daily presence in public spaces in Darwin, which is connected to social factors including unemployment and homelessness, which is in turn a consequence of Australia’s colonial past and the dispossession of Indigenous people from their lands. O’Malley and Smith conclude that ‘the impacts of Smart City programmes on crime control cannot be read off in a technocratically deterministic fashion … but must be situated and analysed in specific contexts’ and that the ‘enduring legacies of colonialism have done much to shape the nature and implications of Smart Cities projects’.Footnote ²⁸

This demonstrates the importance of a focus on social, political, and historical context when thinking about how technology might be ‘biased’ or ‘discriminatory’, and the need to understand the specific use cases of policing technologies, including but not limited to FRT. Even if technologically ‘bias free’ forms of facial recognition were indeed available, we could assume that they will be deployed in ways that are not ‘neutral’ and, rather, would operate to further marginalise, discriminate against, and control certain groups, especially those that are already the most marginalised and oppressed. This is pertinent given critiques by Sara Yates that ‘the narrative that [FRTs] are problematic only due to their lack of transparency and inaccuracy is faulty’.Footnote ²⁹ Yates argues that ‘if these tools are allowed to be used by law enforcement, whether they have been reformed to address the accuracy and transparency issues … they will still be used disproportionally against marginalized groups and people of colour…’.Footnote ³⁰ A focus on addressing discrimination in FRT through only technologically deterministic approaches will not remedy broader historical social injustices and harm done by police institutions and the criminal justice system, nor will banning or outlawing facial recognition. As Yates acknowledges, ‘the greatest harm from these systems does not come from these tools themselves, but instead from the unjust institutions that use them’.Footnote ³¹ While calling for bans on FRT may be intuitively appealing, they will not resolve institutional and systemic racism and injustices perpetrated by such institutions.

The task must be to first address these fundamental injustices, or they will recur in the guise of objective technology.Footnote ³² There is a need to disaggregate the technical and social dimensions to bias and discrimination and seek to better understand the specific use cases of technology within specific institutional and social contexts. It is necessary to understand these various sources of bias and discrimination, for example those that arise from individuals (i.e., police/operator discretion), the way the system is designed (i.e., in public places that racial minorities tend to frequent), and the wider system objectives (i.e., the reason supporting the deployment of technology in that context). Analyses of the interactive effect of social and technological factors are required in order to evaluate whether the objectives and applications of certain technologies in specific contexts are necessary and proportionate, while ensuring that individual rights are upheld (including privacy, anti-discrimination, and equality). Regulatory strategies to address this issue could be targeted according to the level of risk presented in specific contexts and specific use cases of technology. Moving forward, there is a need to consider, implement, and evaluate measures that aim to reduce discrimination and harm in existing systems (including the criminal justice system) and design better systems. In doing so, the structural discrimination that is a feature of many systems must be addressed to ensure that existing inequalities are not perpetuated by new technologies such as facial recognition.

6.5 Conclusion

The use of FRT in the criminal justice system and its association with racial discrimination is an important issue for society, given the rapidly expanding application of the technology and the limited regulation in many jurisdictions. This technology may operate to further historical forms of oppression, discrimination, bias, and over-representation of minority groups in the criminal justice system. There is evidence that FRT may contribute to racial discrimination by operating with reduced accuracy, owing to the fact that the data used to inform the operation of the technology does not include sufficient representation, leading to inaccuracy and mis-identification. While this issue must be dealt with, addressing it in isolation will not be sufficient. The disproportionate focus on minorities is a far bigger problem in the criminal justice system, and the extent to which FRT perpetuates this is a subset of a much bigger, complex and historically entrenched problem. Along with the data problem, this context must be considered by those operating the technology, and by law enforcement organisations and governments, and they should not over-deploy it in areas where these minority groups are concentrated.

Rather than ban the technology altogether, we need to focus on structural discrimination and inequality – calling for a widespread ban of technologies altogether, while it may be appealing to some, is not going to be productive in the long term, nor is it realistic. While there are data-based issues here that can be addressed, this step alone will not be sufficient, and there is a need to address the social issues if we are to achieve meaningful change. Technology is not the problem, nor is it the solution. In conclusion, there are two perspectives to take account of: a data perspective and a social perspective. Although they are inter-related, they need to be disaggregated, and their socio-technical interaction better understood. First, we can see that when technology is based on datasets skewed towards white populations, it does not function as accurately on minorities. Second, technology may further existing bias and racism inherent in the individuals and organisations deploying and operating it, and in terms of inequality within the criminal justice system and society more broadly. We need to ensure that there is representative racial representation in datasets (the technical issue), and ensure that it is not over-used it in areas where racial minorities are concentrated (the social issue).

7.1 Introduction

In 2018, police in India reported that the roll out of facial recognition technology (FRT) across New Delhi enabled their identification of 3,000 missing children in just four days.Footnote ² In the United Kingdom, South Wales Police used live FRT to scan over 50,000 faces at various mass gatherings between January and August 2019 and identified nine individuals for arrest.Footnote ³ The Chinese Sharp Eyes programme, ‘omnipresent, fully networked, always working and fully controllable’, can take less than seven minutes to identify and facilitate apprehension of an individual among a population of nearly 5 million people.Footnote ⁴ In Moscow, 105,000 FRT-enabled cameras have monitored and enforced COVID-19 self-isolation orders,Footnote ⁵ with at least 200 violators being identified.Footnote ⁶

As protest movements are gaining momentum across the world, with Extinction Rebellion, Black Lives Matter, and strong pro-democracy protests in Chile and Hong Kong taking centre stage, many governments – both in the West and in the East – have significantly increased surveillance capacity of the public sphere. City streets and squares, stations, and airports across the globe, and social media and online platforms have become equipped with sophisticated surveillance tools, enabled and made legal through a myriad of complex and ever-expanding ‘emergency’ laws. Irrespective of whether these events and/or political strategies are framed as ‘emergencies’ such as the ‘war on terror’ with its invisible geopolitical enemies for 9/11, or whether they were pro-democracy or anti-racism protests or connected with COVID-19, the state resort to technology and increased surveillance as a tool to control the masses and population has been similar. Examples from varied countries – ranging from China, Russia, and India to the United States and the United Kingdom – tell us that recent technological advances have enabled authoritarian and democratic governments alike to build omnipresent biometric infrastructures that systematically monitor, surveil, predict, and regulate the behaviour of individual citizens, groups, or even entire populations. In this chapter, I focus on the chilling effect of FRT use in public spaces on the right to peaceful assembly and political protest. While technological tools have transformed protest movements widely, both amplifying and undermining them,Footnote ⁷ in this chapter I only focus how protest movements have been tackled with FRT and my emphasis is on political protests and public spaces. Pointing to the absence of oversight and accountability mechanisms on government use of FRT, the chapter demonstrates how FRT has significantly strengthened state power. It draws attention to the crucial role of tech companies in assisting governments in public space surveillance and curtailing protests. I argue for hard human rights obligations to bind these companies and governments, to ensure that political movements and protests can flourish in the post-COVID-19 world.

7.2 Undermining Protest Movements with FRTs

Live automated FRT, rolled out in public spaces and cities across the world, is transforming modern policing in liberal democracies and authoritarian regimes alike. The technology augments traditional surveillance methods by detecting and comparing a person’s eyes, nose, mouth, skin textures, and shadows to identify individuals.Footnote ⁸ The live automated facial recognition can instantaneously assess the facial biometric data in the captured images against a pre-existing ‘watchlist’ and flag it to police officers. Some FRT tools go further, purporting to classify people by gender or race or make predictions about their sexual orientation, emotions, and intent.

This FRT has been used to tackle protest movements globally. For example, the US company Geofeedia has been marketed to law enforcement ‘as a tool to monitor activists and protestors’,Footnote ⁹ incorporating FRT use with Twitter, Facebook, and Instagram databases.Footnote ¹⁰ Rasheed Shabazz, an activist and journalist, believes that his arrest near the Black Lives Matter protests in Oakland in 2014 was as a result of the Geofeedia software.Footnote ¹¹ This same software was also used to monitor civil unrest after the police killing of Freddie Grey and link protesters with their social media profiles.Footnote ¹² Similarly, in 2020, during the protests following the killing of George Floyd in Minneapolis, Minnesota, several people were arrested and charged after being identified through the use of FRT.Footnote ¹³ In another case, the Detroit Police Department used FRT to identify a Black Lives Matter protester who was arrested and charged with reckless driving and resisting arrest.

Similarly, FRT has been used in many other countries. For example, ‘habitual protesters’ in India are included in a dataset used to monitor large crowds,Footnote ¹⁴ which is composed of ‘miscreants who could raise slogans and banners’.Footnote ¹⁵ This database was used to identify dissidents at a prime ministerial rally in December 2019,Footnote ¹⁶ and also resulted in the detention of a ‘handful’ of individuals charged with violent crimes when it surveyed protests in New Delhi and Uttar Pradesh.Footnote ¹⁷ The Hong Kong police used FRT cameras to identify protesters and track their movements during the 2019 pro-democracy protests, which drew criticism from human rights advocates who argued that it violated the protesters’ right to privacy and could lead to their persecution.Footnote ¹⁸ In 2019–2020, FRT cameras were also used in Chile to monitor and identify protesters participating in demonstrations and civil unrest, known as the Estallido Social.Footnote ¹⁹ The cameras were installed in public areas, including train stations and street corners, by the Chilean government to track individuals who were suspected of participating in protests or other forms of civil disobedience. In the face of mounting criticism and protests against the use of this technology, the Chilean government announced that it would suspend the use of facial recognition cameras in public spaces in early 2020.

In all these cases, FRT allowed the authorities to quickly identify individuals who were wanted for questioning or arrest. The cameras were linked to a central database containing photos and personal information of individuals who were known to have participated in previous protests or other activities that the government deemed to be illegal. Such use of FRT cameras sparked controversy and concern among civil liberties groups and privacy advocates, who argued that the technology was being used to stifle dissent and violate the rights of protesters to peacefully assemble and express their opinions. Despite these concerns, governments typically defend FRT use by framing it as a necessary measure to maintain ‘public safety’ and order during a time of civil unrest.

In addition to such ‘top-down’ surveillance by public authorities in USA, India, Hong Kong, and Chile, ‘horizontal’ modes of surveillance have become increasingly popular.Footnote ²⁰ This involves partially outsourcing surveillance functions to individuals and/or tech companies. A vivid example of such outsourced surveillance was the 2020 Black Lives Matter protests in Dallas, during which the police department asked individuals on Twitter to send them videos from protests that showed ‘illegal activity’.Footnote ²¹ A larger-scale example was seen in the aftermath of the 2010 Canadian Winter Olympics riots, in which closed-circuit television (CCTV) footage was used to identify offenders, and private individuals sent the Vancouver Police Department thousands of images and helped them scour social media.Footnote ²² Similarly, tech companies such as Facebook, Twitter, and Instagram have been crucial in surveillance of protesters, as the widespread use of social media has made the monitoring of protest and dissident activities significantly easier.Footnote ²³ For example, in 2014 and 2016, the US government obtained two patents that may facilitate its ability to use social-media to predict when a protest will break out.Footnote ²⁴

Protest movements in USA, Hong Kong, Chile, and beyond have also operated in the shadow of the global COVID-19 pandemic, and together raised questions about unprecedented levels of government power and the expanding regime of mass surveillance in public spaces. The COVID-19 pandemic has given governments a further impetus to explore FRT’s health-related uses – from monitoring compliance with quarantine or social-distancing requirements to tracking (in conjunction with other biometric technologies such as thermal scanning) those who are potentially infected. COVID-19 and the latest protests in Hong Kong, Chile, and the United States have redefined the boundaries of mass surveillance and biometric tracking globally, with irreversible implications for the future exercise of government power and surveillance.

7.3 Lack of Regulation and Dangers of FRT

Despite the increasing deployment of FRT in many city squares and streets across the globe, as many chapters in this book demonstrate, FRT use is not yet regulated. Law enforcement agencies around the world are experimenting with FRT with discretion and on an ad hoc basis, without appropriate legal frameworks to govern its use nor sufficient oversight or public awareness.Footnote ²⁵ For example, there are currently no federal regulations in the United States governing the use of FRT by law enforcement.Footnote ²⁶ In March 2019, two US senators introduced the Commercial Facial Recognition Privacy Act, intended to ban developers and providers of commercial FRT from collecting and sharing data for identifying or tracking consumers without their consent.Footnote ²⁷ However, this only focussed on the commercial use of FRT. Similarly, in the EU, regulation of FRT has been very limited. In February 2020, a draft EU White Paper on Artificial Intelligence appeared to call for a discussion about a temporary five-year ban on facial recognition. However, the final draft of this paper removed mention of such a moratorium.Footnote ²⁸

This lack of oversight of FRT use by public bodies can lead to abuses of power and violations of fundamental rights and civil liberties. As many chapters in this book demonstrate, FRT use can result in discriminatory treatment and undermining of privacy and due process, as well as other concerns. Indeed, the dangers of FRT are gradually being recognised by courts. For example, law enforcement’s use of automated FRT was successfully challenged in 2020 in R (on the application of Bridges) v. Chief Constable of South Wales Police ([2020] EWCA Civ 1058) (‘Bridges’) case, where the Court of Appeal held that the use of automated FRT by South Wales Police was unlawful because it was not ‘in accordance with law’ for the purposes of Article 8 of the European Convention on Human Rights.Footnote ²⁹ In addition, South Wales Police had failed to carry out a proper Data Protection Impact Assessment and had not complied with the public sector equality duty.Footnote ³⁰ While Bridges is the first successful legal challenge to police use of automated FRT worldwide, fresh lawsuits brought by non-governmental organisations in the United States and France are still pending, and they might provide different judicial responses to regulation of police FRT use.Footnote ³¹

Some jurisdictions have already regulated and limited FRT use by law enforcement. In the United States, for example, the cities of San Francisco and Berkeley have banned local agencies (including transport authorities and law enforcement) from using FRT,Footnote ³² some municipalities in Massachusetts have banned government use of facial recognition data in their communities,Footnote ³³ and other US states (California, New Hampshire, and Oregon) have instituted bans on facial-recognition technology used in conjunction with police body cameras.Footnote ³⁴ The United Kingdom also has an Automated Facial Recognition Technology (Moratorium and Review) Bill,Footnote ³⁵ proposing to ban the use of technologies. Yet its future remains uncertain.

Therefore, not only civil right advocates, but also the courts and politicians widely recognise that FRT can be easily misused by law enforcement to target certain groups of people, such as political activists or marginalised communities, and such targeting often leads to further discrimination and injustice. Importantly, the growing prevalence of surveillance through FRT has a chilling effect on public discourse by threatening the right to protest anonymously; a notion fundamental to protest movements.

7.4 Protest Movements, Public Space, and the Importance of Anonymity

Protest movements are collective actions undertaken by a group of people who come together to express their dissent, raise awareness, and advocate for change around a particular issue or cause.Footnote ³⁶ These movements can take many different forms, ranging from peaceful demonstrations, marches, and rallies to civil disobedience, strikes, and other forms of non-violent resistance. Protest movements can emerge in response to a wide range of social, economic, political, and environmental issues. Some of the most common causes of protest movements include discrimination, injustice, corruption, inequality, environmental degradation, and war. Contemporary examples are Occupy Wall Street (2011), Arab Spring (began in 2010), Black Lives Matter (began in 2013), and the Hong Kong pro-democracy movement (began in 2019). Protest movements can also be motivated by a desire to promote social change, challenge existing power structures, and hold those in authority accountable for their actions.

Throughout history, protest movements have played a critical role in advancing social progress and promoting human rights. They have helped to raise awareness of important issues, mobilise public opinion, and influence policy and legislative changes. Examples of protest movements from history include the civil rights movement of the 1950s–1960s, the women’s suffrage movement of the late nineteenth and early twentieth centuries and Vietnam anti-war protests (1960s). Today, protest movements continue to be an important tool for promoting social change and advocating for a more just and equitable world.

Protests movements require a tangible and accessible location, typically in the streets and other public places. Public space has always been central to social movements and political protests as a practical place for citizens to gather and as a symbolic place connected to wider democratic values. It provides a physical location where individuals can come together to voice their dissent, express their grievances, and demand change.Footnote ³⁷ By occupying public spaces, protesters can create a visible and disruptive presence that draws attention to their cause, and can also serve as a symbolic representation of their struggle.

Public spaces, such as city squares, parks, and streets, are often central to the social and cultural life of a community, and their use for protests can be a powerful statement of the collective will of a group of people. Thus, public spaces are the ‘ultimate area of societal interaction’ and occupy a symbolic place in society given their accessibility, openness, and, according to Jens Kremer, inherent freedom.Footnote ³⁸ When protesters occupy public spaces, they are asserting their right to participate in the democratic process and to be heard by those in power. In interrupting these public spaces, protesters ‘touch upon the very core of the current structure and organization of social systems, namely the balance of power, rule of law and democratic governance’.Footnote ³⁹ It questions the ability of government authorities to maintain the integrity of these shared spaces,Footnote ⁴⁰ thus challenging existing power structures.

Historically, protesters have taken the right to protest anonymously largely for granted; a right that is now becoming increasingly more fragile. The right to anonymity has been fundamental to social movements and protesting, as these events require the population to feel confident and safe in their ability to gather in public spaces and manifest their disagreement with the status quo. This is impossible if they fear surveillance tools can be weaponised against them to suppress and punish their dissent. The sense of safety necessary to facilitate robust democratic participation stems from an understanding that an individual, in the act of demonstrating, is expressing something larger than themselves by joining in a collective. They thus sacrifice their individual voice for the benefit of social disruption, and in return are granted the key right that protesters have enjoyed for centuries; the right of anonymity. The anonymity earned by protesters in public spaces has been increasingly challenged and eroded by surveillance infrastructure.

While the relative anonymity of the individual during protest gatherings has typically ‘neutralised’ the effect of surveillance, they have been increasingly subject to ‘counter-neutralization technologies’ that require those individuals to take more active steps to circumvent identification.Footnote ⁴¹ Of course, protest movements have long devised resistance strategies against surveillance. For example, protesters can break a system a surveillance system by flooding it, rendering surveillance inoperable or impractical.Footnote ⁴² Typical examples include crude forms of neutralisation such as disabling phone lines, wearing masks, and destroying cameras. For example, Hong Kong protesters in 2019 used lasers and broke smart lampposts that they believed contained FRT software.Footnote ⁴³ With FRT, protestors are given two choices: first, they can wear a mask and risk arrest and the collection of biometric information in the form of criminal records, or second, they can do without a mask and risk collection of biometric data through FRTs.Footnote ⁴⁴

Surveillance technologies dealing with political protests have become the norm in many countries, and scholars have theorised about the chilling effect of surveillance on dissent.Footnote ⁴⁵ Monitoring, tracking, and detaining individual protesters for their actions in public places significantly shifts the power balance between the state and individuals. Surveillance of political protests undermines the individual as a ‘free autonomous citizen’, and negatively impacts democracy and the rule of law.Footnote ⁴⁶ Protestors become disempowered in relation to their body and biological information,Footnote ⁴⁷ which is not only threatening, in one sense, to discrete individuals,Footnote ⁴⁸ but in another to discretise protesters, breaking down their collective image. Pervasive surveillance tools can be understood as disciplinary, as they are able to threaten and realise retribution against individual protesters who would otherwise have been lost in a sea of voices, but it is in another sense indicative of a ‘controlled’ society in which surveillance is ubiquitous.Footnote ⁴⁹

7.5 Protecting Protesters from Abuse: Potential Ways to Regulate

Given the danger that FRT surveillance in public spaces poses to political protests, the rights to peaceful assembly and association, and wider democratic participation, legislatures should regulate or entirely ban the use of FRT in policing and law enforcement. Regulation of FRT use is a necessary step to ensure the chilling effect of FRT on political expression and freedom of assembly is eliminated.

The chilling effect on freedom of speech and assembly is even stronger in some jurisdictions, such as Australia. This is because, unlike many other jurisdictions discussed in this book, Australia has no human rights protection enshrined in its Constitution and no national human rights legislation.Footnote ⁵⁰ Only three out of eight Australian states and territories have state-level human rights Acts. For this reason, in its recent report, the Australian Human Rights Commission has urged Australia’s federal, state, and territory governments to enact legislation regulating FRT.Footnote ⁵¹

What are the ways to protect protesters and protest movements from abuse by public authorities? Recent literature related to AI and accountability has recommended several avenues, including regulation,Footnote ⁵² the development of technical methods of explanation,Footnote ⁵³ the promoting of auditing mechanisms,Footnote ⁵⁴ and the creation of standards of algorithmic accountability in public and private bodies.Footnote ⁵⁵ Law, of course, should also play a role.

7.6 Conclusion

If the government and law enforcement can resort to FRT without any restrictions or safeguards in place, the right to protest anonymously will be curtailed and political discourse in our democracies will be stifled. For example, the High Court of Australia – Australia’s apex court – has emphasised the centrality of the right to protest to Australian democracy: besides casting their vote in elections, Australians have no other avenues through which to voice their political views.Footnote ⁹³ Adapting Hannah Arendt’s famous quote used at the beginning of this chapter, political freedom must enable a right to participate in government. And in many instances, the only way to do that, in addition to voting, is through political protest.

Before FRTs develop further and become even more invasive, it is imperative that this public surveillance infrastructure is limited. We need laws restraining the use of FRT in our public spaces, and we need hard legal obligations for those who develop and supply law enforcement with them. The reforms could start with an explicit ban (or at least suspension) on police use of FRT in public spaces, pending independent scrutiny of the discriminatory impacts the technology may have against women and other protected groups.Footnote ⁹⁴ These proposed changes are not drastic. In fact, they are a modest first step in the long journey ahead to push back against escalating surveillance of the public sphere worldwide.

8.1 Introduction

Shortly after Russia launched large-scale military action on Ukraine on 24 February 2022, Clearview AI (a US-based facial recognition company) announced that it had given its technology to the Ukrainian government to be used for military purposes in Russia’s attack on Ukraine.Footnote ¹ In mid-March 2022, it was reported that Ukraine’s Ministry of Defence started using facial recognition technology (FRT).Footnote ² In such a way, simply and effectively, without long-lasting political debate and academic or civil society discussions, FRT was brought to a new profitable market and joined a list of tools that can be employed for military purposes.

While the Russian war against Ukraine is not the first time that FRT has been used in a military setting, this conflict has brought the military use of this technology to a different level: FRT was offered openly at the outset of the war to one of the sides, being promptly accepted by the Ukrainian authorities and tested on the ground for a variety of objectives. Before 2022, there was only minimal evidence of FRT employment for military purposes. One might recall that in 2019, Bellingcat, a Netherlands-based investigative journalism group specialising in fact-checking and open-source intelligence, used FRT to help identify a Russian man who had filmed the torture and killing of a prisoner in Syria,Footnote ³ or that in 2021, Clearview AI signed a contract with the Pentagon to explore putting its technology into augmented reality glasses.Footnote ⁴ It has also been reported that Israel performs surveillance of Palestinians using a facial recognition program.Footnote ⁵ However, these cases only provide evidence of incidental use or potential future application of FRT for military purposes.

This chapter discusses how FRT is employed by both sides in Russia’s war against Ukraine. In particular, it examines how Ukraine engages FRT in the country’s defence, testing the different possibilities this technology offers. It also acknowledges the use of FRT on the other side of the conflict, elaborating on how it is used in Russia to suppress society’s potential opposition to the war. The chapter focusses on the potential and risks of using FRT in a war situation. It discusses the advantages that FRT brings to both sides of the conflict and underlines the associated concerns.

8.2 FRT in the Battlefield: Ukraine

Ukraine began exploring the possibilities of FRT use in the military during the first month of the war. There was no time for elaborate learning or training, and the FRT was put directly into the battlefield, applying creative thinking and a trial-and-error approach. As a result, the Ukraine war can be seen as a field trial for FRT, where, faced with the pressing need to defend its territory and people, the country referred to collective efforts to generate ideas on innovative ways to employ modern technologies and is putting them into practice and checking what works well.

It should be admitted that the FRT developed by Clearview AI (perhaps the most famous and controversial facial recognition company) worked for the interests of Ukraine, owing to its enormous database of facial images. The company has harvested billions of photos from social media companies such as Facebook, Twitter, and Russian social media sites (Vkontakte).Footnote ⁶ Such a method of database creation attracted wide criticism in peacetime,Footnote ⁷ but proved beneficial in war, enabling access to facial images of Russian citizens, including soldiers.

One might think that collecting facial images of military personnel, especially those of a higher rank, from social networks might be a challenge – as they are less likely to reveal their identity. But this is not entirely true. While persons who post frequently on social media can be identified more easily, facial recognition systems can also identify and verify those who do not even have social accounts. It is enough that a family member, friend, co-worker, or person also serving in the army post a picture where the person is seen. The technology can even single out a face in a crowd picture and compare it with the face in question. Face recognition can also be used where only some people from a group (e.g., a military unit) have been identified, with the rest of the group being identified through content from any of the identified members. Even if a person appears on the internet in a randomly taken picture, it might also be helpful information allowing their gradual recognition.Footnote ⁸

Here, several ways in which the Ukrainian authorities employed FRT in its efforts to fight Russia are discussed. The author would like to note that the list might be not exhaustive as, at the time of writing (September 2022), the war continues, and part of the information remains undisclosed.

8.3 FRT in Russia: A Government’s Tool in Its Effort to Stifle Anti-War Protests

During the war against Ukraine, the FRT in Russia mainly, though not exclusively,Footnote ²⁵ served a different purpose – to stop any anti-war protests.Footnote ²⁶ While marches and mass rallies against Russia’s attack on Ukraine were taking place all over Europe, protests in Russia had been sporadic and small-scale. In Moscow, with a population of more than 12 million, the number of protesters never exceeded a few thousand. Large numbers of protesters were also never seen on the streets in other cities.

There are different reasons for this. On the one hand, the small number of Russians who expressly oppose Russian aggression might be interpreted to confirm overall society’s support for the current government, prevailing approval of the policies being pursued, and agreement with the arguments put forward by the authorities as to the validity and necessity of the ‘special operation’ (the term used in Russia to refer to the attack on Ukraine). This support arguably stems from strong influence of national mass media, general mentality, and from the iconisation of Russia as a superpower and even ‘true-values protector’,Footnote ²⁷ which has to be respected and abided by. On the other hand, owing to the massive arrests of protesters, those opposing the war see it as dangerous to protest.

From the very beginning of the invasion of Ukraine, Russian authorities effectively stopped any anti-war protest efforts. In addition to prohibiting protests against the Russian military attack on Ukraine and traditional street arrests, Russia employed FRT to track down and apprehend anti-war protesters. Analysis of online posts and social media reveals that Russian citizens are in no doubt that Big Brother is literally watching them, and that the FRTs used by the authorities in public spaces will prevent them from remaining unidentified and simply being part of the crowd.

According to Human Rights Watch, Russian authorities have been integrating public surveillance systems with FRT across the country and using these technologically advanced systems to identify and prosecute peaceful protesters since 2017.Footnote ²⁸ The authorities do not deny this information and do not comment on the details of the extent of use, thus reinforcing the deterrent effect.Footnote ²⁹ Already back in 2017, it was announced on the official website of the Mayor of Moscow that more than 3,500 cameras had been connected to the Joint Data Storage and Processing Centre, including more than 1,600 cameras in the entrances of residential buildings, with many closed-circuit television cameras in the city also reportedly being connected to a facial recognition system.Footnote ³⁰ Additional cameras were placed during later years, and after the start of the war, surveillance was increased in the places where protests typically take place.Footnote ³¹ The collection of biometric data also continues to be strengthened. For instance, in May 2022, Russian authorities demanded that the four largest state-owned banks hand over their clients’ biometrics to the government.Footnote ³² To ensure that the biometric data is collected from the practically entire adult population, the laws were amended in July 2022 to oblige banks and state agencies to enter their clients’ biometric data, including facial images and voice samples, into a central biometrics database. This measure, which does not require clients’ consent to share data with the government, came into force in March 2023.Footnote ³³

Human Rights Watch stated in its submission to the Human Rights Committee on Russia on 10 February 2022 that the use of FRT, including for police purposes, is not regulated by Russian law. It highlighted that such use in the context of peaceful protests contradicts the principle of legal certainty by interfering with the rights to liberty and security by using methods that are not adequately supervised or provided for by law. It also violates the rights to privacy and peaceful assembly and is used in a discriminatory manner on the basis of political opinion. Human Rights Watch suggested that the Committee urge the Russian government to end the use of facial recognition during peaceful protests and ensure that all government use of facial recognition is strictly regulated by law.Footnote ³⁴ It is not likely that Russia intends to implement this proposal in the near future, as the FRT has proved to be a powerful tool to control protests against the country’s policies.

8.4 Concerns Associated with the Use of FRT in Wartime

While there is no doubt that the FRT brings many advantages to both sides of the conflict, it also raises a number of concerns. The main ones are the possibility of false identification, misuse of FRT, and problems associated with its continued use after the war.

8.5 Concluding Remarks: FRT – A New Tool of Military Technology?

Any war is a tragedy for human society, but it also acts as a step in the further development of technologies. This is evident in the current Russian war against Ukraine. The conflict in Ukraine represents a coming of age for a number of advanced technologies, from drones to commercial satellites, loitering munitions, and FRT. As Lauren Kahn notes, Ukrainian steppes have been transformed into a proving ground for next-generation technologies and military innovations in this war.Footnote ⁴⁰

From the perspective of FRT companies, the contribution of FRT to the Ukrainian war effort, in terms of proving ground and use, yields valuable data and – at the same time – visibility and even an advertisement for FRT companies. It is also difficult to argue against the fact that offering FRT technology to Ukraine during the war was a wise choice because it created a chance for the technology to prove its worth. Likely, companies whose products are being deployed in this conflict (both in Ukraine and in Russia) in a short time will become defence contractors offering their FRT as military technology.

In such a way, the broad and effective deployment of modern tools in Ukraine in its efforts to stop Russia’s military invasion is bringing emerging technologies into the mainstream of the military. In an era where technology reigns, it comes as no surprise that artificial intelligence is being employed for military purposes. FRT is advancing and spreading, and it might safely be projected that FRT will be a well-established military – and propaganda – tool in a decade or two. While one can argue that bringing FRT to war is dangerous and should be avoided because of the associated risks, it would be naive to believe that this will not happen. What can be done, though, is to develop international standards on the accepted use of FRT for military purposes – work that awaits the international community in the near future.