State actors in Europe, in particular security authorities, are increasingly deploying biometric methods such as facial recognition for different purposes, especially in law enforcement, despite a lack of independent validation of the promised benefits to public safety and security. Although some rules such as the General Data Protection Regulation (GDPR) and the Law Enforcement Directive (LED) are in force, a concrete legal framework addressing the use of facial recognition technologies (FRTs) in the EU so far does not exist. Given the fact that the technology is processing extremely sensitive personal data, is not always working reliably, and is associated with risks of unfair discrimination, a general ban on any use of artificial intelligence (AI) for automated recognition of human features at least in publicly accessible spaces has been demanded.Footnote ¹ Against this background, this chapter adopts a fundamental rights perspective and examines whether and to what extent government use of FRT can be accepted under European law.

9.1 Government Use of Facial Recognition Technologies within the EU

The government use of FRT in the EU is limited so far. With Austria, Finland, France, Germany, Greece, Hungary, Italy, Latvia, Lithuania, Slovenia, and the Netherlands, eleven countries are already using FRT, with an upward trend, but the deployments are primarily experimental and localised so far.Footnote ² It is mainly used by security authorities for the purposes of prevention, investigation, detection, and prosecution of criminal offences as well as the prevention of threats to public security. In addition to the most controversial use of FRT for (mass) surveillance, especially in publicly accessible spaces, FRT is primarily used among law-enforcement agencies in the EU for the purposes of forensic authentication so far.Footnote ³ The typical scenario is to match the photograph of a suspect (e.g., extracted from previous records or closed-circuit television footage) against an existing database of known individuals (e.g., a national biometric database, a driver’s licence database) for ex-post identification in a criminal investigation. Finally, on grounds of efficiency, FRT is also increasingly used by law enforcement agencies as a tool for analysing large amounts of video footage, for instance to search for a specific person or to track a person over multiple videos, since a manual analysis can be very time and resource consuming.Footnote ⁴

9.2 Suitability Despite Accuracy Concerns

A ban on use of FRT for law enforcement purposes is still discussed under the recurring argument that the performance of such systems is not yet appropriate. The sufficient accuracy rates cannot be achieved in real life settings, the errors are unequally distributed in the referenced population, and minorities are discriminated against.Footnote ⁵

If methods or systems are not reliable and mistakes occur when implementing the method or the system in practice, its use by state authorities may be unlawful. Under European law, the exercise of recognised rights and freedoms can be limited only if such limiting measures are appropriate and necessary to achieve the objectives (see Art. 52 para. 1 EU Charter of Fundamental Rights (CFR)). If this is lacking, the measures are disproportionate and thus unlawful. However, the European Court of Justice (CJEU) allows a wide margin of assessment to the legislator to assess the suitability of the measure. Only if, having regard to its objective, the measure is manifestly inappropriate, can the legality of a measure be affected. As long as the objective is promoted in any way, the measure is presumed to be appropriate, even if the employed method is not wholly reliable. Hence, the CJEU assessed the storage of biometric fingerprints in passports and travel documents for the purpose of preventing illegal entry to the EU as generally suitable, despite a not inconsiderable error rate.Footnote ⁶ The court came to the same conclusion with relation to the automated analysis of passenger name records (PNR) for the purposes of preventing, detecting, investigating, and prosecuting terrorist offences and serious crime.Footnote ⁷ With regard to the automated matching of PNR data with patterns, which is comparable in its functioning to facial recognition systems, the CJEU stated that the possibility of ‘false negatives’ and the fairly substantial number of ‘false positives’ resulting from the use of the system may limit the appropriateness of the system. However, automated processing has indeed already made it possible to identify air passengers presenting a risk in the context of the fight against terrorist offences and serious crime; this is why the system is not inappropriate.Footnote ⁸ Moreover, according to the CJEU, the appropriateness of the system essentially depends on the proper functioning of the subsequent verification of the results obtained under those processing operations by non-automated means.Footnote ⁹

The FRT technologies have made real progress towards accuracy in the last years, owing to the use of convolutional neural networks. Despite this, the accuracy and error rates of FRT systems depend strongly on the task and the conditions under which the technology is used, as well as the quality of training and comparison data.Footnote ¹⁰ The one-to-one variant has become extremely accurate.Footnote ¹¹ It is used to confirm a person’s identity on the basis of clear reference images, such as recognising the rightful owner of a passport or smartphone (verification/authentication). On standard assessments such as the Facial Recognition Vendor Test (FRVT) of the National Institute of Standards and Technology (NIST), accuracy scores can be as high as 99.97 per cent.Footnote ¹² This is, with some reductions in accuracy, true even if the face is partially covered by a mask.Footnote ¹³ The reason for this is that the one-to-one variant is comparatively simple. In one-to-one situations, one typically deals with standardised images often produced under ideal conditions (e.g., consistency in lighting and positioning), which correspondingly leads to a lower number of inaccuracies.

The situation is quite different if FRT is used in the one-to-many variant,Footnote ¹⁴ which receives most attention in the debate. This variant serves to determine an unknown person’s identity by comparing a facial image with a large volume of known faces (identification). For example, it can be used to identify specific offenders or suspects or track down missing persons or victims of kidnapping.Footnote ¹⁵ Compared with the verification systems, the pictures of individuals used for identification purposes were usually captured remotely and in real life settings (‘in the wild’), where the subjects do not know they are being scanned. They may not be looking directly at the camera and/or may be obscured by objects or shadows. Accordingly, the accuracy rates tend to be far lower when compared with the controlled setting. For example, NIST’s FRVT found that, when using footage of passengers entering through boarding gates (a relatively controlled setting), the best FRT system had an accuracy rate of 94.4 per cent.Footnote ¹⁶ In contrast, leading algorithms identifying individuals walking through a sporting venue – a much more challenging environment – had accuracies ranging between 36 per cent and 87 per cent, depending on camera placement.Footnote ¹⁷ These different uses with a broad range of accuracy could cause fundamental rights concerns.

9.3 Fundamental Rights Concerns

The government use of FRT interferes with the European fundamental right guarantees. First of all, the initial video recording, the subsequent retention of the footage, and the comparing of the footage with database records for the purpose of identification (matching) constitutes an interference with the right to data protection, as set out in Article 8 CFR and Article 16 Treaty on the Functioning of the European Union (TFEU).Footnote ¹⁸ Both regulations ensure identical protection of personal data against processing, including in particular the image of a person recorded or, rather, the unique facial features extracted in a template. In addition, the right to private life implemented in Article 7 CFR and Article 8 European Convention on Human Rights (ECHR) might also be violated, depending on how and for what purpose the technology is used. Article 7 CFR protects privacy to ensure the development, without outside interference, of the personality of each individual in his relations with other human beings.Footnote ¹⁹ The protection guaranteed by Article 7 CFR and Article 8 ECHR extends primarily to private zones (a person’s home or private premises). However, there can also be interaction in a public context, which may fall within the scope of private life, when the person can have the reasonable expectation to be in private (e.g., private conversation in a screened area).Footnote ²⁰ Such expectation cannot exist in a public space where everyone is visible to any member of the public.Footnote ²¹

Accordingly, the use of FRT by authorities is not necessarily inconsistent with Article 7 CFR and the right to private life, as long as the video recording is made in a public space where one cannot expect to be in private and is used solely for the purpose of identification. This applies at least as long as the recording is not stored systematically and permanently.Footnote ²² If FRT is used to gain inferences about the person and their personality, for example, their behaviour, whereabouts, movement patterns, contacts, or personal characteristics such as sexual or political orientation, Article 7 CFR might be violated, as the respect of private life includes the protection of private information and free development of personality.Footnote ²³

Furthermore, depending on the task for which the technology is used, FRT may affect other fundamental rights. For instance, if authorities deploy facial recognition systems in the context of public protests, during the protest or ex-post,Footnote ²⁴ to identify participants or locate individuals suspected of offending, interference with the freedom of assembly according to Article 12 CFR and Article 11 ECHR comes into consideration.Footnote ²⁵ In addition, there might a violation of a person’s freedom of opinion and expression as guaranteed by Article 11 CFR. Assemblies and protests are legally protected as spaces for the collective expression of opinions. The use of FRT and the consequent possibility of identification and traceability may discourage individuals from exercising their right to freedom of peaceful assembly as well as their right to freedom of expression.Footnote ²⁶

9.4 Lawfulness under the European Rights Framework

These interferences with European fundamental rights do not make the government use of FRT generally inadmissible. Fundamental rights enshrined in the CFR and, in particular, the rights of data protection and private life, are not absolute rights; they must be considered in relation to their function in society and can be limited under certain circumstances.

9.5 Consequences for the Government Use of FRT

When trying to adopt these guidelines given by the European fundamental rights framework and the associated case-law concerning data retention in the use of FRT, it must be considered that facial recognition systems process data of similar or even higher sensitivity than traffic or location data. The processing of facial biometric data does not only enable the identification and verification of individuals. The systematic collection and evaluation of such data might – as described earlier – lead to conclusions about persons’ behaviour and whereabouts; apart from the fact that increasing attempts are being made to draw inferences about individual personal attributes from facial appearance, such as sexual or political orientation or violent tendencies.Footnote ⁵⁴ However, the face is a highly personal feature that cannot simply be amended, given that FRT even works if the face is partially covered.Footnote ⁵⁵ Accordingly, the existing rules addressing the processing of biometric data – the GDPR and LED – impose particularly high requirements on the processing of such data and only permit it for the prevention of threats to high-priority legal interests, including the prosecution of serious criminal offences.

The analysis shows that, despite the interference with fundamental rights such as privacy or data protection as well as possible high error rates, the European fundamental rights framework does not preclude government deployment of FRT in principle. However, a specific legal basis is required, defining clearly and precisely the purposes for which and by whom FRT can be used, who has access to the generated data, and how to proceed with the data once collected (e.g., retention and deletion periods). The law must not only consider the various applications and sectors where FRT can be used, but also address the different phases of the use, including the creation of a reference dataset and its deployment.Footnote ⁵⁶ Furthermore, safeguards against abuse and any external (unauthorised) use are needed as well.

A government use of FRT for general preventive and indiscriminate mass surveillance purposes, in which individuals are recorded without a reasonable suspicion, would not be compatible with the European fundamental rights framework. In particular, establishing a state-owned biometric reference database with face images of persons without any specific reason (e.g., in order to be able to easily identify individuals in the future), would be contrary to fundamental rights. It would be nothing else but general and indiscriminate data retention. Hence, only individuals who have given the authorities a reason to do so, because they are dangerous or are suspected of having committed a crime, for example, may be recorded in the reference database. A deployment of FRT in publicly accessible spaces can only be allowed if it serves to avert threats to high-priority legal interests or to prosecute serious criminal offences. Such deployment should be geographically limited to high-risk areas or to areas with high probability of locating wanted persons.Footnote ⁵⁷ This is likely to apply even if the facial image is deleted automatically and immediately after the comparison with the database is completed and no matches are found. The use of FRT systems is therefore conceivable, for instance, when tracking terrorists or serious criminals in highly frequented areas or strategic locations, such as airports, stations, or tollbooth areas. It could also be used for the surveillance of events or places where the risk of serious criminal offences is high. Moreover, the deployment of FRT may also be compatible with fundamental rights if it is used for ex-post identification of criminals, terrorists, or other persons of interest, or as a tool for the effective image and video evaluation (e.g., to recognise or track individuals in a video recording). Most importantly, the decisive factor for using FRT, which complies with fundamental rights, is that the incoming data should not be stored longer than necessary for the intended purposes and cannot be used for other purposes.

9.6 Conclusion

The analysis here leads to the conclusion that the government use of FRT can be permissible under the European fundamental rights framework if subjected to specific and strict conditions. In order to allow FRT use, a legislator should provide a specific legal basis regulating the deployment of FRT that is compatible with fundamental rights. In light of this, the EU AI-Act,Footnote ⁵⁸ which provides general limitations to FRT use,Footnote ⁵⁹ will not be sufficient as a legal basis, especially for the present main application of FRT by authorities: the prevention and prosecution of crime. There should be a legal basis directly legislated by the member states,Footnote ⁶⁰ as the protection of national security as well as law enforcement fall under the legislative competence of the states and not the EU. In addition, an empirical study of the real effectiveness of FRT would be sensible and desirable considering the fundamental rights violations before widespread use. So far, the advocates of this technology have failed to provide enough evidence to prove that this technology can genuinely ensure public safety and security.

10.1 Introduction

In the era of biometric mass surveillance, novel technological implementations have led to an unprecedented monitoring of sensitive data. Among other purposes, this data has been used to discriminate based on certain characteristics (from sex to ethnic or social origin), contrary to multiple protective declarations, or draw insights into people’s emotions. Such applications call for concrete regulatory intervention that is expressly targeted at practices that may interfere with fundamental human rights, including the right to privacy and personal data protection.

Despite promising initiatives, such as the European Citizens’ Initiative’s ‘Civil society initiative for a ban on biometric mass surveillance practices’, which was registered by the European Commission in 2021,Footnote ¹ regulators have failed to readily intervene (before the materialisation of the harm) with a view to banning, halting, or sanctioning certain intrusive practices. Although this failure might to some extent be justified by lengthy law-making procedures, there is an acute social need to protect people’s facial and other biometric data from constant watching by public or private actors, including for-profit firms, whose exercise of surveillance activities appears unregulated or under-regulated.

After discussing new challenging trends in the technological arena, this chapter emphasises the need for concrete rules surrounding specific technological uses and their possible harms. Technological uses (and misuses) can have a global reach, meaning they pose a global risk, with a potential for global harm that may affect numerous citizens simultaneously. Hence, there is a need for precise law-making and uniform enforcement – via joint-intervention and collaboration between regulatory entities around the globe – with a view to halting, banning, and sanctioning targeted practices interfering with fundamental human rights.

Section 10.2 discusses trends such as remote biometric surveillance, biometric monitoring targeted at classifying people on legally protected grounds, biometric processing drawing inferences on emotions or intentions, and traditional practices, such as closed-circuit television (CCTV) surveillance, whose regulation appears to require updating. It then makes the argument that these four trends must become a warning for regulators, because they have resulted in the emergence of new needs of the citizens.

Section 10.3 summarises findings of our comparative study of US initiatives that regulate facial recognition or biometric data processing. Relying on these initiatives, we highlight three regulatory building blocks for the EU. First, concreteness and precision of the law: US legal texts appear clear and expressly targeted at technological uses, vulnerable groups, or coercive state powers. Second, bright-line bans: the US prohibition-agenda includes moratoria and other techniques that may, in some instances, reach the level of unconditionality. Third, practical organisation of remedies: it is not only the civil/administrative route that citizens can follow; rather, many areas, from competition and market to criminal law, are combined to enhance effectiveness of protection.

Since the surveillance-effect appears ubiquitous and the technological reach seems transnational, the solution may lie not only in concrete law-making, but also in uniform or global enforcement. Section 10.4 discusses the 2021 Clearview-case to demonstrate that in this targeted case, joint scrutiny by different national entities and joint regulatory intervention (via rigorous investigations), had a positive effect and led to a considerable degree of enhanced protection for those affected by the firm’s mass surveillance practices. Section 10.5 summarises, comments, and makes more concrete recommendations.

10.2 Biometric Surveillance: Four Critical Trends

New technological implementations have allowed for an unprecedented regime of observation, rendering the people and their biometric data particularly vulnerable to unregulated or under-regulated state and business practices.

First, remote biometric surveillance may be aimed at matching citizens to reference datasets without their knowledge.Footnote ² In the absence of concrete laws targeted at such practices, states can hardly guarantee their citizens that firms – whose for-profit activities may be exercised around the globe and operate without enhanced checks and balances (known from public law) – will not collect this data unnoticed. Neither can it be guaranteed that firms will not share collected biometric data with law enforcement, who may subsequently exploit such data and inferences in the name of national security or the need to effectively fight against crime. In the Clearview case (discussed in Section 10.4), citizens became explicitly exposed to a giant firm’s mass processing and excessive sharing of sensitive data with law enforcement agencies around the world.

Second, biometric monitoring can be targeted at classifying people based on specific attributes, ranging from gender and age to political views.Footnote ³ With no specific regulation, citizens are unaware of how they may be protected against these unfairly discriminative practices – as discrimination on such bases is expressly prohibited under the Charter of Fundamental Rights of the European Union and the European Convention on Human Rights (ECHR).Footnote ⁴ Such protections are particularly important in an era when sensitive data is processed in an uncontrollable data-tsunami-fashion that becomes sharable with various state entities, and given that the European Court of Human Rights has held the view (and emphasised) for more than a decade that mere retention/collection of personal data may raise serious privacy-concerns.Footnote ⁵

Third, biometric watching can today be directed to processing with the further objective of drawing inferences on emotions or even intents.Footnote ⁶ Orwellian fears become relevant if citizens could suffer any detriment or mistreatment on the basis of ideas, feelings, or thoughts that, as regulators would agree, must stay untouched by any law or practice.

Fourth, old-school surveillance, for instance via CCTV systems, is no more old-school. With new applications and improvements of old technologies, citizens have come to realise that legal regimes, introduced to regulate old technological implementations, have failed to evolve and are apparently lagging behind rapidly developing tech-trends.Footnote ⁷ Gone are the days of a simple CCTV camera announced by an information notice that a location is under surveillance. These notices are hardly effective against powerful cameras capable of capturing detailed images from miles away.

These developments, leading to ubiquitous monitoring of all earth-citizens, must become a three-prong warning for regulators. First, although surveillance practices are very well targeted at citizens and their sensitive data, laws are not. Especially at the EU level, laws have remained untargeted, general, abstract, and neutral. Technologies such as cameras or drones are unmentioned in the 2016 General Data Protection Regulation (GDPR) or the 2016 Law Enforcement Directive (LED).Footnote ⁸ Much criticism has also surrounded recent efforts in the proposed AI Act to address more expressly certain emerging or materialised harms,Footnote ⁹ (potentially) caused by biometric and other un(der)regulated technologies.Footnote ¹⁰ Second, regulatory responses and checks, such as proportionality assessments performed by courts, must focus on and properly balance what is actually at stake, without fearing that they might look political or too activist.Footnote ¹¹ This risk is only heightened when a regulatory framework is lacking or too vague. Third, fundamental human rights demand priority and enforcement – an argument closely linked to the second point. While the risk-based, cost/benefit rationale already underlying many fields, from environment to data protection,Footnote ¹² could entertain utilitarianism-advocates, it cannot and should not replace the logic of the ‘fundamental’. There are certain sensitive areas where financial interests and security must not be over-prioritised; where fundamental human rights cannot be outweighed by being attributed numerical values in a mathematical fashion.Footnote ¹³

These technological trends and regulatory challenges must catch the eye of the regulator; for the watching of anyone anywhere, their sorting into whatever classes on whatever bases and for whatever purposes, the foreseeing of people’s thoughts and feelings, and the rebirth of old-school technologies escaping old-school laws have given birth to new citizens’ needs.

10.3 Regulatory Strategy: Focus on Concrete Technological Uses and their Possible Harm

The need for bright-line rules directed to concrete technological uses and possible harms has long been identified and stressed in privacy-related contexts;Footnote ¹⁴ and, in recent publications, we have resorted to the US legal regime and its piecemeal approach to make concrete recommendations that might be useful for EU audiences.Footnote ¹⁵ More concretely, we have digested about fifteen US-initiatives at federal, state, and local level. These initiatives refer either to biometrics or to face recognition.Footnote ¹⁶ On biometrics there is the federal 2020 National Biometric Information Privacy Act, which aims to tackle biometric data exploitation by private entities. What caught our attention was the setting out of concrete bans on specific manners of obtaining, exploiting, and sharing biometric data:

In the same vein, the 2008 Illinois Biometric Information Privacy Act sets out a number of targeted prohibitions on the processing (again, mainly obtaining, profiting, and disseminating) of biometrics by private entities (prohibitions that will play a crucial bright-line-rule role in the Clearview case discussed in Section 10.4).Footnote ¹⁸ We also appreciated the imposition of a standard of care (regarding storing, communicating, and securing) that ensures biometrics are treated in a similar way to, or are more shielded than, other confidential and sensitive information in that industry:

Similar is the 2009 Texas Business and Commerce Code Sec 503.001 ‘Capture or Use of Biometric Identifier’ (obviously influenced by the Illinois Act), which forbids the capturing, disclosing, or exploiting of biometrics in commercial contexts, save for exceptional circumstances. It further requires that when securing biometrics, ‘reasonable care’ must be shown and that any measures taken must have the same level of protection (or be more shielding) than the measures taken to store their own confidential data.

The 2019 California’s Assembly Bill No. 1215 is expressly aimed at forbidding biometric surveillance by law enforcement through cameras. There is not much to say about such a clear-cut provision targeted at avoiding abuse of law enforcement powers: ‘A law enforcement agency or law enforcement officer shall not install, activate, or use any biometric surveillance system in connection with an officer camera or data collected by an officer camera [….].’Footnote ²⁰

The 2020 California Privacy Rights Act is an EU-like tool targeted at businesses and the protection of consumers. Not only does it use GDPR-like terminology, but it also grants consumers various GDPR-like rights (including the right to correct inaccurate data or opt out of automated decision making), imposes on businesses GDPR-like obligations (such as the duty to conduct audits or risk assessments), and includes GDPR-like principles (such as data minimisation, purpose limitation, and storage limitation).

The 2020 Indiana House Bill 1238 imposes on law enforcement actors a duty to conduct a ‘surveillance technology impact and use policy’, make that policy available to the public, and update it prior to altering the technology’s function or purpose. Interestingly, these duties are set out using brief and simple phrasing:

The 2020 New York’s Assembly Bill A6787D aims to protect children by suspending the use of biometric technologies (including face recognition) in public and private schools. It does so through a moratorium on purchases and uses of technologies for a concrete period of time or until these technologies are proven safe: ‘Public and nonpublic elementary and secondary schools […] shall be prohibited from purchasing or utilizing biometric identifying technology for any purpose, including school security, until July first, two thousand twenty-two or until the commissioner authorizes such purchase or utilization […] whichever occurs later […].’Footnote ²²

The 2021 proposed Virginia’s Senate Bill 1392 focusses on private for-profit entities that process significant amounts of personal data, including biometrics. This Bill offers clear rules protecting biometric data as sensitive personal information, whose processing is in principle prohibited. What we found novel, compared with the GDPR-regime, is the prohibition on discrimination against consumers: ‘A controller shall not discriminate against a consumer for exercising any of the consumer rights […] including denying goods or services, charging different prices or rates for goods or services, or providing a different level of quality of goods and services to the consumer […].’Footnote ²³

Moving on to the US initiatives on face recognition, the proposed federal 2019 Commercial Facial Recognition Privacy Act bans the use of face recognition technology (FRT) by private actors (save where there is consent and, where possible, notification) for the purposes of facial recognition data collection, discrimination, purposes other than those of initial processing, and the sharing of facial recognition data. Though conditional, the ban on discrimination is, again, a novelty, when compared with the EU regime: ‘[I]t shall be unlawful for a controller to knowingly […] use the facial recognition technology to discriminate against an end user in violation of applicable Federal or State law […].’Footnote ²⁴

The federal 2020 Facial Recognition and Biometric Technology Moratorium Act forbids the federal government from using face recognition or other biometric technology until expressly allowed by the law: ‘[I]t shall be unlawful for any Federal agency or Federal official […] to acquire, possess, access, or use in the United States (1) any biometric surveillance system; or (2) information derived from a biometric surveillance system operated by another entity […] The prohibition […] does not apply to activities explicitly authorized by an Act of Congress […].’Footnote ²⁵

Washington’s Engrossed Substitute Senate Bill 6280 (2020) is targeted at state/local authorities using facial recognition services and imposes several concrete duties (such as conduct of accountability reports that are reviewable by the public), as well as restrictions (such as preventing the application of the technology to persons on concrete discriminatory grounds). What appeared interesting to us (in addition to the regulator’s concern about discrimination) was the clear ban on reliance upon the facial recognition service as the only basis for establishing ‘probable cause’ in criminal contexts or image-tampering in face recognition contexts. Nothing similar or even close to this exists in the LED:

The 2020 New Jersey’s Assembly Bill 989 is targeted at subjecting facial recognition technologies to accuracy- and bias-checking; again, the focus is placed on avoiding discrimination on concrete grounds: ‘The testing and auditing is required to determine whether there is a statistically significant variation in the accuracy of the facial recognition systems on the basis of race, skin tone, ethnicity, gender, or age of the individuals portrayed in the images, whether or not those categories are applied individually or in combination […].’Footnote ²⁷

Portland’s ordinances (2020) ban the application of face recognition to public spaces and by private entities, as well as the use of FRTs by the city’s public actors (‘bureaus’). Portland clearly says ‘no’ to both state and private entities.

Baltimore’s ordinance (2021) prohibits, first, the city of Baltimore from obtaining a face recognition system and contracting other entities with a view to using such systems (some biometric security systems are exempted) and, second, private actors from obtaining, retaining, accessing, or using a face recognition system or information gathered from such a system (certain biometric security systems and Maryland’s Image Repository System are exempted). Remarkably, in case of violation of the provisions on the ban related to private actors, the ordinance provides not only for civil, but also for criminal remedies: ‘§ 18-3. Penalties […] Any person who violates any provision of this subtitle is guilty of a misdemeanor and, on conviction, is subject to a fine of not more than $1,000 or imprisonment for not more than 12 months or both fine and imprisonment […] Each day that a violation continues is a separate offense […].’Footnote ²⁸

After analysing these US texts, we detected three key ideas that encapsulate the overall approach followed by the US regulators:Footnote ²⁹

Concreteness and precision: We appreciated the unambiguous clarity of the US initiatives, which appear to have clear objectives and target concrete and intrusive technological uses. Compared with the EU regime, US provisions are more demanding with respect to various requirements. First, although some bans are conditional upon consent, the latter goes beyond the EU model – demanding not only that consent be ‘informed’, ‘specific’, and so forth (terms also present in the GDPR), but also focussing on the independent, genuine will of the person concerned, who must be free from outside control. These demands make the US prohibition stronger and more honest than the EU’s ban, which is accompanied by a long list of exceptions.Footnote ³⁰ Second, some duties and prohibitions concretely set out in the US texts are completely absent in the EU. These include the prohibition on discrimination, the prohibition on profiting, the application of standards of care, and the treatment of biometric data as particularly sensitive and confidential information.

Bright-line bans: We saw explicit prohibitions on certain technologies or surveillance practices, often reaching the level of unconditionality. In this regard, Portland and its ordinances very well illustrate how both private and public actors can be prohibited from using FRTs. Remarkably, the US prohibitions aim to protect vulnerable groups (such as children) and anticipate, or probably avoid, possible abuses of coercive powers (for instance, by prohibiting law enforcement from using surveillance cameras). Even where ban-techniques, such as moratoria, can end upon the (future) introduction of laws that would allow for relevant uses, the United States demands that such laws be particularly detailed in various terms, ranging from lists of authorised entities to operation-standards, auditing duties and compliance-mechanisms. Probably, the best example is given by section 3(a)–(b) of the Federal 2020 Facial Recognition and Biometric Technology Moratorium Act quoted earlier.Footnote ³¹

Practical organisation of remedies: We found the United States’s supremacy in combining several legal fields (e.g., market, competition or criminal law/procedure) with a view to enhancing effectiveness of their remedy-scheme. Good examples can be found in the 2019 Commercial Facial Recognition Privacy Act (section 4(a)),Footnote ³² and in the Ordinance ‘Surveillance Technology in Baltimore’.Footnote ³³

One could argue that the EU’s general approach allows for an always-present regime covering any technological implementation; and, in our recent EU–United States comparative analysis, we addressed pros and cons of both general and concrete law-making, finding persuasive arguments for both approaches.Footnote ³⁴ However, in our opinion, what makes bright-line regulation more desirable (and more protective) is the very principle of legality.Footnote ³⁵ If laws are general and abstract by-design, then they risk becoming human rights-incompatible by default. If law enforcement and other state actors are not told by the lawmaker in simple, clear, and detailed language what they can and cannot do, not only are citizens under-protected, but also regulators are confused. Experience has indeed shown that lack of bright-line-rule-setting has confused and puzzled regulators, who may not be able to fully foresee or tell the legal grounds upon which proposed bans can be introduced.Footnote ³⁶

Today, with the tremendous challenges posed by the global reach of any anywhere-based tech-firm,Footnote ³⁷ as well as the mass adoption of latest technologies and pilot programmes in both private and public arenas,Footnote ³⁸ we encounter concrete risks from concrete uses (from school-areas involving vulnerable children to work environments obliging employees to be surveilled) that appear to demand concrete rule-setting.Footnote ³⁹ And, in our view, effectiveness of such precise rule-making can be enhanced by uniform enforcement aimed at scrutinising, banning, or sanctioning specific surveillance practices. At least one case, namely Clearview (discussed in Section 10.4), can support the claim that the ideal solution can include both precise rule-making and uniform enforcement.

10.4 Regulatory Strategy: Uniform Enforcement

In May 2021, several national data protection authorities and organisations submitted complaints against Clearview, an American face recognition-tech firm. The firm had in its hands the (allegedly) largest known database (more than 3 billion facial images). With its AI technology, it searches for human (face) photographs in the web, stores them on its proprietary database, and sells access to other firms or law enforcement authorities.Footnote ⁴⁰

Elsewhere, we have critically approached the Clearview-case, questioning the legal grounds for data collection and further processing, as well as doubting the lawfulness of sharing practices – particularly in relation to EU law enforcement authorities.Footnote ⁴¹ These concerns were recently shared by two national authorities.

Upon joint scrutiny conducted by the United Kingdom’s Information Commissioner’s Office (ICO) and the Office of the Australian Information Commissioner (OAIC), initiated in July 2020, these authorities gathered evidence from the web and searched separately for uses of relevant data by their law enforcement entities.Footnote ⁴² After stressing the global nature of the digital space and the resulting need for a global regulatory approach, they highlighted new challenges posed by Clearview’s practices.Footnote ⁴³ According to the ICO’s preliminary opinion, the firm had probably failed to comply with data protection laws in various respects (including unfair processing, lack of mechanisms to avoid forever-storage, no legal basis, and opaque processing).Footnote ⁴⁴ After expressing its intent to impose on the firm a provisional fine and after issuing its provisional notice to halt processing and erase relevant data,Footnote ⁴⁵ the ICO imposed a fine of £7.5 million and ordered deletion.Footnote ⁴⁶ While it was clarified that the firm’s services are no longer offered in the United Kingdom, the ICO stated that there is no guarantee that Clearview will stop processing data of UK citizens, in light of its opaque practices.Footnote ⁴⁷

What the Clearview-case can reveal is that uniform enforcement, collaboration (in the sense of looking for ways to make different approaches work), and co-ordination can successfully tackle the transnational, global reach, risk, and potential harm of surveillance practices. The success is not the imposition of the huge fine; rather, it is the desire of the regulators (ICO and OAIC), which was actually expressed and materialised via rigorous investigations and targeted application of the law, to a concrete technological use: Clearview’s risky, opaque, and harmful practice, exercised at global level, potentially affecting each individual citizen.

Such global exercise can very well be halted and sanctioned by collaborating regulators at national level(s). One could claim that Clearview’s fine and order to delete data may fail to ‘frighten’ gigantic firms; albeit, if collaboration between national authorities were embraced by various states, then analogous fines and orders imposed/issued by various domestic entities could have a considerable impact on the financial status of Clearview and similar big firms. Indeed, state authorities, finding absence of a legal basis, have taken steps in that direction and against Clearview: Italy, for example, imposed a fine of EUR 20 million,Footnote ⁴⁸ and France ordered the firm to halt processing.Footnote ⁴⁹ For a further discussion of the Clearview case, we refer to the discussion by Orla Lynskey, insisting on the limits of a European human rights approach.Footnote ⁵⁰ Judges and data protection authorities are inclined to avoid general statements about facial recognition and limit their intervention to cases involving facial recognition brought before them. The UK and French data protection authorities demand ‘settled evidence’ about the negative impact of this technology. Rather than banning a technology, they opt for prohibiting a certain processing activity. The Greek and Italian data protection authorities did indeed ban the Clearview processing activity, but only for future collection and processing of data through the company’s facial recognition system. The Italians moreover only ordered the company to erase the data relating to individuals in Italy. The United Kingdom’s ICO only ‘banned’ the web scraping by Clearview, but did not put a ban on Clearview’s facial recognition activities.

While in the EU Clearview’s abuses were sanctioned with fining and halting-orders, in Illinois, the firm was given a clear, quasi-permanent, and almost erga omnes-ban. More concretely, the American Civil Liberties Union (ACLU), a US-based organisation fighting for human rights and freedoms, brought its case against the giant firm, claiming violation of the Illinois Biometric Information Privacy Act. On 11 May 2022, there was a settlement accepted by the court, under which Clearview is permanently prohibited from offering its services to numerous private entities in the entire United States, as well as all entities (including the police) of the state of Illinois (the latter ban for the following five years).Footnote ⁵¹ The result is a settlement with compromises.Footnote ⁵² Clearview AI settled the lawsuit without admission of liability. There is a nationwide ‘Private Entity Ban’,Footnote ⁵³ supplemented with an ‘Illinois State Ban’ (no facial recognition services for state or local government entities including Illinois law enforcement),Footnote ⁵⁴ but for the law enforcement services outside Illinois there is also a law enforcement friendly ‘Savings Clause’,Footnote ⁵⁵ a shaky system to prevent further web scraping without consent for Illinois residents, and with no obligation to delete past collected data.Footnote ⁵⁶ It is not simple to compare the outcomes of this settlement with the preceding outcomes in the EU. Within the state of Illinois, the Illinois Biometric Information Privacy Act has delivered some of its promises and even more: Clearview is permanently banned, nationwide, from making its faceprint database available to most businesses and other private entities. The company also has to cease selling access to its database to any entity in Illinois, including state and local police, for five years. The Illinois Act was already used successfully to settle facial recognition practices by Facebook,Footnote ⁵⁷ and IBM,Footnote ⁵⁸ and has clearly brought the message to the United States that even for publicly available data, a citizen may claim that processing personal data without consent violates the law.Footnote ⁵⁹

Two remarks before concluding. First, in the EU, national authorities successfully defended citizens’ rights and freedoms by jointly investigating the firm’s practices and, after seeing the harm done, enforced the law and proceeded to various sanctions including halting and fining. Second, in the United States, there was a forever – and almost toward-any-party – ban prohibiting Clearview from selling its technology. Clearly enough, if the United States’s clear law-making was combined with the EU’s uniform enforcement, citizens would be better and more effectively protected against surveillance practices.

10.5 Conclusion: Precise Rule-Making and Uniform Enforcement as a Twofold Solution against Undesired Surveillance Practices

This analysis has shown that new technological trends, from monitoring of emotions to attempts to predict feelings, can pose novel, serious challenges that existing laws have failed to adequately tackle. This has in turn created new needs for global citizens: in particular, enhanced protection against increasing tech-interference. Looking to other jurisdictions for insights into how their targeted and precise regulations may better address new threats can offer useful lessons. Indeed, the US approach could offer insights into how specific uses and concrete harms could be more effectively avoided. Our argument for supremacy of the US initiatives is neither to dignify nor to deify the United States. Rather, it is to support the view that targeted and precise law-making is a matter of legality; in its absence, laws risk violating human rights by simply being abstractly designed. This is a claim we have already raised in previous publications;Footnote ⁶⁰ in this chapter, we have engaged in a meta-analysis to further argue that effectiveness of bright-line-ruling can be enhanced by uniform enforcement. The Clearview-section exemplifies how collaboration in enforcing the rules can work.

In our opinion, precise laws banning, halting, and sanctioning certain practices are not to be seen as vengeance; as revenge, fighting back against firms and their mass and over-surveilling technologies. Rather, they are to be seen as sincere manifestations of legality. And, when uniformly enforced, they are to be seen as honest manifestations of fairness. If numerous firms are bringing technologies into the market, into the court, into the law enforcement area, into the school, into the employment arena, and into any other domain one might imagine, technologies could be abused by strong entities such as the state, and used against weak parties such as the individual citizen; it would therefore make sense to demand that multiple actors (from investigating entities to administrative supervisory authorities) jointly enforce precise rules from various areas, such as competition or criminal law.

With these recommendations, we do not suggest that all tech-pioneers be treated as possible criminals, who should be chased by the entire enforcement-mechanism for designing technologies that might then be abused by the state. Such a far-reaching scenario, an erga omnes-regime attacking any tech-developer, would probably not be desirable. What is desirable in our opinion is a targeted, clear, and rigorous scheme applicable to those disrespecting legality and fairness at the detriment of anyone – from our children to our neighbours, ethnic or other minorities. If, for instance, a law bans our kids being watched in classrooms or when they play in the schoolyard, because such a monitoring would have a hostile impact on their personality development, their freedom of expression, their privacy, or their very dignity, then maybe the tech-developer that violated that law by selling surveillance cameras to schools should have its criminal record permanently marked to remind society of the harm suffered by those kids. Even though, in this example, no blood was spilled and no kid died of the camera-watching, citizens may want to remember the detriment this for-profit designer caused to our kids, their personality, their freedom of expression, their privacy, and their dignity – things any citizen would die and spill blood for.

11.1 Introduction

Police use of facial recognition is on the rise across Europe and beyond.Footnote ¹ Public authorities state that these powerful algorithmic systems could play a major role in assisting them to prevent terrorism, reduce crime, and to more quickly locate and safeguard vulnerable persons (online and offline).Footnote ² There is also an international consensus among policymakers, industry, academia, and civil society, that these systems pose serious risks to the rule of law and several human rights integral to the existence of a democratic society.Footnote ³ These include the rights to private life, freedom of expression, freedom of assembly and association, and equality as guaranteed under the European Convention on Human Rights (ECHR).Footnote ⁴

In response to these ‘profound challenges’, policymakers and researchers have called for law reform that would provide greater clarity on the limits, lawfulness, and proportionality of facial recognition and other emerging AI-based biometric systems (such as gait and emotion recognition).Footnote ⁵ Consequently, some local and state governments in the United States have placed legal restrictions or banned law enforcement use of facial recognition technologies.Footnote ⁶ During the pre-legislative stages of the proposed EU AI Act, the European Parliament has also issued calls for a ban on the use of private facial recognition databases in law enforcement.Footnote ⁷ The world’s first case examining the legality of a facial recognition system deployed by police, Bridges v. South Wales Police, thus remains an important precedent for policymakers, courts, and scholars worldwide.Footnote ⁸

This chapter focusses on the role and influence of the right to private life, as enshrined in Article 8 ECHR and the relevant case law of the European Court of Human Rights (ECtHR), in the ‘lawfulness’ assessment of the police use of live facial recognition (LFR) in Bridges. A framework that the Court of Appeal for England and Wales ultimately held was ‘not in accordance with the law’ for the purposes of Article 8(2) and therefore in breach of Article 8 ECHR.Footnote ⁹ The analysis also considers the emerging policy discourse prompted by Bridges in the United Kingdom (UK) surrounding the need for new legislation.Footnote ¹⁰ This marks a significant shift away from the current AI governance approach of combining new ethical standards with existing law.Footnote ¹¹

11.2 Facial Recognition Systems in Law Enforcement: Legal and Technical Issues

Within a legal context, the use by public authorities of any preventive measures that will indiscriminately capture and analyse the biometric data of a vast number of innocent individuals raises significant questions. These include whether rule of law requirements developed for ensuring adequate limits, safeguards, and oversight for police surveillance systems in the pre-Internet era, such as closed-circuit television (CCTV), remain adequate and relevant to facial recognition systems and other modern internet-enabled and automated monitoring systems; furthermore, whether such novel and powerful AI-based technologies are strictly necessary in a democratic society and respect the presumption of innocence.Footnote ¹² Privacy and data protection concerns have also been raised regarding the transparency and oversight challenges posed by the increasing role of the private sector within the areas of law enforcement and public security. These developments range from law enforcement use of data-driven tools and systems developed by industry, including commercial facial recognition software,Footnote ¹³ to tasking industry itself with law enforcement functions.Footnote ¹⁴

11.3 Justifying an Interference with Article 8 ECHR

11.4 Bridges v. South Wales Police: The ‘Lawfulness’ of AFR Locate

11.4.3 Was SWP Deployment of AFR Locate ‘In Accordance with the Law’ under Article 8(2)?

11.5 Conclusions

While some valuable guidance has been provided by the Court of Appeal in Bridges, which draws (albeit in a limited way) on the lauded legality case law of the ECtHR dealing with Article 8 ECHR and police investigatory powers, the current patchwork of law governing police use of facial recognition in the UK falls short of lawful and trustworthy public policy.

As the UK House of Lords rightly points out, it is not for the courts to set the framework for the deployment of new technologies. This chapter argues that the reasoning for this is threefold. First, court judgments are not systematic, comprehensive, or evidence based. Secondly, they represent ad hoc reviews of problematic public policymaking that only occur when (and if) a legal challenge is brought before them. Thirdly, the courts will assess only a narrow scope of issues relevant to that specific case. The implications posed by the lack of an accessible and foreseeable framework for police use of AFR in the UK are significant. This gap represents a source of confusion and legal uncertainty for policymakers, police, industry, courts, and citizens, thereby giving rise to gaps and patchy protection of affected rights and safeguards, including but not limited to the right to private life. These all serve to undermine adequate and effective compliance, oversight, evaluation, and thus public trust in the use of these novel and increasingly sophisticated police powers.

There is, however, a post-Bridges discourse on lawfulness that has moved towards enacting a law specifically tailored to regulating use of facial recognition. Such reform could address the current obscurity and uncertainty in the current patchwork of legal rules in England and Wales governing the limits and safeguards underpinning police use of facial recognition, particularly the compilation and application of watchlists. This legislation could then meet the accessibility and foreseeability tests under the legality condition of Article 8 ECHR, the ‘minimum degree of legal protection’ to which citizens are entitled under the rule of law in a democratic society.Footnote ⁹⁸ Such reform would also enable ‘greater certainty and accountability’ around police use of AI-based biometric surveillance systems and other emerging technologies.Footnote ⁹⁹

12.1 Introduction

Facial recognition technology (FRT) functions by analysing key facial features to generate a mathematical representation of them, and then comparing these against the mathematical representation of known faces in a database to determine possible matches. This is based on digital images (still or from live camera feeds). In a policing context, FRT is used to help verify the identities of persons ‘of interest’ to police. State-operated surveillance involving FRT is hardly a novel phenomenon in the United Kingdom (UK). The UK has been the crib of the use of FRT. A technology that initially was used by public entities, it is now widespread also in the private sector.Footnote ¹ According to a recent study, there are more than 6 million closed-circuit television (CCTV) cameras in the UK, more per citizen than in any country apart from China.Footnote ² These cameras can take images of faces they film and compare them against a pre-defined database of images to determine if there is a match. That means they can be used to quickly identify individuals even in crowded areas such as shopping centres, airports, railway stations, and city streets. Even when a face is partially covered – by a cap or glasses, for example – they can still usually match it up with a stored image.Footnote ³

The extensive presence of FRT in the UK raises concerns from the angle of democracy and individual freedoms: is the UK becoming an ‘Orwellian’ society where all individuals are monitored, identified, and potentially controlled? As observed in the literature, mass surveillance has immediate implications on privacy rights, but the knowledge gathered through monitoring can be used to compress other individual freedoms.Footnote ⁴ It follows that regulation on FRT should strive to minimise the interferences with privacy, and thus other individuals’ rights, if democratic values of human dignity and pluralism are to be truly achieved.

Several non-governmental organisations (NGOs) protecting privacy rights established in the UK became the centre of important strategic litigation to protect privacy rights.Footnote ⁵ For instance, in the Bridges case,Footnote ⁶ supported by the NGO Liberty, the Court of Appeal has not only invalidated the use of facial recognition technology by South Wales Police (SWP), but also raised the attention to some unsolved issues regarding the use of FRT for law enforcement. As a matter of fact, notwithstanding the presence of multiple legal sources governing FRT, several legal and ethical issues are still unsolved. Clear legislation on FRT is missing.Footnote ⁷ In which circumstances should FRT not be used? What information duties should be discharged by those utilising FRT? What remedies should exist for individuals to address abuses of this technology? These are only some of the questions that should be addressed by legislators in order to prevent the emergence of an Orwellian society. What the future holds for FRT in the UK remains to be seen. Uncertainty is even higher in light of Brexit and the potential reforms to be introduced in the UK on the data protection framework.Footnote ⁸

This chapter outlines the framework on FRT in the UK and offers reflections on the future of this technology in that jurisdiction. It is structured as follows. First, it discusses the uses of FRT in the UK and the public perceptions surrounding this technology. Second, it explores the UK relevant legal framework, and highlights its gaps. Third, the chapter discusses the Bridges saga and its implications. Fourth, the chapter highlights selected regulatory matters on FRT that are currently unsettled and on which legislative guidance appears necessary to prevent the establishment of an Orwellian society in the UK. Conclusions follow.

12.2 FRT in the UK: Between Public and Private

To assess the impact of FRT in the UK, we need first to explore its use in this jurisdiction. The first observation is the extensive use of this technology by both private and public entities. Starting from the public sector, the first CCTV system in the UK was set up in 1953 in London for the Queen’s coronation.Footnote ⁹ By the 1960s, permanent CCTV began to cover certain London streets. Since then, the reach of CCTV surveillance has expanded in sporadic bursts, with many cameras installed in response to the 1990s IRA attacks and then again after 9/11 and the London Underground bombing.Footnote ¹⁰ Currently, CCTV cameras embed FRT that allows the identification of individuals against the information included in databases managed by law enforcement bodies. Policy documents produced by Metropolitan Police and the College of Policing indicate that FRT can be used to improve the fight against crime and make people’s lives safer.Footnote ¹¹ Moreover, the British government specifies that CCTV serves four purposes: the detection of crime and emergency incidents, the recording of events for investigations and evidence, direct surveillance of suspects, and the deterrence of crime.Footnote ¹² In the past, critics argued there is little evidence to support the proposition that its use has reduced levels of crime. An internal report dated 2009 produced by London’s Metropolitan Police revealed that only one camera out of every 1,000 had been involved in solving a crime.Footnote ¹³

However, recent documents produced by the Metropolitan Police indicate that the main advantage of using FRT is that of making manhunts more effective. It was observed that many manhunts for offenders wanted for very serious offences such as murder involve hundreds of officer and staff hours. When aggregated together, manhunts cost many thousands of policing hours across London. By comparison, the four recent trial deployments of live facial recognition (LFR) resulted in eight arrests.Footnote ¹⁴ It was also reported that LFR deployments provide opportunities for police officers to engage with a person potentially wanted by the police and the courts. Another relevant comparative metric for LFR is the policing outcomes resulting from ‘stop and search’. According to a report published in February 2020 by the Metropolitan Police,Footnote ¹⁵ 13.3 per cent of stops resulted in an arrest in 2019. By contrast, 30 per cent of engagements following an adjudicated alert from the LFR system resulted in the arrest of a wanted person.Footnote ¹⁶ While the enhancement of public security and safety via FRT is a valuable goal, we should not lose sight of the significant implications of this technology on individual freedoms.

Such implications are amplified by the substantial employment of FRT by private entities in the UK. For instance, Clearview AI has collected more than 20 billion images of people’s faces and data from publicly available information on the internet and social media platforms all over the world, including in the UK, to create an online database. The Information Commissioner’s Office (ICO) has recently sanctioned this company for violation of data protection rules.Footnote ¹⁷ Further examples are supermarkets such as Tesco, Budgens, and Sainsbury, and start-ups such as Yoti and Facewatch. Such private entities utilise FRT in different fashions. For instance, Yoti, an FRT software, is used in UK cinemas to verify the age of customers,Footnote ¹⁸ while a growing number of businesses use Facewatch to share CCTV images with the police and identify suspected shoplifters entering their store.Footnote ¹⁹ Such widespread use of FRT by private entities is likely to cause invasive interferences with individual entitlements. Let us consider, for instance, the employment of FRT in supermarkets and in the workplace. The data gathered through FRT used in supermarkets might increase the potential for profiling consumers and thus limiting their choices based on selected biometric features.Footnote ²⁰ Similarly, the use of FRT by employers could potentially facilitate profiling and monitoring employees’ behaviours and even emotional states. As a result, employees may be controlled and ultimately prevented from exercising their fundamental rights, such as the freedom of expression. Constraining and regulating the use of this technology by private entities becomes essential to prevent indiscriminate restrictions of fundamental rights.

Another peculiarity of the use of FRT in the UK is that, especially in the field of law enforcement, the deployment of this technology has occurred via partnerships between private entities providing digital services or infrastructures and public entities. For instance, the Japanese technology company NEC provides cameras to the Metropolitan Police and SWP.Footnote ²¹ There is no transparency on how NEC was identified as supplier to SWP. The only publicly available information is contained in a series of statements published by NEC’s and SWP’s websites.Footnote ²² This example raises the question of how the selection of specific technologies provided by private entities may shape public services. As a subsequent matter, the issue arises as to what values, principles and rules should guide public–private partnerships in the field of law enforcement, especially when dealing with the processing of sensitive personal data.

The diffusion and evolution of FRT in the UK has led to the development of a system in which civil society has been crucial in casting light on the issues attached to FRT technology and its impact on individuals’ rights. The establishment of numerous privacy-related NGOs appears to be a direct consequence of the spread in use of this technology on the UK territory. To name but a few, Privacy International, Liberty, Open Rights Group, and Big Brother Watch were all born out of the concerns surrounding mass surveillance in the UK.Footnote ²³ These entities have contributed to many strategic litigation cases that have shaped the legal landscape of FRT regulation in the UK. The Bridges case, discussed in Section 12.4, is an instance of strategic litigation relating to FRT driven by the NGO Liberty. It is difficult to draw a clear connection between the work of civil society in the field of FRT and the impact of advocacy and strategic litigation on public awareness regarding the FRT challenges and risks. However, recent studies have highlighted that the UK public has a contradictory stance with reference to this technology.

In a study conducted by Steinacker and his colleagues involving more than 6,000 respondents, it was observed that while an overall of 43 per cent of respondents supported the use of surveillance, 26 per cent opposed it.Footnote ²⁴ In the same study, 39 per cent of the interviewees expressed the view that FRT increases privacy violations and 53 per cent were of the opinion that FRT enhances surveillance.Footnote ²⁵ These findings were confirmed by a study conducted by the Ada Lovelace Institute in 2019. The Institute commissioned YouGov to conduct an online survey with over 4,000 responses from adults aged sixteen and above. The survey asked respondents to express their views on a range of uses of FRTs in a number of settings including law enforcement, education, and in the private sector.Footnote ²⁶ The report found that support for the use of FRT depends on the purpose. Notably, the study found that 49 per cent of the respondents supported its use in policing practices with the presence of appropriate safeguards, but 67 per cent opposed it in schools, 61 per cent on public transport, and a majority of 55 per cent wanted restrictions placed on its use by police.Footnote ²⁷

In light of these findings, it appears that the public perception of FRT in the UK depends on the use of that technology. While this research illustrates that individuals appreciate the potential of FRT in the field of security and law enforcement, the general impression emerging from these surveys is that there is still a lack of awareness regarding the full consequences and impact of FRT on individual rights beyond privacy. This conclusion is further strengthened when one considers the significant gaps existing in the UK regulatory approach to FRT. It is argued that were the implications of FRT on individual rights’ protection entirely appreciated, a stronger social resistance against FRT would emerge in light of the current limited framework. The attention on safety and security as one of the advantages of FRT would most likely be reassessed against the worrisome implications that mass surveillance, and ultimately a police state, would have on individual freedoms. The following paragraphs outline the UK legal framework on FRT and its limits.

12.3 The Legal Framework

Until 2019, the Law Enforcement Facial Images and New Biometrics Oversight and Advisory Board oversaw the police use of automated facial recognition (AFR), LFR custody images, and new biometrics. The last meeting of the Board took place in September 2019 and alternative governance arrangements are now in place.Footnote ²⁸ Currently, two bodies supervise the use of FRT: the ICO and the Biometric and Surveillance Camera Commissioner. The legal framework governing FRT in the UK is multi-layered. It is composed of human rights law, but also by data protection and law enforcement rules. As a result, the rights to privacy and data protection, being the most immediately entitlements affected by FRT, are to be balanced with public security and law enforcement objectives.

The starting point for analysing the UK FRT framework is the Human Rights Act, which gives effect to Article 8 ECHR, protecting the right to privacy, in the UK territory. In addition, the Data Protection Act (DPA) of 2018,Footnote ²⁹ which transposed the EU’s General Data Protection Regulation (GDPR) in the UK, plays a crucial role in governing FRT. This Act provides the duties for controllers and processors and rights for data subjects. It grants enhanced protection for sensitive personal data,Footnote ³⁰ and imposes specific requirements for personal data used in the context of law enforcement.Footnote ³¹ While under EU law data protection is a fundamental right, in the post-Brexit era data protection has lost this status since the EU Charter of Fundamental Rights is no longer binding in the UK.Footnote ³² Additionally, the UK GDPR framework may be subject to evolution in light of recent plans of the UK Government to depart from the EU legislation and case law.Footnote ³³

The Protection and Freedoms (PoFA) Act 2012 is also of relevance, since it regulates the use of evidential material, including biometric material that may be gathered through FRT. Furthermore, mention should be made of the Surveillance Camera Code of Practice, originally published in 2013 and amended in November 2021. This code is an implementation of Section 29(6) of PoFA and is to be taken into account by a relevant authority in the exercise of its functions when involving the operation or use of any surveillance camera systems, or the use or processing of images or other information obtained by virtue of such systems. The code sets out twelve guiding principles, such as that there should be effective review of audit mechanisms to ensure respect for legal requirements, policies, and standards. While this code applies to public authorities, private entities are not constrained by it. The ICO has issued guidance harmonising the Surveillance Camera Code of Practice with the GDPR requirements.Footnote ³⁴ In this sense, the guidance has a broader scope than the code. In addition, we should mention that public authorities using FRT technology have produced policy and guidance documents. To name but one example, the Metropolitan Police have issued several LFR policy documents, including Data Protection Impact assessments and the ‘Standard operating procedure’.Footnote ³⁵ Similarly, SWP has produced multiple documents stating their approach to the deployment of FRT.Footnote ³⁶ Finally, several guidance documents, such as those issued by the British Security Industry Association regarding the ethical and legal use of AFR,Footnote ³⁷ or the Data Ethical Framework prepared by the UK Government, provide directives on the employment of FRT.Footnote ³⁸ The effects and status of these guidance documents is unclear. While they may be used to guide the action of public authorities, whether or not they are binding is allegedly different from the law.Footnote ³⁹

Overall, the private use of FRT appears less regulated than the public enforcement. However, the presence of a more developed legislative framework for the public sphere does not equate to effective FRT regulation in that sector. In 2019, the London Policing Ethics Panel advanced several recommendations concerning LFR, such as that there should be enhanced ethical governance of policing technology field research trials, and that regulation of new identification technologies should be simpler.Footnote ⁴⁰ The ICO also issued an opinion on the use of LFR technology by law enforcement authorities in public places, which concluded that the use of that technology should meet the threshold of strict necessity.Footnote ⁴¹ For example, it was suggested that FRT could be used to locate a known terrorist but not indiscriminately in order to identify suspects of minor crimes.Footnote ⁴² The 2022 report of the Minderoo Centre for Technology and Democracy found that the use of FRT by the UK police did not meet fundamental rights standards.Footnote ⁴³ Yet, as mentioned, private parties may also be extremely intrusive when utilising FRT. One may wonder whether this different treatment for private bodies, which are subject to less cumbersome duties when utilising FRT, is at all justified.

In the UK, the ICO, former Biometrics Commissioner, and former Surveillance Camera Commissioner have all argued that the law relating to biometric technologies is no longer fit for purpose.Footnote ⁴⁴ The same point was advanced by the Court of Appeal of England and Wales in August 2020 in its judgment on the Bridges case, concluding that there were ‘fundamental deficiencies’ in the legal framework surrounding the police use of facial recognition.Footnote ⁴⁵ The next paragraphs offer an overview of this case, which is pivotal in identifying existing regulatory gaps concerning FRT in the UK.

12.4 The Bridges Case

The case concerned the deployment of AFR Locate, a technology that involves the capturing of digital images of members of the public, which were then processed and compared with digital images of persons on a watchlist compiled by SWP. The claimant in the case, Edward Bridges, supported in his action by the NGO Liberty, raised complaints against the use of this technology against him on two occasions and against the use of AFR Locate in general. The watchlists used in the deployments contested by Mr Bridges included, among others, persons wanted on warrants, individuals who were unlawfully at large having escaped from lawful custody or persons simply of possible interest to SWP for intelligence purposes.

At first instance, the Divisional Court declared that Article 8 ECHR was not violated. This was because of ‘the common law powers of the police to obtain and store information for policing purposes, and [the fact] that the compilation of the watchlists is both authorised under the Police and Criminal Evidence Act 1984 and within the powers of the police at common law’.Footnote ⁴⁶ The court also found that DPA 2018 and the Code of Practice on the Management of Police information provided a legal basis for the use of AFR Locate. Overall, the ‘accordance with the law’ requirement laid down in Article 8(2) ECHR was satisfied. Furthermore, the Divisional Court rejected the pleas based on data protection law. Of interest is the way in which the court delineated the scope of the margin of appreciation enjoyed by the ICO. Notably, it concluded that it was for the ICO to assess whether the documents adopted by the SWP complied with Section 42(2) of the DPA 2018, requiring the adoption of a policy document by public entities processing personal data for law enforcement purposes. The court also rejected the claim that SWP had failed to comply with the Equality Act 2010.

Mr Bridges challenged the Divisional Court’s judgment and was granted leave to appeal. In its judgement, the Court of Appeal began by considering whether the interference of privacy rights caused by the SWP was in accordance with the law, as demanded by Article 8(2) ECHR. While it found that the action of the SWP was carried out pursuant to a legal basis, it embraced a relativist approach: it advanced the view that ‘the more intrusive the act complained of, the more precise and specific must be the law said to justify it’.Footnote ⁴⁷ After acknowledging that the technology involved in the case was different from that considered in previous judgments,Footnote ⁴⁸ the court held that ‘the legal framework that the Divisional Court regarded as being sufficient to constitute the “law” for the purposes of Article 8(2) is on further analysis insufficient’.Footnote ⁴⁹ In particular, the Court of Appeal argued that two issues remained open under the framework in place, the ‘who’ and the ‘where’ questions. As a matter of fact, the applicable law did not clarify who could be placed on the watchlist, nor was it clear that there were any criteria for determining where AFR Locate could be deployed. On this issue, the court advanced the view that the legislator should provide clearer guidance on the erasure of data of individuals who are captured by FRT but do not match the identity of any person included in the watchlist. Subsequently, the judgment moved on to the analysis of the Surveillance Camera Code of Practice. The court noted that ‘the guidance does not contain any requirements as to the content of local police policies as to who can be put on a watchlist. Nor does it contain any guidance as to what local policies should contain as to where AFR can be deployed.’Footnote ⁵⁰ The court also assessed the documents issued by the SWP, and concluded that they too left unsolved the ‘who’ and ‘where’ questions. As a result, the first ground submitted by Mr Bridges concerning the violation of the legal basis requirement under Article 8 ECHR was well founded.

The court then tackled the second ground raised by Mr Bridges; that is, whether the SWP complied with principle of proportionality in the deployment of AFR Locate. The judgment found that the Divisional Court did not err in the assessment of proportionality. While the appellant had suggested that the balancing under proportionality should consider not only the FRT’s impact on a single individual, but also on the public as a whole, the Court of Appeal held that the assessment of proportionality should occur as a matter of legal principle,Footnote ⁵¹ and therefore not in abstract terms. The second ground was thus dismissed.

However, the court allowed the appeal on the third ground submitted by Mr Bridges, notably that the data protection impact assessment (DPIA) carried out by the SWP did not comply with the DPA 2018 requirements. On this issue, the Court of Appeal ruled that, since SWP had failed to comply with Article 8 ECHR, and especially the ‘in accordance with the law’ requirement, the DPIA was not compliant with the DPA 2018.

Subsequently, the Court of Appeal evaluated whether the SWP had failed to respect Section 35 of the DAP 2018, detailing the first data protection principle. The combined reading of Sections 35, 42 and Schedule 8 DPA 2018 requires public entities processing personal data for law enforcement purposes to have appropriate policy documents in place. The Court of Appeal held that, since the ICO had found that the SWP documents contained sufficient information in compliance with Section 42(2) DPA, the Divisional Court did not err in law. The fact that the ICO had later revised the guidance on FRT and law enforcement could not change the validity of the ICO’s opinion on the policy documents. Putting it differently, the updated guidance of the ICO could not have retroactive effects and invalidate the policy documents adopted by SWP.

Finally, the court considered whether SWP had breached the Equality Act 2010. To address this plea, the court evaluated the robustness of the verifications carried by SWP with reference to the potential biases entailed by the FRT. The court observed that ‘SWP have never sought to satisfy themselves, either directly or by way of independent verification, that the software program in this case does not have an unacceptable bias on grounds of race or sex. There is evidence […] that programs for AFR can sometimes have such a bias.’Footnote ⁵² As a result, the court concluded that the safeguards employed by the SWP were insufficient, and therefore this ground of appeal was allowed.

The Bridges saga prompts several observations. First, it demonstrates that the central provision in the reasoning of the parties as well as the court in drawing the boundaries for the use of FRT was the fundamental right to privacy protected under the ECHR. By contrast, the data protection framework was employed to ‘compensate’ and strengthen the fundamental right to privacy. Through the prism of Strasbourg case law, Article 8 ECHR appears to offer ample guidance to courts on how to achieve the protection of privacy even in the face of technological advancements such as FRT.Footnote ⁵³ Second, the Bridges case showcases the intersectionality of FRT. This technology does not only impact privacy and data protection rights, but also other fundamental entitlements, such as the right not to be discriminated against. Yet additional fundamental rights could be found to intersect with the use of FRT, such as the freedom of expression or the right to liberty. Third, the case suggests that different understanding of the principle of proportionality and its interplay with fundamental rights can allow for stricter or laxer scrutiny over the employment of FRT. In Bridges, the Court of Appeal did not consider the ‘necessity’ requirement or the ‘stricto sensu’ proportionality; rather, it carried a soft scrutiny over the choices of the SWP. Hence, owing to the malleability of proportionality, one may wonder whether this is an effective principle to carry a precise scrutiny over the deployment of this technology and its implications. The answer to this matter depends on personal views on the very principle of proportionality. Fourth, one may wonder how much ‘law’ is needed to regulate FRT. While the Court of Appeal considers that it is not the place of the judges to dictate what the law should look like, at the same time it cast light on selected drawbacks and limitations emerging from the current framework. The Court of Appeal invited the legislator to clarify the ‘who’ and ‘where’ questions and to detail rules on the deletion of personal data for individuals captured by FRT. One could think of additional questions and issues that require legislative action. Indeed, the Bridges saga highlighted only selected open questions concerning the use of FRT in the UK. The future of FRT regulation in the UK will depend on how the uncertainty surrounding these issues is tackled.

12.5 The Future of FRT in the UK

While the Bridges case has powerfully illustrated some of the crucial gaps in the current framework on FRT, there are further unsletted matters. To name but a few: For what purposes and in what contexts is it acceptable to use FRT to capture an individual’s image? What checks and balances should be in place to ensure fairness and transparency in the use of FRT? What accountability mechanisms should be established for different usages? The list could continue. Several NGOs have produced reports partially addressing these matters. Interestingly, there seems to be convergence towards the (at least partial) halting of FRT under the current rules. For instance, the Ada Lovelace Institute commissioned the Ryder Review,Footnote ⁵⁴ published in June 2022, which recommended that the use of live FRT should be suspended until the adoption of a legally binding code of practice governing its use. The presence of binding rules identifying accountable entities and means of redress for individuals are considered as crucial to enhance the protection of individuals against FRT technology.Footnote ⁵⁵ The report specified that the code should not only address the public use of the technology, but also its deployment by private parties. Furthermore, the Mideroo report on the use of FRT,Footnote ⁵⁶ published in October 2022, went as far as calling for a ban of FRT in the context of police activities. The report justified this recommendation in light of the blatant violations of fundamental rights by way of deployment of FRT by the police.

Interestingly, these proposals are to a certain extent in line with the position of EU institutions. For instance, the European Data Protection Board called for a general ban on any use of AI for an automated recognition of human features in publicly available spaces, as well as for AI systems categorising individuals from biometrics into clusters.Footnote ⁵⁷ Moreover, in the European Parliament there is growing consensus on banning the use of this technology.Footnote ⁵⁸ Whether the UK legislator and authorities involved in the regulation of FRT will reach a similar conclusion requiring the suspension, if not the banning, of FRT remains to be seen. In July 2022, Liberty published a tweet indicating that the Metropolitan Police used FRT at Oxford Circus. As a result, thousands of people walking in that area were monitored and captured by cameras. Such overt and extensive use of FRT in the UK might signify that this jurisdiction is still far away from undergoing a serious reconsideration of FRT’s limited benefits and high risks. However, several crucial changes to the current rules seem necessary. These reforms should involve increasing public awareness of the implications of FRT as well as enhancing transparency on the deployment of that technology. Another point that future legislation should tackle is how to ensure that public–private partnerships involving the digitisation of public services respect public goods and values. The opaque co-operation between NEC and SWP suggests that the public is unable to scrutinise how public entities build their co-operation with private digital providers, and therefore how much power private parties have in shaping the public sphere. Until the day such legislation is in place, it is legitimate to ask: ‘Does Big Brother exist in the UK?’

12.6 Conclusion

The UK has been a crib for the development and deployment of FRT. Since the 1950s, this technology has been largely used in the public sphere, and especially for law enforcement purposes. However, FRT has rapidly expanded, and it is now omnipresent, having landed also in the private sector. As a result, the UK legal order offers a remarkable case study to reflect on the future of FRT regulation. The existing FRT framework in the UK is multi-layered but also fragmented and incomplete. The loopholes of the rules currently in place became evident in the Bridges saga. While the first instance court considered the use of FRT by SWP lawful, the Court of Appeal identified violations of Article 8 ECHR, data protection rules, and the Equality Act 2010. Accordingly, the UK judicature has revealed the power of fundamental rights in regulating FRT and cast light on the limits of existing rules. In particular, the Court of Appeal observed that the legislator should clarify who can be placed on watchlists and where the FRT can be employed. Yet additional questions remain open, beyond those identified by the Bridges case: For what purposes and in what contexts is it acceptable to use FRT to capture an individual’s image? What checks and balances should be in place to ensure fairness and transparency in the use of FRT? What accountability mechanisms should be established for different usages? The list could continue. Several NGOs have called for halting or even banning FRT in the UK. There is general consensus that the current UK framework is insufficient. Until the point when the UK legislator takes charge of enhancing regulation relating to FRT, it is legitimate to ask: ‘Does Big Brother exist?’

13.1 Introduction

Facial recognition technologies (FRTs) have raised concerns in Germany,Footnote ¹ and have not been put to use on a widespread basis. This may not be expected to change in the near future, as the current coalition treaty between the German government parties rejects comprehensive video surveillance and the use of biometric measurement for surveillance purposes.Footnote ²

This reluctance to put FRT to use may explain why, so far, the use of FRT has seldom come before German courts: Only fifty-three court decisions out of a total of 1.6 million decisions of German courts in the legal database juris include a textual reference to ‘Gesichtserkennung’, the German term for facial recognition.Footnote ³ A search for ‘Biometrie’, equivalent to ‘biometrics’, yields 991 decisions.Footnote ⁴ However, many of these latter decisions only have a tenuous link to FRT. These numbers suggest that FRT has rarely been the subject-matter of legal proceedings in Germany.

Nevertheless, there are individual instances in which FRT is already being employed – or has been employed – in the public sphere in Germany. Three prime examples of real-life use cases of FRT in the public sector in Germany will be discussed in further detail.

The first example concerns the pilot study involving the continuous use of FRT without specific cause, conducted at the Berlin Südkreuz train station (which has received a high degree of public attention). The second example is the use of FRT in the aftermath of the G20 riots in Hamburg. Here, FRT was employed to analyse video recordings from mass gatherings to identify suspects. As a third example, FRT cameras are being used in the city of Görlitz to combat serious border crime. In Görlitz, FRT is employed for a limited time and for specific cause. Hence, these examples illustrate different scenarios of the application of FRT. They will be discussed in turn to illustrate specific requirements and challenges, particularly with a view to the varying degree of detail of relevant legal provisions.

13.2 Constitutional Framework for FRT in the Public Sector in Germany

All cases of FRT use take place within the constitutional framework of the Grundgesetz (Basic Law – GG). FRT mainly raises concerns with regard to the right to informational self-determination (Art. 2 (1) in conjunction with Art. 1 (1) GG), which has first been recognised in a decision by the Bundesverfassungsgericht (Federal Constitutional Court – BVerfG) on the 1983 Federal Census Act.Footnote ⁵ Additionally, and depending on the specific context, FRT may affect other fundamental rights, such as the right to assemble (Art. 8 (1) GG).Footnote ⁶ And, even more fundamentally, the BVerfG has acknowledged that the constitution entails a ban on total surveillance,Footnote ⁷ and underlines its importance as part of Germany’s constitutional identity: ‘It is an integral part of the constitutional identity of the Federal Republic of Germany that the state may not record and register the exercise of freedoms by citizens in its entirety.’Footnote ⁸ So far, the BVerfG has not decided a case that directly involved the use of FRT. Absent a pertinent judgment, a recent decision by the BVerfG on automatic licence plate recognition (ALPR) may provide orientation, and guidelines for FRT can be derived a fortiori from this decision.Footnote ⁹ As Martini points out, both ALPR and FRT aim at automated surveillance of the public sphere and would lend themselves as tools for permanent surveillance.Footnote ¹⁰ Personal data collected via ALPR or FRT can be used to draw inferences about the persons monitored.Footnote ¹¹ While ALPR uses information that may indirectly relate to persons, FRT surveillance directly pertains to biometric data. Thus, even higher legal standards would apply to FRT than for ALPR.Footnote ¹²

Specifically, in its decision on ALPR, the BVerfG has first ascertained the broad scope of the right to informational self-determination (which would be relevant both for ALPR and FRT): ‘The right to informational self-determination covers threats and violations of personality that arise for the individual from information-related measures, especially under the conditions of modern data processing.’Footnote ¹³

The right to informational self-determination applies even in the public sphere, where individuals have an interest in ensuring that their personal information is not collected and stored without their consent (which, again, equally concerns ALPR and FRT): ‘Even when individuals go out in public, the right to informational self-determination protects their interest in ensuring that the associated personal information is not collected in the course of automated information collection for storage with the possibility of further exploitation.’Footnote ¹⁴

Different stages of data processing have to be distinguished and need respective justification, in particular the collection, the storage and the use of data: ‘Regulations that allow for the handling of personal data by government authorities generally justify various interventions that build on each other. In particular, a distinction must be made in this respect between the collection, storage and use of data.’Footnote ¹⁵

For all stages of data processing, the basic principles of proportionality, clarity of legal rules, and certainty apply:Footnote ¹⁶

Proportionality in this context is understood in a narrower sense, as a prohibition of excessiveness. The pursued purpose must be proportionate to the impact on the individuals’ right to informational self-determination (the comparatively deeper impact of FRT on individual rights would affect this analysis accordingly, and a more important purpose would be required): ‘The principle of proportionality in the narrower sense as a prohibition of excessiveness is only satisfied … if the purpose pursued is not disproportionate to the weight of the intervention entailed.’Footnote ¹⁸ To be justified, automated licence plate checks must be prompted by a sufficiently concrete, objectively determined reason. Furthermore, the conditions for a check must meet a certain threshold and allow for compliance review.Footnote ¹⁹ Checks cannot be carried out arbitrarily or without a valid reason.

Furthermore, the BVerfG stressed that surveillance measures must serve ‘to protect legal interests of at least considerable weight or a comparably weighty public interest’.Footnote ²⁰ It is crucial to note that FRT raises additional concerns about privacy and the potential for misuse. Thus, the standard for the use of FRT would be higher than for automated licence plate checks.

Moreover, the general framework for surveillance measures must also be proportionate in a broader sense of the term; that is, in an overall assessment:

From these considerations, the BVerfG derives more specific procedural requirements to protect individual rights: ‘In addition, the proportionality requirements include requirements relating to transparency, individual legal protection and supervisory control as well as regulations on data use and deletion for all individual acts.’Footnote ²²

13.3 Application in Specific FRT Use Cases

This general framework sets the standard for the application of FRT in specific cases and its legal basis. In this section, the chapter discusses in turn the aforementioned instances in which FRT has already been applied: a pilot study that entailed the continuous use of FRT without specific cause at Berlin Südkreuz (Section 13.3.1), the use of FRT for analysis of video recordings from mass gatherings at the G20 summit in Hamburg (Section 13.3.2), and the use of FRT for a limited time and with specific cause in the city of Görlitz (Section 13.3.3).

13.4 Conclusion

So far, FRT has only been employed in individual cases in Germany. The BVerfG has acknowledged that a ban on total surveillance is part of Germany’s constitutional identity. For individual measures, constitutional key considerations concern their proportionality, the clarity of legal rules, and certainty. Furthermore, specific procedural safeguards are required.

This arguably amounts to a high threshold for FRT measures, as the examples discussed show. The permanent use of FRT without specific cause cannot be based on the existing provision Section 27, sentence 1 no. 2 BPolG, as it is not sufficiently specific. The use of FRT for analysis of video recordings from mass gatherings (as after the G20 riots) based on Section 48 (1) BDSG was viewed in a positive light by an administrative court. However, commentators raise the issues of legal clarity and certainty, and point out that the administrative court has not sufficiently explained the proportionality of the measures in this instance. Finally, even a very specific provision on the use of FRT for a limited time and with specific cause has been criticised for a potential lack of proportionality and for not being specifically determinate in certain regards.

14.1 Introduction

In Lithuania, rather than being determined by the intrinsic needs of society, legal regulation of face recognition technology (FRT) came merely as a part of the EU’s general data protection framework. Prior to this, the rules governing facial image usage of private persons were regulated mainly by the Civil Code of the Republic of Lithuania,Footnote ¹ which provides that if a photo (or a part thereof), portrait, or other image of a natural person is to be reproduced, sold, displayed, and printed, the person may be photographed only with their consent – but this is not required if these actions are related to the person’s social activities, their official position, the requirement of law enforcement authorities or if the photograph is taken in a public place. However, a person’s photo (or part of it) taken in these cases may not be displayed, reproduced, or sold if this would degrade the person’s honour, dignity, or professional reputation.Footnote ² In terms of the work of law enforcement institutions, as will be seen from the analysis presented in this chapter, the laws regulating the work of separate law enforcement institutions or laws regulating specific activities of law enforcement (as a general rule) provide that the law enforcement institutions may collect and process personal data, usually without specifying the regime applicable to the collection and processing of biometric data.

As in all of the EU member states, law enforcement institutions in Lithuania have to adhere to EU standards of FRT usage, especially those laid down in the Directive (EU) 2016/680 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data by competent authorities for the purposes of the prevention, investigation, detection, or prosecution of criminal offences or the execution of criminal penalties, and on the free movement of such data (the Law Enforcement Directive).Footnote ³ However, each country also has local national standards that transpose other requirements to the FRT framework and its practical use.

From the perspective of the effectiveness of legal regulation, it should be noted that a country might have a very definite and clear legal rule or a set of rules regulating a particular field of social relations; however, this regulation may rendered as declarative and not implemented in practice. In Lithuania there are some examples of such, and one of the most prominent involves the legal regulation of lobbying activities. At the time of consideration of the Law on Lobbying Activities in the Parliament of Lithuania, one of the Members of Parliament noted that the relations that were going to be regulated were little known in Lithuanian society, so the law was not expected to be accepted in practice. He also said during the parliamentary session that it looked as if the law was aiming to ‘prepare cosmonaut suits and then see if there would be cosmonauts willing to try them on’.Footnote ⁴ Indeed, this law (adopted in 2000) was one of the worst examples of legislation in Lithuania, as lobbying activities were practised despite what was stated in it until 2018 – when the law was amended significantly, this time following broad discussions with stakeholders and society. This and similar examples imply that in order for legislation to be applied in practice, it needs to fit both the legal culture and legal system of a country as well as fall in line with the views of wider society.

Keeping this in mind and recognising that society has an important role in controlling the implementation of legal acts, especially where they relate to human rights, the proper implementation of FRT regulations also relies on society and related interest groups deeming them necessary, otherwise they may remain declarative and void. If public awareness and pressure to have a law implemented properly are high, the implementing institutions are forced to take action. Usually, strong players in the performance of social control are non-governmental organisations (NGOs), especially where regulations or their improper implementation pose a threat to human rights. Therefore, it is of major importance that society and NGOs accept and understand the need and the usage of FRT in law enforcement institutions.

This section analyses the regulation of FRT usage by Lithuanian law enforcement institutions, as well as the public discussion relating to FRT usage in the media, NGO involvement, and other types of social control. Finally, the chapter considers what changes may be brought to national regulation of FRT by the EU Artificial Intelligence Act.

14.2 Legal Framework for the Use of FRT in Law Enforcement in Lithuania

In general, the basis for the use of biometric data (including facial recognition data) in Lithuania is the Law Enforcement Directive,Footnote ⁵ which was transposed into the Law of the Republic of Lithuania on the Legal Protection of Personal Data Processed for the Prevention, Investigation, Disclosure or Prosecution of Criminal Offences, Execution of Sanctions or National Security or Defence and other legal acts.Footnote ⁶ Biometric data are classified as a special category of personal data that reveal racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership; they include genetic data and data concerning health or a person’s sex life or sexual orientation. Processing of these personal data categories is only allowed when strictly necessary, subject to appropriate safeguards for the rights and freedoms of the data subject, and only where it is authorised by the EU or Lithuanian law – for example, when it is needed to protect the vital interests of the data subject or another person; or when such processing relates to data that are manifestly made public by people themselves.Footnote ⁷

In Lithuania the collection and use of facial images on the one side, and processing of personal data such as biometric data, on the other, are regulated in law enforcement institutions by different laws and other legal acts. For example, the Law on Police of the Republic of Lithuania provides that with a person’s consent and/or in cases established by law, police officers are entitled to take photos and make audio or video recordings. Without a person’s consent, a police officer can take pictures of unidentified persons, persons in a helpless condition, unidentified corpses, risk group persons, and temporarily detained persons; they can be measured and their external features described, audio or video recordings can be made, fingerprints can be taken, samples can be taken for genetic testing to perform typification or for comparative research and identification, and all these data can be processed.Footnote ⁸ The law also states that the police can process personal data necessary for the implementation of police tasks, including the personal code, without the consent of the data subject, and that when processing data, the police have the right to collect them using technical means.Footnote ⁹

The Penal Code provides that the Probation Service may receive data, documents, and other information necessary for the execution of public service sentences (or to get acquainted with this information) from the state, municipalities, and other institutions, bodies, or organisations with state information resources. The Probation Service is also entitled to process the personal data of convicted persons.Footnote ¹⁰

In criminal procedure there is a general requirement that the use of technical means and their results are also subject to the requirements of public information, personal data protection, the right to inviolability of private life, and the protection of personal honour and dignity established in other laws.Footnote ¹¹ This means that all steps in criminal procedure involving the use of biometric data should be in line with the previously mentioned provisions of the Law of the Republic of Lithuania on the Legal Protection of Personal Data Processed for the Prevention, Investigation, Disclosure or Prosecution of Criminal Offenses, Execution of Sanctions or National Security or Defence. It is therefore quite natural that, for example, the Law on Prosecution of the Republic of Lithuania,Footnote ¹² or the Law on Financial Crime Investigation Service,Footnote ¹³ do not mention handling of any type of personal data at all.

However, a number of laws regulating the activities of law enforcement institutions do not provide clear wording on the possibility of collecting and using facial images. For example, the Law on Intelligence of the Republic of Lithuania provides only that state intelligence institutions have a right to process personal data, without clarifying what kinds of data these might be, and the provision for the performance of particular activities after having received a court permit only mentions ‘access to a person’s home, other premises or vehicles, their inspection and recording’,Footnote ¹⁴ without a clear reference to collection or use of facial images. Similarly, the Law on Criminal Intelligence does not provide clear grounds for collecting and using facial images; it does not speak about handling personal data at all. This law only mentions that criminal intelligence activities (meaning the activities of criminal intelligence officers in collecting, recording, evaluating, and using available information about criminal intelligence objects) must be carried out in accordance with the procedure established by the law, and that methods of collecting criminal intelligence information are agency activity, survey, inspection, control inspection; controlled transportation; imitation of a criminal act; ambush; tracking; covert operation; and tasks of law enforcement authorities. However, this law also mentions that human rights and freedoms cannot be violated during criminal intelligence activities. Individual limitations of these rights and freedoms are temporary and can only be applied in accordance with the procedure established by law, in order to protect the rights and freedoms of another person, property, or public and state security.Footnote ¹⁵ The Code of Administrative Offences also generally states that investigative activities may include photography, video recording, audio and video recording, footprints and casts, plans and diagrams, and other recording techniques.

Regarding the activities of the Special Investigation Service, the handling of personal data is indirectly mentioned in the law that regulates the institution’s analytical intelligence activities. It is stated that analytical anti-corruption intelligence means analytical activity carried out by the Special Investigation Service, which includes the collection, processing, comparison of information on corruption and related phenomena with other public or classified information available to the Service, obtaining qualitatively new data that is the result of these information processing processes, and use by and provision to state or municipal institutions and officials authorised to make significant decisions in terms of reducing the prevalence of corruption. The possibility of using available biometric data is provided, as in order to achieve its operational goal and implement the tasks assigned to it, the Special Investigation Service has the right to receive relevant documents from all public institutions.Footnote ¹⁶ Additional rules are applied in respect of the collection and usage of facial images and the usage of biometric data in the process of issuing identity documents and migration.Footnote ¹⁷

Thus, it can be stated that the legal rules on the collection and usage of facial images and generating/usage of biometric data in Lithuania are rather fragmented and vague. As may be seen, in most cases it is stated that law enforcement institutions may collect and process personal data needed for the fulfilment of their duties without specifying any additional restrictions or criteria. Based on the personal nature of biometric data and the rigorous collection and processing of facial images, in accordance with the laws provided here, it is possible that every person may be affected: not only those who are subject to the issuance of personal identity documents or involved in migration issues, or those in any way involved in criminal proceedings or other proceedings that relate to national security and state interests, but also any other persons who act or appear in public places.

The second important issue is that the legal acts implementing the provisions of laws stray even further from the requirements applied to data collection and processing. For example, based on the provision of the Law on Police (the police have the right to collect and process personal data necessary for the implementation of their tasks without the consent of the data subject), all municipalities in Lithuania have adopted separate rules on the use of video surveillance cameras and the data they record.Footnote ¹⁸ Video surveillance may be established with the aim ‘to identify persons who may have committed administrative offences and criminal acts’. Consequently, this means that cameras can be established in any public place and may collect video data on all persons appearing there.

Still, a nonetheless important issue is the processing of the video surveillance data and other facial images. According to the aforementioned and related legal acts, facial images can be stored in a number of databases (which are usually interlinked): the Police Information System and other police department registers and information systems, the Criminal Intelligence Information System; databases of detention facilities (prisons, probation offices, etc.); court and other authorities’ databases; databases of institutions issuing identity documents; databases of the Migration Department and the State Border Guard Service; databases of the state enterprise Regitra, which issues driving licences; databases of institutions issuing personal documents; and municipal databases.Footnote ¹⁹

In this context it is important that according to the law, as well as to the Law Enforcement Directive and the General Data Protection Regulation (GDPR),Footnote ²⁰ ‘biometric data’ means personal data resulting from specific technical processing relating to the physical, physiological, or behavioural characteristics of a person, which allow or confirm the unique identification of that person. The definition of biometric data in GDPR (as well as in the Law Enforcement Directive) has generally been restricted to mean a technically defined digital representation of bodily traits that has been processed for machine or algorithmic analysis. This is suggested by the wording that data have to be subject to ‘specific technical processing’. Speaking more broadly, data processing systems do not need all of the data, but instead rely on extracting meaningful sub-parts from voice or image data, which can then be easily compared to existing ‘templates’ in a database. This implies that photographs and video images of faces are expressly excluded from the definition of biometric data both in GDPR and the Law Enforcement Directive.Footnote ²¹ Therefore, there is a difference between the regulatory rules applied in respect of data images (which may be regarded as personal data) and images processed with FRT technology (which then is regarded as biometric data).

To summarise, in Lithuania there is quite a significant gap between the regulatory rules that set requirements for FRT from a data protection perspective and rules regulating activities of separate law enforcement institutions and law enforcement activities. Although the standards of data protection in general seem to be sufficient, the specific laws on law enforcement institutions solely provide the possibility to collect and process personal data, including facial images, without making it known whether FRT will be used to process such images or not. Therefore, there is a possibility that the general data protection rules are only declarative and not enforced in practice. Thus, in order to understand whether the legal regulation of FRT usage in Lithuanian law enforcement is sufficient, a deeper analysis of the practical implementation of regulatory rules on the usage of FRT in Lithuania is needed.

14.3 FRT Usage in Practice

According to the respondents to the Government Use of Facial Recognition Technologies: Legal Challenges and Solutions project,Footnote ²² the volume of FRT usage in law enforcement institutions is not clear. In the course of this project the team sought to interview representatives from the institutions that are responsible for (or directly participate in) the processing of personal data, including facial images and biometric data. However, only very few representatives were willing to participate, whereas others stated they had insufficient knowledge of or competence in these issues. Moreover, according to a couple of respondents from the private sector, who were trying to investigate the use of FRT in the context of human rights, the representatives of law enforcement institutions disclosed to them that they felt comfortable as they benefited from having considerable latitude for when and how to use FRT as a result of the vague legal background.

As an example of insufficient regulatory basis for handling biometric data, including facial data processed using FRT, the case of the Register of Habitoscopic Data of the Republic of Lithuania may be analysed. This is a component of the Internal Affairs Information System – a general system for storing detailed personal identification data in a single database, which stores data on convicted persons, persons who have served a sentence of arrest or fixed-term imprisonment in the Republic of Lithuania, temporarily detained persons suspected of having committed a criminal act, wanted persons, identification marks of unidentified dead bodies or unknown helpless persons, and other categories. This data is used by pre-trial investigation institutions, border protection, customs, the prosecutor’s office and other law enforcement institutions in order to ensure the prevention of criminal acts and the fight against crime. It processes personal data for the following purposes: (1) to investigate criminal acts and ensure their prevention, organise, and carry out the search for persons, as well as to identify both unidentified corpses and unknown helpless persons according to personal identification marks; and (2) to determine the identity of a person in order to ensure the control of the movement of foreigners who have been detained by the competent control authorities for illegally crossing the state border by sea, land, or air from a third country and who have not been returned to that country.Footnote ²³ The Register of Habitoscopic Data contains data on the external characteristics of a person, obtained by photographing, measuring, and describing the person’s appearance. According to this definition and the list of processed data presented in the same Order, the Register of Habitoscopic Data processes personal data that does not fall under the definition of biometric personal data, meaning no additional rules on the processing of biometric data should apply. The Order does not mention or otherwise provide grounds for processing of FRT-related biometric data. However, there was a public announcement in the media about the reorganisation and improvement of the Register of Habitoscopic Data through the ‘Modernisation of Register of Habitoscopic Data using advanced technologies of face recognition and identification tag search’ project.Footnote ²⁴ The project description states:

The Police Department website provides information about a related project. It is stated that:

However, as mentioned earlier, there is no legal ground for processing biometric personal data in the Register of Habitoscopic Data, nor are there security measures to be applied in order to ensure the protection of biometric personal data based on the criteria established in the Law Enforcement Directive and the Law of the Republic of Lithuania on the Legal Protection of Personal Data Processed for the Prevention, Investigation, Disclosure or Prosecution of Criminal Offenses, Execution of Sanctions or National Security or Defence. Furthermore, there are no terms for storage of biometric data (data processed by facial recognition technologies that allows identification of a specific person) in the Register of Habitoscopic Data.

Moreover, the Order of the Minister of the Interior establishing the Register of Habitoscopic Data allows the linking of the Register of Habitoscopic Data with other state registers (Residents’ Register, Addresses’ Register, Register of Application of Preventive Measures, Official Register of Wanted Persons, Unidentified Corpses and Unknown Helpless Persons, Register of Suspected, Accused and Convicted Persons, Official Register of Criminal Acts, Register of Dactyloscopic Data, Register of DNA Data, Register of Foreigners, and Register of Events registered by the Police). However, in the description of the ‘Modernisation of Register of Habitoscopic Data using advanced technologies of face recognition and identification tag search’ project, it is stated that ‘three new integration interfaces have been created: with the Integrated Criminal Procedure Information System (IBPS), the Register of Administrative Offences (ANR), and the Lithuanian National Second Generation Schengen Information System (N.SIS)’. In other words, the Register of Habitoscopic Data has interconnections with other registers that are not found in the relevant regulatory document.

The Ministry of the Interior of the Republic of Lithuania was officially asked to provide an explanation of the differences between the current regulatory framework for the operation of the Register of Habitoscopic Data and the declared updates to the register, which are said to have been already implemented.Footnote ²⁷ However, no response was received.

Such a situation implies not only that the regulation of collection of facial images (which falls outside the scope of ‘biometric data’ definition) and the processing of such images to generate biometric data is not regulated properly, but also that the current practices (given no information is provided about any unpublished legal regulations – which is unlikely given the requirements of transparency in the field of human rights and data protection) are likely to be in breach of the existing legal basis for such activities. First, as already mentioned, the data and information, as regulated by the Order of the Minister of the Interior on the Register of Habitoscopic Data, would be limited only to facial images and their description, with digital processing using FRT not being mentioned. The use of FRT brings the activities of the Register to a different level of legal requirement – that is, the obligation to conform with the rules applicable to biometric data processing. Second, the interconnections between the Register of Habitoscopic Data and other registers are not clear. It appears that in practice there are links to more registers than provided in the relevant Order of the Minister of the Interior, however, it is not clear what data could be exchanged. It should also be noted that a special Order of the Commissioner General of the Police restricts the transfer of facial image data (received via public surveillance cameras) to state registers to situations when there is a need to verify or specify information on a particular criminal or administrative offence.Footnote ²⁸ However, it is still plausible that all facial images of both recognised and unrecognised persons, who may be captured by video or photo cameras established for public surveillance by accident and without taking any part in an offence, could be automatically processed for facial biometric data.

14.4 Public Acceptance of the Usage of FRT by Government Authorities in Lithuania

To begin with, the issues surrounding FRT usage by Lithuanian government authorities are not commonly mentioned in media, NGOs, or social networks. Similarly, as with all advances in artificial intelligence (AI), FRT is welcomed positively as a facilitator of general life in Lithuania. For example, the Strategy of Artificial Intelligence in Lithuania encourages integrating AI, including FRT, into all economic sectors. Specifically, regarding the public sector, it is stated that AI will be helpful in the field of crime control, optimising the daily work of public institutions and improving the provision of public services.Footnote ²⁹ In particular, the optimisation of work is a rather attractive promise for most institutions – for example, the Kaunas Information Technology School carried out the ‘Attendance Marking Powered by Face Recognition’ project, which revealed that teachers would save time significantly if attendance of students was checked by using FRT rather than manually.Footnote ³⁰ Moreover, FRT was even suggested as a practical solution for simplifying the checking of persons who had been vaccinated against COVID-19, with proposals to use FRT instead of the official ‘opportunities passport’ system, which was declared to be ‘outdated’.Footnote ³¹ In Lithuania the case law on application of FRT is a rarity. However, a recent court decision directly relating to FRT usage demonstrates how the argument about convenience may easily transform into an argument about public interest. The State Data Protection Inspection challenged an order made by a university regarding the procedure for students’ remote examinations and measures related to the processing of personal data in order to ensure fair behaviour during examinations. This document established that the following personal data will also be processed during the state-level emergency brought about by COVID-19: surveillance photos, facial biometric data, audio recording of the exam. In this case the court declared that the rules of the university were legitimate as they were necessitated by public interest.Footnote ³²

Quite a strong argument with the public in favour of FRT use is the possibility of increasing public safety. Municipal institutions boast that they have introduced surveillance cameras that increase the safety of citizens. For example, the Mayor of Marijampolė municipality publicly announced that the network of sixty-four video surveillance cameras installed in 2020 has raised security in the city to a new level:

Or, for example, a local internet news portal of Mažeikiai district proudly presents:

It seems that residents are confident and satisfied with such usage of FRT in public places. People have even complained that the video cameras do not adequately ensure safety, as upon an accident the recording is too blurry or badly angled so that not all persons captured can be identified:

Moreover, FRT in public places is used not only for safety reasons, but also for fun: in Vilnius there was a two-year experiment in which researchers’ devices measured the face temperature, breathing rate, heartbeat, and emotions of any passers-by. The explanation was that this experiment was intended to substitute for a public poll on how people feel at a given moment in a given place, as it was a much more precise way to do so.Footnote ³⁶

On the other side, certain aspects of FRT usage have also been criticised in the media. For example, it has been widely and critically discussed that Lithuanian institutions are using video surveillance cameras made in China, which raises doubts as regards the safety of the data recorded and potentially threatens state security.Footnote ³⁷ Moreover, the potential for the misappropriation of FRT footage was revealed to the public in a well-known case concerning a policeman who had published online a video that had been recorded in a police car in which a drunk women took off her clothes.Footnote ³⁸

Nonetheless, these examples of the usage of FRT being publicly criticised are rather rare, and public attention is paid only to cases that raise state security issues or where there is a manifest infringement of professional duties. The overall attitude of Lithuanian society towards FRT usage seems to be positive – at least this is what can be seen from media sources. It seems that priority is given to the vast development of FRT and other AI technologies because the public can benefit from increased convenience and safety, while human rights issues related to threats to privacy, discrimination, or false accusation are left aside. Indeed, no civil society organisations in Lithuania prioritise threats posed by usage of FRT and AI. Therefore, it may be assumed that public discourse is driven by the position of state institutions and any developers’ interests in this field – thus a critical standpoint is lacking.

14.5 What Impact on FRT Usage by Law Enforcement Institutions is Expected upon the Application of the EU Artificial Intelligence Act?

As has been noted, the fragmented regulatory basis and rather weak public control of FRT usage in Lithuania could lead to the uncontrolled usage of FRT in law enforcement. Hopefully, the application of the EU Artificial Intelligence Act may bring about some changes to this situation. In April 2021, the European Commission presented the draft Artificial Intelligence Act, which is intended to introduce high standards for an EU trustworthy AI paradigm. It sets out core horizontal rules for the development, trade, and use of AI-driven products, services, and systems across all industries within the territory of the EU. This proposal introduces a ‘product safety regime’ that is constructed around a set of four risk categories. It imposes requirements for market entrance and certification of high-risk AI systems through a mandatory CE-marking procedure. This pre-market conformity regime also applies to machine learning training, testing, and validation datasets. Thus, according to Mauritz Kop,Footnote ³⁹ the draft AI Act combines a risk-based approach (based on the pyramid of criticality) with a modern, layered enforcement mechanism. This means that as risk increases, stricter rules apply.

Regarding the definition of ‘biometric data’ in the law enforcement area, the proposed AI Act makes a reference to the Law Enforcement Directive.Footnote ⁴⁰ However, the draft Act provides separate definitions for ‘remote biometric identification system’, ‘“real-time” remote biometric identification system’, ‘“post” remote biometric identification system’, and so on., with a specific regime being applicable to these categories. For example, the draft Act states that it is prohibited to use the ‘“real-time” remote biometric identification systems’ in publicly accessible spaces for the purpose of law enforcement, unless and in as far as such use is strictly necessary for one of the following objectives:

1. (1) the targeted search for specific potential victims of crime, including missing children;

2. (2) the prevention of a specific, substantial, and imminent threat to the life or physical safety of individuals or of a terrorist attack;

3. (3) the detection, localisation, identification, or prosecution of a perpetrator or suspect of a criminal offence.Footnote ⁴¹

As stated in the Explanatory Memorandum to the proposed Artificial Intelligence Act, the choice of a regulation as a legal instrument is justified by the need for a uniform application of the new rules, such as definition of AI, the prohibition of certain harmful AI-enabled practices and the classification of certain AI systems. The direct applicability of a Regulation, in accordance with Article 288 TFEU, should reduce legal fragmentation and facilitate the development of a single market for lawful, safe, and trustworthy AI systems. It is expected to introduce a set of harmonised core requirements regarding ‘high-risk’ AI systems and construct obligations for providers and users of those systems – improving the protection of fundamental rights and providing legal certainty for operators and consumers alike. At the same time, the provisions of the regulation must not be too prescriptive and should instead leave room for different levels of member state to take action regarding elements that do not undermine the objectives of the initiative, in particular the internal organisation of the market surveillance system and the uptake of measures to foster innovation.Footnote ⁴²

To summarise, the adopted Artificial Intelligence Act should bring more precision to the types of FRT used in law enforcement activities, and apply more controls to its use. However, the issue of transparency of FRT usage and making information available to the public or academics may still remain restricted as it is now, unless rising social pressures force such a practice to change.

14.6 Conclusions

There is quite a significant gap between the regulatory rules, which set requirements for FRT from a data protection perspective, and rules regulating the activities of separate law enforcement institutions and law enforcement activities. This may be because the usage of personal data, including facial images and their processing, was established in the specific laws regulating law enforcement much earlier than 2016, when the general data protection framework was established. Therefore, the Lithuanian legal framework clearly demonstrates that there are separate rules allowing the collection and processing of personal data (i.e., biometric data) in law enforcement activities, as well as separate rules that are more general and require a specific protective regime to be applied for the collection and processing of personal, including biometric, data.

Notwithstanding the fact that in theory the standards of data protection in general seem to be sufficient to protect against the rapid progression of technologies processing personal (and biometric) data and the evident threats to privacy and other human rights they pose, it still seems that the specific requirements on processing of personal data, especially processing biometric data, are not yet fully included in the practices of law enforcement in Lithuania. Moreover, it may be seen that the practices used in the development and usage of the Register of Habitoscopic Data do not comply with the regulatory requirements, in particular with the rules regulating the establishment, structure, and use of habitoscopic data. Rules on processing and sharing biometric data contained in this Register are not sufficient to ensure its proper protection, as required in data protection laws and EU documents.

Regarding the public attitudes to the regulation and usage of FRT in law enforcement in Lithuania, it may be noticed that neither society nor NGOs working in the field of human rights show any particular interest in analysing or restricting the usage of FRT in law enforcement institutions. On the contrary, media sources indicate that society at large is satisfied with the fact that the number of surveillance cameras in public places is increasing, and feels that it is a good and acceptable development that the possibility particular persons in public spaces can be recognised is increasing, as this brings the feeling of safety and order.

Although the adopted EU Artificial Intelligence Act should bring some discipline and clarity to the national regulation of FRT systems as well, as the reasons for using FRT, there are still doubts as to whether the transparency of FRT usage will be increased if societal and organisational attention and interest regarding FRT remains at the same level.

15.1 Introduction

The United States generally takes a light-touch approach to regulation, and notably so in the technology sector. This approach applies equally to facial recognition technology (FRT). But while the US regulatory touch may be light, this does not mean that it is either simple or non-existent. This chapter chronicles regulation of FRT in the United States at federal, state, and local levels, and considers potential regulatory issues on the horizon.

Every chapter in this volume discusses FRT, so the reader is assumed to have more than passing familiarity with its salient technological capabilities and limitations. Very briefly, for purposes of our analysis, FRT is a tool by which computers can identify individuals by an image of their face, generally using sophisticated algorithms that compare visual characteristics of an image to vast databases of other faces. We emphasise the role of image-based analysis, because there are other FRTs that may use other indicators. For instance, specialised systems may use infra-red cameras to recognise the structure of blood vessels underneath the skin and use these to identify individuals. This is certainly a form of FRT, but because high-resolution infra-red imaging is not a pervasive technology (yet?), technologies such as this are not currently a focus of this debate. Similarly, research into FRT benefits adjacent fields, such as autofocus technologies that dramatically improve the quality of devices such as cameras. Because such technologies are not used for identification purposes, they generally are not directly implicated by the discussion in this chapter. However, restrictions on the use of one application of FRT could well affect the development or use of other applications of related technology.Footnote ¹

FRT presents unique challenges as an identification technology. The simple explanation for this is that faces are pervasive. Humans present them publicly every time we are outside. We cannot meaningfully obstruct them, for both practical and social reasons – and, it bears note that FRTs are increasingly able to identify individuals based even upon obstructed images of their faces. And like all biometric markers, individuals cannot alter the appearance of their faces in the same way that they can, for instance, change a password or have multiple email addresses. Taken together, this makes faces both uniquely pervasive and uniquely persistent as identification tokens.

There is a wide range of potential use cases for FRT, ranging from contextually positive, to neutral, to potentially problematic. For instance, at the possibly innocuous end of the spectrum, FRT can be used to facilitate consumer convenience features, such as information kiosks that recognise individuals and automatically present relevant information to them. Or FRT could facilitate contactless payment or check-out features. Social media platforms can use FRT to automatically identify individuals in posted images. This information could be used for tracking purposes, or to learn information about those individuals that could be used for advertisement targeting. It could also be used to alert individuals when others post recognisable images of them, potentially giving them an opportunity to take privacy-protecting steps in response.

Perhaps the most potentially concerning class of FRT use cases stems from the widespread use of surveillance cameras in public settings – all of which can, in principle, incorporate FRT systems. In this setting, FRT can be used expansively by both private and public actors. For instance, venues such as shopping malls can use FRT-enabled security systems.Footnote ² FRT can be used to help law enforcement identify or track fugitives or other wanted individuals in public places – or to help search for missing persons. In the most extreme setting, FRT effectively eliminates any sense of privacy or anonymity that individuals may have when in public spaces. Where before the advent of FRT there was ‘anonymity in crowds’, today there is none.

This chapter proceeds in four parts. US law presents a complex set of regulatory tools and institutions – institutions that are often overlapping and often competing with one another. Section 15.1 provides a brief overview of this myriad of institutions. Sections 15.2 and 15.3 then bifurcate the discussion along two sets of institutions: federal and state-level regulatory efforts. Section 15.2 looks at recent federal efforts relating to FRT; Section 15.3 looks at recent state-level efforts. Section 15.4 offers some observations on issues that are on the horizon for the regulation of FRT technology in the United States.

15.2 Setting the Stage: Technology Regulation in the United States

US law does not present a single, unified, legal system. The United States is a federation of more than fifty states and territories, each with unique constitutions and legal environments. The federal government has its own Constitution and enacts its own laws, which sometimes displace, sometimes co-exist with, and other times are secondary to state law. Beyond that, most ‘law’ in the United States actually comes in the form of regulations enacted by federal or state agencies. And in many settings – perhaps most notably those relating to the technology sector and privacy-related concerns – US law relies extensively on self-regulation and sectoral regulation.

Before turning to any specific US regulatory approaches to FRT, this chapter presents a brief overview of these interrelated regulatory institutions.

The starting point for understanding the US legal approach to FRT – as well as many related issues, such as privacy issues generally – is to understand US law’s emphasis on protecting individual autonomy from intrusion from the government. That is, US law is largely premised on negative rights, or rights to be free from interference from the government. This stands in stark contrast to many other legal systems that are premised on positive rights, or guarantees that the government provide or protect individual liberties.

Thus, for example, the US tradition of privacy law is largely anchored in the First and Fourth Amendments.Footnote ³ Beyond those found in these amendments, Americans have limited fundamental privacy rights against the government. These amendments protect the freedom of speech and limit the government’s ability to encroach upon individuals’ other rights without due process of law. And the privacy rights facilitated by these amendments look very different from those anchored in other concepts, such as a right to dignity or self-determination.Footnote ⁴ The First Amendment guarantees that individuals cannot be compelled by the government to speak, including potentially by disclosing information about themselves. And similarly, the Fourth Amendment prohibits the government from searching and seizing individuals’ property – again preventing compelled disclosure of information to the government – absent obtaining a specific warrant from a federal court subject to due process of law. Critically, both of these amendments only run against the government. Neither prohibits private entities from compelling speech or disclosure of information, such as a condition of service. And neither prevents others from sharing or disclosing facts known about others, absent specific indicia of harm.Footnote ⁵

These principles give rise to a defining doctrine of US privacy law: the third-party doctrine. This doctrine says simply that one has no reasonable expectation of privacy in information disclosed or publicised to a third party. If a user shares information with another private entity, such as an online service, that entity is largely (though not entirely) free to do with that information as it pleases, and the Fourth Amendment provides the user with no protection against government efforts to obtain that information. And if an individual shares their information publicly, that information may be used generally. This includes merely being seen in public – with few exceptions, under US law an individual may have their public activities tracked, documented, and shared by other individuals and private entities. The Fourth Amendment may prohibit the government from tracking individuals in this way, but does not reach other private entities – indeed, government actors may even be able to acquire information about an individual from third parties, even where the government could not have created that information itself.

There is some limited ability for the government to impose narrowing laws. For instance, it can write laws that constrain its own conduct. Laws such as the Wiretap and Stored Communications Acts were adopted principally to limit the conduct of law enforcement agencies that might have otherwise been considered permissible under the Fourth Amendment. In other cases, most notably where concrete and particular harms are identifiable from information disclosure, the government may be able to proscribe such disclosures. This most often happens in heavily regulated industries that transact in sensitive information, such as health and financial information. For instance, the Stored Communications Act prohibits electronic communications services from disclosing the contents of users’ communications except under specific circumstances. Even then, however, the First Amendment limits the extent of such regulations. For instance, a law that prohibited the exchange of consumer data for marketing or promoting prescription drugs has been found to violate the First Amendment rights of pharmaceutical research companies and manufacturers who may have reason to use that data.Footnote ⁶

The discussion so far has focussed primarily on the federal government as a regulator. The federal government may regulate by writing laws (which happens through Congress); it also relies extensively on federal agencies to promulgate and enforce regulations. In the United States, these regulators are generally sector-specific. For instance, the Federal Aviation Administration regulates the airline industry; the Federal Communications Commission regulates communications industry; and the Department of Health and Human Services regulates the healthcare sector. Regulators such as the Federal Trade Commission have more general regulatory authority – but the courts and Congress have generally been sceptical of efforts by such generalist regulators to use their authority to regulate pervasive or cross-industry practices.

In addition to the federal government, the states play an important role in regulating these issues. For instance, every state recognises various ‘privacy torts’. These cover harms such as intrusion upon seclusion, disclosure of private information, presenting someone in a false light, and appropriation of likeness. This means, among other things, that enforcement generally occurs in the courts of the state where a given injury occurred. There may also be substantive differences in these laws between the states, including both whether specific causes of action are even recognised and the damages available for violating them. Each state also has its own constitution and legal system – there are sometimes important differences between these constitutions, both between the states and between the states and the federal Constitution. For instance, the federal Constitution has more onerous standing requirements than many state constitutions, which means that federal courts may not be able to recognise certain types of harms as allowing judicial remedies, whereas state-level courts may be able to adjudicate those same claims.

More recently, many states have adopted specific statutes that may relate to FRTs. Illinois’s Biometric Information Privacy Act (BIPA), discussed in Section 15.4, for instance, directly affects the use of FRT and has caused firms such as Facebook to alter the services they offer to individuals located in that state. State-level laws create a complex set of implementation issues, including compliance costs and difficulties, especially where the specific requirements of a law may not be clear at the time it is enacted and the need (at times) to comply with contradictory requirements between state laws. The relationship between state and federal law can also be uncertain. In many cases, the existence of a related federal law will pre-empt state laws – it can even be the case that the non-existence of a federal law or regulation can prevent the adoption of a state law. While these issues are foundational to the operation of any legal system, as modern technology has increasingly brought state-level regulations into tension with those of other states and the federal government, there has been a surprising amount of debate in the United States as to how they ought to play out.

Extra-regulatory tools play a significant role in governing technologies such as FRT in US law. Such tools include mechanisms such as self-regulation and self-regulatory organisations, and executive regulatory tools such as government procurement policies. Self-regulation comes up in many contexts. Legally, it is closely related to consumer protection law: self-regulation often requires firms or industries to publicly disclose governance principles and to implement them in a binding way. A failure to do so might be the basis for liability based upon unfair or deceptive practices claims. Self-regulation can also be based upon the threat of legislation or even mere investigation: Congressional hearings into a firm’s or industry’s business practices can be disruptive or costly.

The use of procurement policies as a regulatory tool draws from the government’s role as a large purchaser of goods and services, including of technology products and services. Government entities can decide which goods and services to purchase without the need for legislative or regulatory authority. For instance, the president or a local government entity can often issue a policy directive that prohibits law enforcement from using certain technologies (such as FRT) or that directs how they may be used (such as for locating missing persons but not for tracking criminal suspects). Because the federal government is one of the largest purchasers of goods or services in the country (even the world), these policies have the potential to shape entire industries. A decision to use, or to not use, certain types of technology can cause private industry to invest billions of research and development dollars to develop technologies that meet those needs.

The brief discussion here offers a capsule summary of many aspects of US legal institutions that are relevant to regulation of FRT. It is far from a comprehensive introduction to US law. But for readers unfamiliar with these institutions it introduces several important idiosyncrasies and provides context for the discussion that follows – and for all readers it begins to develop themes that will be seen in the remainder of this chapter.

15.3 Federal Regulation of Facial Recognition Technology

Federal FRT regulation is still nascent in the United States. Administrative agencies have played the biggest role so far, approaching the issue through standard-setting and existing consumer protection regulation. For instance, since 2017, the National Institute of Technology and Standards (NIST), a non-regulatory agency under the auspices of the Department of Commerce, has developed standards for absolute and comparative accuracy of facial recognition algorithms and publishes results for software available through commercial vendors.Footnote ⁷ And these federal standards inform state approaches. For instance, Virginia allows its police to use only facial recognition software that performs well according to NIST standards.Footnote ⁸

The Federal Trade Commission, the US consumer protection agency, published a set of ‘best practices’ regarding the use of FRT in 2012.Footnote ⁹ Publications such as these ‘best practices’ may inform future Commission activity, but do not constitute legally binding rules. More recently, the Commission has enforced general consumer protection principles against at least one software company for misleading consumers about when and how facial recognition software would be used on photo and videos uploaded to the app. In 2021, the Commission settled with a photo app company called Everalbum for allegedly only using facial recognition after affirmative consent from users, when, in reality, the company automatically activated the feature.Footnote ¹⁰

The US Congress has begun debating legislation that would regulate both the US government’s own use of federal regulation technology, as well as private use of regulation technology. But the fates of each of these bills is far from certain. In June 2021, Senator Markey proposed a bill that would regulate the federal government’s own use of facial recognition.Footnote ¹¹ The bill, which was not enacted into law but was reintroduced again in 2023, would prohibit any federal agency from using FRT, or information obtained from any such technology, without specific approval from Congress.

Congress also commissioned a study of the federal government’s use of FRT by the Government Accountability Office, which was published in August 2021.Footnote ¹² Most of the agencies used some form of facial recognition to help ensure the digital security of agency devices. Six reported using the tool for law enforcement purposes and five for security purposes, including live monitoring of locations.

The Internal Revenue Service (IRS), the United States’ taxation authority, came under bipartisan scrutiny in 2022 for using FRT to verify the identities of taxpayers online. Lawmakers criticised the agency’s use of the tool as intrusive and requiring taxpayers to sacrifice privacy for data security. Advocates criticised the tool’s potential for bias.Footnote ¹³ The IRS eventually reversed its plans and now offers an identity verification tool that does not involve facial recognition software. But even after this controversy, other federal agencies, including the US Patent and Trademark Office, are still moving forward with plans to use the same software.Footnote ¹⁴

The US Congress has recently turned serious attention to a potential federal privacy regulation bill that would cover many contexts, including facial recognition.Footnote ¹⁵ The bill’s political future is uncertain, but the proposed language would place restrictions on the purposes for which companies could collect certain data, including facial recognition data, requires privacy policies, requires consent from consumers, and prohibits forms of algorithmic bias. Again, this bill would apply to FRT, but also to much wider categories of data. Similarly, the Federal Trade Commission (FTC) has recently announced a potential proposed rulemaking relating to ‘Commercial Surveillance and Data Security’ in which the Commission is considering, among many other issues, ‘limiting commercial surveillance practices that use or facilitate the use of facial recognition, fingerprinting, or other biometric technologies’.Footnote ¹⁶ As with legislative proposals, the future of this potential rulemaking is uncertain.

15.4 Regulating Facial Recognition Technology in the United States

States and localities have a primary advantage over the federal government when regulating new technologies: They can usually get regulations on the books faster. And, indeed, states and localities have taken an interest in regulating FRTs, but these non-federal approaches have been varied and fluid. FRT has many uses, so regulatory approaches target a similarly broad span of conduct. Some states regulate government or law enforcement use of FRT.Footnote ¹⁷ Some only regulate a sub-set of government use, such as banning use of facial recognition on drivers’ licences or on police body cameras.Footnote ¹⁸ Other states regulate the technology only as applied to vulnerable populations, although the efficacy of the laws varies.Footnote ¹⁹ Yet other states regulate commercial applications.Footnote ²⁰

Illinois’s BIPA (2008) was one of the first state regulations of commercial use of FRT, although the bill encompasses more than just facial recognition. Before a private entity collects biometric information, the law requires (1) notice to the consumer, (2) informed consent from the consumer, (3) written policies about retention and destruction, and (4) limits retention of and profit-making from that information.Footnote ²¹ Suing under BIPA, consumers reached a landmark $650 million settlement against Facebook for using FRT on photos uploaded to the site without consumer consent.Footnote ²² In another major lawsuit, plaintiffs sued Clearview AI, a company that provided facial recognition software to law enforcement agencies and sector companies, under BIPA. Clearview attempted to argue that its activities – selecting and curating facial images – were protected under the First Amendment in the same way a search engine’s results might be.Footnote ²³ But the suit settled, and the terms of the settlement prohibit Clearview AI from selling its database to most private companies.Footnote ²⁴

Other states have comprehensive privacy laws that cover facial recognition data. California’s comprehensive privacy law, the California Consumer Privacy Act, for example, applies to facial recognition data, requiring companies to conform to certain obligations, including giving the consumer notice, access, and the right to have the data deleted.Footnote ²⁵ Texas and Virginia also have some of their own state privacy laws that apply to biometric information.

State laws that allow consumers to sue companies face two possible hurdles as federal regulation catches up. The first is pre-emption, which is when a new federal law essentially is substituted for a state law on the same topic. Pre-emption could happen if federal legislation regulating FRT is passed. But the most recently proposed federal privacy legislation, the American Data Privacy and Protection Act, would not pre-empt state laws that solely cover FRTs.Footnote ²⁶ BIPA would also remain un-pre-empted in a special carve-out.

State facial recognition laws that end up challenged in federal court – which could happen when a state resident sues a company that is based in another state – face a standing problem. In US law, standing refers to one’s legal ability to bring a suit. To have standing, a person must typically have suffered a concrete injury. Under BIPA and other state FRT laws, an injury might be defined as use of biometric data without consent. Federal courts have split on whether such an injury is concrete enough to satisfy the federal requirements for standing.Footnote ²⁷

State regulation of police or other government actors within its own borders will not face pre-emption or standing problems. Vermont’s law regulating police use of FRT, as one of the most comprehensive such laws, provides an example at one end of the spectrum of regulation. Vermont’s law is straightforward: With one exception (child sexual exploitation), until otherwise approved by the legislature, police may not use FRT or information derived from such technology.Footnote ²⁸ But Vermont is an outlier: At the other end of the spectrum, some states, such as Oregon, only regulate certain police or government use of FRT. For example, Oregon’s law prevents FRT from being used in conjunction with police body cameras.Footnote ²⁹

State regulation of this technology continues to be fluid. Many state and local regulations have taken the form of moratoriums with sunset provisions, merely delaying an ultimate decision about regulation until a future date. Others have already added exceptions to their bans, as in the case of Vermont. The Vermont legislature added an exception to its ban on police use of FRT in all circumstances.Footnote ³⁰ Now, police can use the technology in cases involving sexual exploitation of children.Footnote ³¹ And Virginia repealed its de facto ban on police use of the technology a little more than a year after the ban was passed.Footnote ³² As of July 2022, police in Virginia can use facial recognition software in certain investigatory circumstances, with ‘reasonable suspicion’, and only if the software achieves an accuracy score of at least 98 per cent (measuring true positives) on NIST metrics, across all demographic groups.Footnote ³³ The new Virginia law demonstrates the interplay between state and federal regulation: A state law uses a federal standard to guide the technology’s use within its borders.

States, cities, and other localities have also enacted regulations on FRT use within their borders. These efforts are part of a broader trend in localities regulating the use of law enforcement technology. Advocates argue that the benefits of such governance include expanded democratic control over technology, improved responsiveness to changing technology, and more timely governance than post-hoc rules developed through challenges brought through criminal litigation.Footnote ³⁴ At least sixteen localities throughout the United States had passed facial-recognition specific regulations as of July 2022, with others having comprehensive police surveillance regulations that also apply to facial recognition.Footnote ³⁵

Both states and localities can regulate technology such as facial recognition in ways and at speeds that the federal government cannot. And experimentation with regulation of such technology at state and local level demonstrates ways in which these governance units are the laboratories of democracy. At the same time, the impact of these laws is limited to the boundaries of states and localities, affecting fewer people than federal regulation would. And states and local governments have their own types of problems with interest capture, raising concerns that, for instance, large companies might be able to outgun local privacy advocates. Local and state regulations are also much more easily reversible than certain federal regulations, as we have already seen with facial recognition regulation in some states and cities.

15.5 Issues on the Horizon

Prediction is an always-fraught, if often necessary, endeavour. When it comes to predicting the future of FRT, and regulation of FRT, in the United States, it is more fraught than ever. US legal and political landscapes today are tempestuous, perhaps nowhere more so than where they relate to technology. There has been growing concern about technology in recent years on both the political left and the political right – albeit animated by very different concerns. At the same time, recent judicial decisions have made the prospects of regulation less, not more, likely. A number of issues relating to FRT that are on the horizon are identified and discussed here.

We start with topics that are most likely to be discussed but that also seem least likely to actually translate into action: Federal legislation or regulation intended to broadly limit or even prohibit the use of FRT.Footnote ³⁶ Such legislation is unlikely to come to pass in the United States without a strong bipartisan coalition supporting its adoption. Given the potential for abuses of FRT by the government and the American tradition of scepticism of government power, an outside observer might think this coalition would readily manifest. But there is strong countervailing support for law enforcement and ‘law and order’ policies. Narratives about the use of FRT to find missing children and track dangerous criminals – whether substantively valid or not – are likely to have great valence in policy discussions and make it unlikely that a necessary coalition will be able to form.

The dynamics are somewhat different when it comes to the potential for administrative regulation of FRT. Regulatory agencies enjoy some insulation from the political process: Congress has already delegated authority to agencies that may empower them to adopt regulations. The political question is therefore more limited to whether an agency’s leadership is interested in adopting given regulations. In the case of the current FTC, for instance, it is overwhelmingly clear that the agency is interested in adopting rules that could address the use of technologies such as FRT: as discussed previously, the Commission has recently issued an Advance Notice of Proposed Rulemaking relating to Commercial Surveillance and Data Security practices, which includes some consideration of rules that could limit, or at least affect, the use of FRT in the United States.

However, while the FTC may have the political will to adopt such regulations, it is less clear whether the courts will hold that it has the legal authority to do so. Recent trends in US administrative law have been hostile to expansive claims of authority by federal agencies, especially when adopting regulations that would affect entire industries or areas of commerce.Footnote ³⁷ It does seem likely that the FTC would be able to adopt narrow rules that prescribe specific, and likely modest, requirements governing the use of FRT; it seems less likely, however, that the courts would uphold any broad regulatory moves that FTC makes, especially were they so broad as to proscribe the use of FRT or similar technologies.

Looking beyond the borders of the United States, questions will likely arise about whether US law can be harmonised with, or otherwise show comity for, FRT regimes adopted in other countries. This will most likely come up in the context of European regulations. The relationship between US and European regulations is likely to follow much the same trajectory as we have seen in the context of privacy regulations – most notably the challenges to the Safe Harbor and Privacy Shield in the Schrems litigation. To the extent that European regulations are based in European conceptions of fundamental rights, those regulations are likely to conflict with US regulations; and conversely, US regulations based in the First and Fourth Amendments are likely to conflict with European regulations.

The examples here are all likely to dominate discussion about FRT, but also seem unlikely to prove viable pathways for such regulation. This does not mean, however, that FRT regulation is unlikely. Indeed, as discussed in Sections 15.2 and 15.3, we are already seeing FRT regulation at federal and state level. And these are also where we are likely to see substantive debates over the scope, impacts, and implementation of such regulations. Such regulations, and debates, are likely to focus on government use of FRT, government access to information collected through private FRT systems, specific uses of FRT or FRT-related practices, and generally issues arising from the relationship between competing states’ laws and the federal regulations.

We are likely to continue to see governmental entities at federal, state, and local level consider whether, and under what circumstances, to use FRT. It is unlikely that there will be significant uniformity in approaches adopted. While most of these efforts will result from legislatures acting to limit the scope of their executive’s authority – for instance, by prohibiting the use of FRT by law enforcement, school systems, or other public entities – it is also conceivable that some states could find their hands forced. State constitutions embody myriad conceptions of privacy. It is certainly possible that use of FRT by governmental entities may be deemed to violate constitutional privacy protections in some states, and it would not be surprising to see litigation pushing theories such as this in coming years.

To take an example, a federal judge in Ohio recently held that a public school’s use of proctoring software that uses a student’s computer’s video camera to ‘scan’ the room in which they are taking an exam may constitute a search of private property in violation of the Fourth Amendment.Footnote ³⁸ Similar claims could potentially be levelled at FRT systems: if courts hold that they enable pervasive tracking of individuals on an automated basis, they might be deemed to violate a reasonable expectation of privacy. Litigation challenging the constitutionality of such systems is at least possible, and probably likely, at both the state and federal level. Such restrictions would be unlikely to apply on government property or in government facilities, but could easily apply in private facilities or even in public places.

Government access to private FRT systems or data, discussed earlier, will also continue to be an issue in coming years. Here we are already seeing moves to limit government access to these systems, including requirements for judicial oversight of the processes by and circumstances in which law enforcement requests these materials. Of all efforts to regulate FRT in the United States, this is likely the least controversial and the most likely to continue to develop apace.

Limitations on government access often operate in practice by forbidding private entities from disclosing information to law enforcement. These regulations might not directly prohibit government use of information unlawfully disclosed to law enforcement (although, increasingly, they do). In this sense, they illustrate a general approach to regulating the private use of FRT: Most regulation will not prohibit the general development or use of FRTs; rather, restrictions on private entities will likely focus on specific use cases (e.g., disclosure of information generated by an FRT to law enforcement). One can speculate on a range of use cases for FRT that could be subject to regulation. For instance, the use of FRT to help evaluate job candidates could conceivably be regulated – depending upon the circumstance, such uses could even already run afoul of existing anti-discrimination laws.

A final set of issues on the horizon relate to the interplay between potential FRT regulations at the state and federal level. Importantly, these issues may arise in a range of contexts adjacent to FRT regulations. For instance, state-level privacy regulations that require disclosure of information collected about individuals, or minimisation of such information, would easily affect the technological and business practices of firms using FRT. If multiple states adopt conflicting FRT-related regulations there will be complex questions about how those regulations are applied in practice. And if the federal government also adopts other regulations – or if it deliberately decides not to adopt such regulations – there will be complex questions over whether the federal approaches to FRT pre-empt state-level regulations. On balance, this is all to say that regulation of FRT in the United States, to the extent that there are efforts to adopt such regulations, will remain fraught and unsettled for many years to come.

15.6 Conclusion

This chapter has considered the state of FRT regulation in the United States. The United States is not a monolith. It is a federation comprising a central federal authority along with more than fifty states and territories and hundreds of localities – all of which have legislative, executive, and administrative regulatory apparatuses. But they are also governed by the federal Constitution and share common foundational values. These values tend to limit the extent to which FRT can be regulated as a matter of law, as well as carrying a general disposition towards light-touch regulations.

This is not to say that FRT is, or will remain, entirely unregulated in the United States. For instance, the same disposition against government interference in private matters has already begun to result in regulations restricting the use of FRT by government actors. We are likely to see more of these regulations, including restrictions on private parties sharing access to their FRT systems with state actors (much in the same way that laws such as the Stored Communications Act prevent electronic communications services from sharing the content of communications with law enforcement without a court-issued warrant).

Outside limited circumstances, however, more expansive regulation of FRT in the United States is unlikely in the foreseeable future. While Congress and the Federal Trade Commission are both currently considering privacy regulations that might bear upon FRT to some extent, it is uncertain whether these efforts will be successful. Even if they are, those regulations will almost certainly face serious challenges under contemporary understandings of the United States Constitution, so will be subject to extensive and lengthy litigation. The US approach to FRT regulation is ultimately governed by broader US conceptions about privacy and regulation generally, which remain narrower than other jurisdictions and contested.

16.1 Introduction

Facial recognition technology (FRT) has been in use by the Brazilian public administration for various purposes since at least 2011. It has seen an uptick in the 2018–2019 period, with noteworthy implementations in Rio de Janeiro and São Paulo, among others.Footnote ¹ Nonetheless, there is no general legislation or sectoral regulation on the use of FRT – thus leaving unregulated both its general implementation and specific uses, such as for public security, public transportation systems, or identification.Footnote ²

This chapter aims at identifying vulnerabilities and opportunities posed by the use of FRT in Brazil, focussing on the current legislative and regulatory landscape. Thus, it shall attempt to describe the evolving legislative framework and assess its adequacy to deal with the risks to fundamental rights posed by such technologies.

To do so, we assume the reader’s prior knowledge of the basic functioning of facial recognition. This allows us to dive deeper into the literature concerning the adoption of FRT in Brazil (in Section 16.1), prior to reviewing the existing legislation (Section 16.2) relating to its deployment, especially in the context of law enforcement. A final section (Section 16.3) concludes with a brief analysis of this normative framework and puts forward a few suggestions on how to improve the national normative framework.

16.2 Implementations of FRT in Brazil

Information on the implementation of FRTs in Brazil is scattered. States, municipalities, and the federal government have all implemented projects utilising the technology, frequently without prior notice or consultation with civil society, which has hampered transparency and accountability. FRT is frequently introduced in the context of ‘Smart City’ programmes aiming at enhancing urban safety, in the absence of specific regulation and with no guidance from the National Data Protection Authority (ANPD) on how data protection impact assessments must be performed.Footnote ³ Alongside this, there are private implementations of FRT, which are even less transparent since there is no disclosure obligation of any kind.

Among several attempts at mapping FRT implementations in Brazil. the most recent one is Venturini and Faray,Footnote ⁴ which, drawing on access to information requests, search engines, and interviews with key actors, identifies six projects where facial recognition was being implemented. One is the emotion recognition contract for advertisement display purposes between Via Quatro, a private operator managing one of the subway lines at the city of São Paulo, and AdMobilize, an artificial intelligence (AI) analytics company headquartered in the United States.Footnote ⁵ Given the lack of notice and information over this contract, Idec, a civil society organisation acting in consumer rights issues, obtained a blocking injunction pursuant to a civil public action to uphold the rights of the users of the São Paulo subway system, where it argued that there was no consent for the collection and use of biometric data, no information on the functioning of the technology, the data processing, and its purposes, or the possibility to exercise data subject rights. Another project involved the subway administrator, Companhia do Metropolitano de São Paulo,Footnote ⁶ with the aim of installing FRT cameras for subway security in stations.Footnote ⁷

Two other projects involved surveillance of public spaces: one in the city of Campina Grande, in the state of Paraíba, and the other in Itacoatiara, in the state of Amazonas. The former involves FRT-enabled cameras running Facewatch installed during the city’s São João festival, beginning in 2019. The cameras were still being utilised in 2022, when they aided in the arrest of twenty-five people during that year’s festival and were expanded to two other cities in the same state (João Pessoa and Patos) via a command-and-control centre, totalling 1,600 cameras.Footnote ⁸ In Itacoatiara, a command centre was also created, with sixteen face recognition cameras for public security purposes.Footnote ⁹

Finally, the authors highlight the use of FRT by the Federal Data Processing Service (SERPRO), a public company, to confirm the identity of driver’s licence holders; and by SERPRO and the Social Service’s information technology company (DATAPREV) to confirm identity and provide proof of living for social security beneficiaries.

A paper focussed on FRT application in public security and police work reports on the use of these technologies in the states of Bahia, Rio de Janeiro, Santa Catarina, and Paraíba from March to October 2019.Footnote ¹⁰ Although the specifics (contracting parties, public procurement format, etc.) are not disclosed, the article contains insightful information on the efficacy of such systems, which led to 151 arrests in total. Particularly, out of forty-two cases where information on race was available, 90.5 per cent of suspects were black and 9.5 per cent were white.Footnote ¹¹ The research also analyses one specific case where FRT was applied for four days during the Carnival at Feira de Santana, a city in the state of Bahia, with an efficacy rate of less than 4 per cent.Footnote ¹²

A more recent work by Nunes and colleagues goes into more detail about FRT in Rio de Janeiro.Footnote ¹³ The researchers scrutinise a pilot-project involving the deployment of FRT in Copacabana, during Carnival 2019, which was later expanded to two more areas of the city. The two-phase FRT programme for public security was managed by the State Military Police Office (SEPM) in a partnership with Oi, one of the major telecommunications operators in Brazil. Firstly, thirty-four FRT-enabled cameras were installed in Copacabana during a ten-day period, and coordinated by four military policemen trained by Oi and Huawei.Footnote ¹⁴ This programme was extended for two more months in the same year in additional locations in the city, increasing the number of cameras to ninety-five.

The database against which matches were checked was fed by information from the state’s Civil Police Office (Sepol), the Department of Motor Vehicles (Detran), and the missing and wanted persons database. SEPM indicated that the data was encrypted, and information regarding persons identified via facial recognition was stored and made available to public security organs and criminal justice for purposes of planning, investigation, and enforcement, while false positives were immediately discarded by the system operator at the monitoring site.Footnote ¹⁵

During the first phase, 2,993,692 facial images were captured, with 2,465 face correlations being established between those and the database records. This amounts to a 0.082 per cent match rate. There are no specific numbers for the second phase alone, but in total, from March to October 2019, sixty-three people were arrested, two missing persons were located, and five vehicles were recovered thanks to the use of FRT.Footnote ¹⁶

Another study by Instituto Igarapé identifies forty-seven use cases of FRT in Brazilian cities from 2011 to 2019, spanning sixteen states out of the twenty-seven federal units composing the Brazilian federation.Footnote ¹⁷ Most instances (twenty-one) were related to public transportation – fraud prevention in free passes. These were followed by public security (thirteen cases), education (five cases), and border control (four cases).Footnote ¹⁸ Critically, the researchers report that ‘many of the publicly announced cases focus mainly on the expected efficiency and implementation and less so on informing results’.Footnote ¹⁹ This is a perception shared by Nunes and colleagues when analysing the aforementioned case of Rio de Janeiro, pointing to a lack of metrics enabling performance reviews and stressing several instances where clarifications are needed to evaluate the projects’ objectives and results.

Traditionally, Brazilian municipalities have adopted poor data governance practices, with sensitivity to personal data protection only kicking in after the applicability of sanctions in the General Data Protection Law (LGPD) in August 2021.Footnote ²⁰ From this perspective, a central concern regarding FRT use is the possible re-purposing of the personal data that has been collected, notably the sharing of such information with the government. For instance, in the case of FRT usage to prevent abuse of gratuity programmes in public transportation, it was revealed that the processed data may also be shared with the security forces ‘when requested’.Footnote ²¹ Similar concerns apply when FRT is used to monitor student attendance, such as a case in the municipality of Itumbiara, in the state of Goiás. Questioned by researchers, the municipal education office made assurances that the data were stored in the same device it was captured on and a prior Data Protection Impact Assessment had been done, although the assessment was not shared publicly or with the researchers.Footnote ²²

Despite existing assurances from public bodies responsible for FRT implementation, the risk of surveillance creep remains significant – not only involving a possible transferring of biometric data to third parties, but also the receiving of such data from third parties. In July 2021, for instance, the governor of Bahia announced the expansion of a FRT project from Feira de Santana to seventy-six other cities in the state, for a total of 4,095 cameras, on a R$ 665 million partnership with a conglomerate formed by Oi and the security tech company Avantia. In making the announcement, the governor also revealed an ambition to have private security cameras connected to the system, allowing for ‘banking agencies, shopping malls and condominiums […] to connect their cameras and deliver the movements and faces of passers-by to authorities’.Footnote ²³

16.3 Current Legislation, Regulation, and Governance

There is currently no specific law regarding FRTs in Brazil, whether for public or private ends, and whether in security, transportation, or any other area. Furthermore, there is no specific law or regulation framing the usage of AI systems in Brazil, although legislative efforts are being made. There is, however, a set of laws that regulate specific areas of FRT and can be used to build the basis for a regulatory framework; they are briefly explained in this section.

16.4 Discussion: Is the Existing Framework Adequate?

To assess if the existing (or proposed) legal framework regarding AI and FRT in Brazil is adequate for the protection of fundamental rights, it is first necessary to understand the risks associated with the application of these technologies. A brief discussion of these risks is presented here. Based on such an understanding, we then draw some necessary conclusions.

16.5 Conclusions

All in all, there are still substantial gaps in the regulation of AI and, consequently, FRT in Brazil, although a strong basis of principles is in place and there are important laws working to provide the necessary basis for the judicial protection of fundamental rights – as demonstrated by the Via Quatro case. A deeper issue with the implementation of these technologies is its scattered character – popping up in news announcements as sure techno-solutions to issues such as efficiency and public security. As discussed, this scattered nature is equally observed in the legislative scenario, with federal, state, and municipal norms and proposed bills creating a cacophony that ultimately impairs advancement of a strong position on the role that these technologies should play in society.

In this context, one major institution that might play an important role is the ANPD, which was given ample ground to not only control, but also guide data processing activities in Brazil. ANPD must embrace its role as a technical agency aimed at providing market and public implementations of innovative data-based technologies with the guidelines necessary to build technological solutions that respect fundamental rights and the means to innovate within those limitations.

Another institutional actor that could play a bigger role in the future is the Governance Committee created to implement the National Artificial Intelligence Strategy. This multi-stakeholder body is seated within the Ministry for Science, Technology, and Innovation, a crucial actor in promoting the full enjoyment of the benefits that may arise from science and innovation, especially in promoting economic and social development. However, this must be guided by a strategic vision that recognises the position that Brazil occupies in the process of recovering its industrial basis and catching-up with advanced economies.

Overall, the debate on FRT in Brazil has been marked by two movements that appear contrary to each other. On the one hand, reliance on FRT as a solution to immediate issues brings about hastened implementations that do not provide the necessary degree of transparency, accountability, proportionality analysis, and sensitivity to the fundamental rights to privacy and data protection. On the other hand, as mentioned in the opening of Section 16.2, civil society has reacted with increasing degrees of rejection of these technologies, reaching a generalised sentiment for the ban of FRT in the surveillance of public spaces. In the midst of these movements, existing and proposed norms seem to tackle some of the problematic aspects of FRT use, but fall short of giving a systematic and unified answer.

17.1 Introduction

Facial recognition technology (FRT) applications enjoy a staggering level of penetration in China. Valuing the technology’s function in facilitating social control and public security, the Chinese government has not only implemented it widely,Footnote ¹ but also used it to build a national surveillance architecture together with other mechanisms, such as the social credit system.Footnote ² When providing telecommunications, banking, and transportation and other services, an increasing number of state-owned enterprises record citizens’ facial data for their FRT systems.Footnote ³ FRT-empowered applications are also commonly adopted in the private sector,Footnote ⁴ for functions such as online payment, residential security, and hospital checking in.Footnote ⁵ The rapid development and wide adoption of FRT has made China a global leader in this field. In a recent round of the 1:N section of the US National Institute of Standard and Technology’s (NIST’s) Face Recognition Vendor Test, where algorithm providers compete for accuracy, the Hong Kong-based industry giant SenseTime came out on top, together with another China-based service provider.Footnote ⁶ SenseTime, as Asia’s largest artificial intelligence (AI) software company, has 22 per cent share of China’s computer-vision market.Footnote ⁷ Moreover, surveillance camera makers, such as Hangzhou Hikvision Digital Technology, Zhejiang Dahua Technology, and Megvii Technology, are also leaders in the industry and provide essential equipment for China’s pervasive implementation of FRT.Footnote ⁸

FRT has triggered serious privacy concerns in many countries, and China is of no exception. Although some commentators indicate that Chinese culture is more tolerant towards privacy violations than that of Western countries and many Chinese favour FRT because of increased security or convenience,Footnote ⁹ there have been extensive debates concerning the justification and proper scope of FRT adoption in the country. China has been working on developing a regulatory framework for FRT since 2020. Although this framework aimed to substantially enhance personal data protection, there have been increasing risks and challenges to protect citizens’ data in the FRT environment.

This chapter first introduces China’s legal framework regulating FRT and analyses the underlying problems. Although current laws and regulations have restricted the deployment of FRT under some circumstances, these restrictions may function poorly when the technology is installed by the government or when it is deployed for the purpose of protecting public security. We use two cases to illustrate this asymmetric regulatory model, which can be traced to systematic preferences that existed prior to recent legislative efforts advancing personal data protection. Based on these case studies and evaluation of relevant regulations, this chapter explains why China has developed this distinctive asymmetric regulatory model towards FRT specifically and personal data generally.

17.2 Regulating FRT in a Fishbowl Society

Given China’s over-arching national security drive built on a strong state-centric approach to data governance, its turn to strengthen personal information protection can be somewhat of a puzzle.Footnote ¹⁰ Heavy investment in FRT and the extensive use by the Chinese government in security applications often portray an invasively transparent ‘fishbowl society’ straight from Orwellian nightmares.Footnote ¹¹ Although the move to more robust protection of personal information appears to conflict with this perception, China has provided an interesting example regarding how authoritarian states balance their digital surveillance and the protection of individuals’ personal data. The case of FRT regulations and their enforcement is a particular case to illustrate the challenges of maintaining this balance in China.

17.3 Case Studies

In recent years, several FRT-related incidents have caught wide public attention and led to lively debates on the potential harm brought by this technology to society.Footnote ⁴¹ The most noticeable two cases were both raised by law professors challenging the justification of FRT use in citizens’ daily lives. Their outcomes, however, differed significantly. While one professor successfully convinced the court that enterprises could not unilaterally impose FRT on its consumers, the other failed to stop its pervasive use in Beijing metro stations.

17.4 FRT in the Surveillance State

Although the Civil Code and PIPL have advanced personal data protection in China, Sections 17.2 and 17.3 have revealed that FRT used by the public sector has not been subject to much limitation. The government can always justify such use for the purpose of public security. This asymmetric regulatory model is rooted in China’s unique political economy and regulatory philosophy.

First, the asymmetric regulatory model has been hugely influenced by China’s unique human rights values. The fundamentals of China’s human rights are different from those of the Western world. In the Western world, human rights were designed to protect individuals from state power from the beginning.Footnote ⁶⁵ However, China has viewed human rights as derived from the state, which reigns supreme over the individual.Footnote ⁶⁶ Consequently, China’s approach to human rights has been largely state-centric and emphasises individual responsibilities over individual rights.Footnote ⁶⁷ Privacy is no exception. China’s data protection philosophy is built on the view that data collection and analysis should be actively cultivated to boost state capacity to achieve a wide range of social governance objectives.Footnote ⁶⁸ Although the law provides citizens with considerable protection for their data privacy, it also creates numerous opportunities for the government to infringe upon citizens’ privacy. This understanding well explains why the public security interest, which is usually represented by the government, is always superior to personal data rights.

Second, Chinese law’s tolerance of FRT is closely related to its real-name registration policy. While anonymity is an important instrument to promote citizens’ free speech and to protect them against government retribution in many countries,Footnote ⁶⁹ the Chinese government has strictly enforced a nationwide ‘real-name registration’ policy to maintain social and political stability by eliminating digital anonymity.Footnote ⁷⁰ Under this policy, Chinese authorities have required users to register their real identities with internet and telecommunications service providers when using their services through various authentication mechanisms for easy traceability since the early 2000s.Footnote ⁷¹ The wide adoption of FRT has been a natural development to streamline the enforcement of the real-name registration policy because this technology has become the most efficient and effective identity verification technique.Footnote ⁷² Mobile users, for example, are required to register through facial scanning when buying new SIM cards.Footnote ⁷³

Third, China is an unparalleled surveillance state extensively using digital technologies to maintain its regime. Personal data, including facial data, is a key resource for the Chinese government to implement its ambitious national plans towards an algorithmically governed socialist state.Footnote ⁷⁴ The collection and processing of facial data has become increasingly essential for the government to build an effective surveillance system and to carry out economic plans, such as the ambitious ‘smart city’ initiative.Footnote ⁷⁵ According to a recent report analysing more than 100,000 government bidding documents from China, one FRT-based project in Fujian Province alone could produce more than 2.5 billion images to be stored by the police in the cloud at any given time.Footnote ⁷⁶ Given the extensive integration of FRT in public infrastructures, it is unlikely that the Chinese judiciary and government would easily declare such use illegal or unjustified. Similarly, it will be too costly for the legislators to roll back FRT deployment prescribed by other branches of the authorities.Footnote ⁷⁷

17.5 Conclusion

With the enactment of the Civil Code and PIPL, China has substantially enhanced its personal data protection. According to these two laws and the Judicial Interpretation on FRT, facial data is defined as sensitive personal information, and the deployment of FRT is more restrictive. The case of HSP represents the country’s determination to prevent the over-use of facial data in the private sector. However, China still faces serious challenges regarding FRT-related personal data protection under its asymmetric regulatory framework. While the use of FRT is increasingly regulated in the country, the regulatory restrictions can be invariably lifted for the reason of public security. Government agencies have invariably claimed this regulatory exemption for its massive FRT deployment. Moreover, the liability for the government’s abuse or misuse of personal data is quite insignificant compared with that that for private parties. This asymmetric framework has resulted from China’s unique human rights philosophy, the endeavour to enforce a real-name registration policy, and, more importantly, its determination to sustain a digital surveillance state.

18.1 Introduction

Scholarly treatment of facial recognition technology (FRT) has focussed on human rights impacts,Footnote ¹ with frequent calls for the prohibition of the technology.Footnote ² While acknowledging the potentially detrimental and discriminatory uses that FRT use by the state has, this chapter seeks to advance discussion on what principled regulation of FRT might look like. It should be possible to prohibit or regulate unacceptable usage while retaining less hazardous uses.Footnote ³ In this chapter, we reflect on the principled use and regulation of FRT in the public sector, with a focus on Australia and Aotearoa New Zealand. We draw on our experiences as researchers in this area and from our professional involvement in oversight and regulatory mechanisms in these jurisdictions and elsewhere. Both countries have seen significant growth in the use of FRT, but regulation remains patchwork. In comparison with other jurisdictions, human rights protections and avenues for individual citizens to complain and seek redress remain insufficient in Australia and New Zealand.

A note on scope and terminology. In this chapter we concentrate on FRT use by the state or public sector – by which we mean government, police, and security use. Regulation of private sector use is a wider issue that is outside the scope of this chapter.

18.2 Context

18.3 Options for Principled Regulation

Despite the considerable impact on individual and collective rights and interests, there is no discrete law governing the use of FRT in either Australia or New Zealand. Patently, FRT can be subject to existing legislative regimes such as privacy and search and surveillance, but unlike other forms of biometrics, such as fingerprints and DNA, the collection and processing of facial images remains largely unregulated.

In this section we canvass various options for principled regulation of FRT, at state and international level, with different degrees of specificity and latitude. These include proposals for domestic legislation, a case study of cross-national regulation, state-level principles, and self-governance.

18.4 Conclusion

While biometric technologies such as FRT have become more prevalent and more complex, and are being utilised in increasingly diverse situations, legislation, regulation, and frameworks to guide ethical use are less well developed.

This chapter has demonstrated how state agencies, particularly in policing and security services in New Zealand and Australia, have a broad discretion as to their use of FRT.

We suggest that FRT should be used only when predicated upon explicit statutory authorisation and following appropriate ethical review.Footnote ⁷⁸

Principled regulations should comprise a national statutory framework with a concomitant code of practice. Moreover, we recommend independent approval and oversight of the proportionality and necessity of operations. Jurisdictions should have a robust regulator, with the Scottish Biometrics Commissioner being a good example.

19.1 IntroductionFootnote ^*

Owing to advances around artificial intelligence (AI), such as computer vision and facial recognition, digital surveillance technologies are becoming cheaper and easier to use as everyday tools of governance worldwide.Footnote ¹ Typically developed by companies and governments in the Global North and tested in the Global South or on the ‘periphery’ of powerful actors,Footnote ² they are becoming key tools of governance in both democratic and authoritarian contexts.Footnote ³ As the AI Global Surveillance Index shows,Footnote ⁴ countries with authoritarian systems and low levels of political rights are investing particularly heavily in AI surveillance techniques such as advanced analytic systems, facial recognition cameras, and sophisticated monitoring capabilities.Footnote ⁵

AI surveillance offers governments two major capabilities. First, it allows regimes to automate many tracking and monitoring functions formerly delegated to human operators. This brings cost efficiencies, decreases reliance on security forces, and over-rides potential principal–agent loyalty problems.Footnote ⁶ Second, as AI systems never tire or fatigue, AI technology can cast a much wider surveillance net than traditional control methods. As Feldstein points out, ‘this creates a substantial “chilling effect” even without resorting to physical violence as citizens never know if an automated bot is monitoring their text messages, reading their social media posts, or geotracking their movements around town’.Footnote ⁷

Some scholars have observed the radical interdependence of the global AI development ecosystem, as only a few countries can afford to build their own local AI ecosystems.Footnote ⁸ For example, China is a major supplier of AI surveillance, with Huawei alone providing technology to at least fifty countries. France, Germany, Japan, and the United States are also major players in this sector.Footnote ⁹ As a consequence, the governance challenges around AI-enhanced technologies are inherently trans-national.

Indeed, the rise of AI accentuates several existing challenges for human rights law around (digital) technology. For example, such technology obscures the identity of the violator and makes violations themselves more invisible. This makes it much harder for citizens to hold duty bearers accountable.Footnote ¹⁰ It is also becoming much less clear whom citizens should try to hold to account in the first place. The current framework for addressing human rights harms inflicted by business entities is built on the distinction between public authority exercised by the state (which gives rise to a binding obligation to respect and protect rights) and private authority exercised by a company (which gives rise to a moral responsibility to respect rights). However, the distinction between the public and private spheres is becoming increasingly blurred, and as a result, it is less clear how human rights law is applicable.Footnote ¹¹ Instead, citizens must rely on states to take seriously their duty to protect individuals from harms by non-state actors, such as requiring private companies to institutionalise the practice of technology risk and impact assessments.Footnote ¹² It is clear that this is a formidable challenge in liberal democratic countries, let alone in authoritarian ones such as Morocco.

In this chapter, we focus on the role played by AI-enhanced surveillance tools in Morocco’s governance of cities and borders. We ask to what extent AI technologies are deployed in Morocco, and how they could reshape existing modes of public governance. We address these questions in two areas: urban surveillance and the control of migration at the Moroccan–Spanish border. We focus on the use of facial recognition technologies (FRT) in AI-enhanced cameras in particular, but we also address other technologies and other uses of AI, following a pragmatic approach that investigated where it was possible to access data. Indeed, AI surveillance is not a stand-alone instrument of repression, but complements existing forms of repression. As Feldstein observes, ‘it forms part of a suite of digital repression tools – information and communications technologies used to surveil, intimidate, coerce, and harass opponents in order to inflict a penalty on a target and deter specific activities or beliefs that challenge the state’.Footnote ¹³

The chapter is structured as follows. First, we outline the legal framework and governance context around Morocco’s use of AI technologies for urban and border surveillance. We then discuss our methodological approach, including some of the key limitations we faced during the research, before sharing our findings with respect to the use of FRT in the governance of cities and borders, respectively. Subsequently, we discuss AI-enhanced surveillance as an intrinsically transnational challenge in which private interests of economic gain and public interests of national security collide with citizens’ human rights across the Global North/Global South divide. We also reflect on the challenges and opportunities of monitoring human rights in the face of increasing deployment of AI-enhanced technologies in authoritarian governance.

19.2 The Legal Framework and Governance Context in Morocco

The Moroccan governance system has been described as ‘an entrenched neo-authoritarian system’.Footnote ¹⁴ Over the past decades, the monarchy has repeatedly weakened the political opposition by co-opting major parties into government. Human rights violations, lack of press freedom, and the harassment of human rights non-governmental organisations (NGOs) persist. However, while these deficiencies have attracted the attention of human rights organisations and press freedom watchdogs, they have not been properly taken up by inter-governmental actors. Quite the contrary: in the wider regional context, Morocco’s political stability has been viewed as an asset and is likely to become even more valuable (to the EU and United States), further insulating the regime from critiques of its civil and human rights records.Footnote ¹⁵

At the same time, Morocco is one of the highest performers in e-governance in Africa.Footnote ¹⁶ Morocco has more than 27 million internet users, or 75 per cent of its population,Footnote ¹⁷ and ranks high in the UN’s 2016 E-Government Survey in terms of e-participation, e-consultation and online service delivery as well as in its E-Government Development Index, a composite indicator used to measure the willingness and capacity of national administrations to use information and communications technologies to deliver public services. Indeed, Morocco’s new development model focusses on consolidating technological added value, and public administrations are increasingly making use of algorithms in online public services.Footnote ¹⁸ The combination of authoritarian rule and advanced use of e-government makes Morocco a particularly interesting case to study the role of AI in technologies in public governance.

At first glance, Morocco’s legal framework around privacy seems robust. The constitution contains an explicit protection of the right to privacy (Art. 24), there is a data protection law (Law n^o 09-08, promulgated in February 2009), and a data protection agency, the Commission nationale de contrôle de la protection des données à caractère personnel (CNDP). In addition, Morocco is a signatory of a number of treaties with privacy implications, including the Council of Europe’s Convention for the Protection of Individuals with Regard to Automatic Processing of Personal Data and its additional Protocols.Footnote ¹⁹ Furthermore, in 2018, Morocco joined the Open Government Partnership, an inter-governmental organisation promoting government transparency and citizen participation, and in 2021 it submitted its second two-year action plan, including twenty-two commitments that span a wide range of participatory, accountable, and transparent governance areas.Footnote ²⁰ Since 2010, the security sector has also been put on a clearer legal footing: private security was regulated, state security services were given statutes and mandated to better respect human rights, thanks in no small measure to the efforts of human rights organisations.Footnote ²¹

Yet, as HagmannFootnote ²² points out, some laws still lack implementation decrees. In addition, clientelist and political interests continue to influence whether and how penal provisions are implemented, human rights abuses investigated and reprimanded, or demonstrations and NGOs authorised and banned. Indeed, Privacy International notes that ‘there remains vast grey areas regarding the discretionary powers offered to judges and intelligence agencies’ when it comes to rules around legitimate breaches of individual privacy.Footnote ²³ This situation is worsened by the fact that the judiciary is not independent and that public scrutiny and democratic oversight over the work of intelligence services is lacking. In addition, the law on the protection of personal data (09–08) does not cover those data collected in the interest of national defence or the interior or exterior security of the state.Footnote ²⁴

Against the backdrop of this legal framework, technological tools are already integrated in everyday authoritarian governance and surveillance, especially at urban level. Traditionally based on a wide network of informants (car guards, local shop owners, informal vendors and beggars, etc.), mass surveillance is evolving through the use of technology. Phone tapping is common for listening to conversations, and more refined tools for surveillance have also been employed. For example, during the regional and local elections of September 2015, 30,000 mobile phone lines of candidates and regional or provincial party officials, in addition to local government officials and others, were reportedly tapped at the request of the Ministry of Interior.Footnote ²⁵ Another example is the Moroccan government’s use, at least since October 2017, of Pegasus spyware produced by the Israeli firm NSO Group, to surveil and attack human rights defenders.Footnote ²⁶ The general impression is that these technologies have allowed the bringing about of a more ‘surgical’ approach to the repression of dissent, one that systematically targets key figures,Footnote ²⁷ instead of the population as a whole.

Israeli companies are not the only provider of surveillance technology to Moroccan authorities. In 2015, there was a leak confirming that Morocco had bought the technology of Italian spyware company Hacking Team.Footnote ²⁸ In June 2017, an investigation by BBC Arabic and the Danish newspaper Dagbladet revealed that UK defence firm BAE Systems had sold mass surveillance technologies – called Evident – through its Danish subsidiary ETI to six Middle Eastern governments, including Morocco.Footnote ²⁹ There are also concerns that the European Neighbourhood Instrument may have been used to fund the training of Moroccan authorities in ‘telephone tapping and video recordings’ and ‘special investigation techniques for electronic surveillance’.Footnote ³⁰ More recently, there have been plausible but unconfirmed reports that the Moroccan police used COVID-19 mobile passport application check-ins to track the movements of citizens and identify those who disobeyed the rules of the state of emergency linked to COVID-19 measures.Footnote ³¹

In terms of surveillance in public spaces, protests still see a heavy deployment of security forces. Violence and arrests of demonstrators are still common and recent research shows how the strategic use of violence to clamp down on protest events can serve as a tool for regime survival.Footnote ³² However, whether and to what extent AI-enhanced technologies are used to respond to and control these events remains unclear. So far, most information on the use of AI-enhanced surveillance of public spaces has come from news reporting on the business arrangements and calls for tenders – either leaked or public – concerning the development of these technologies. This includes a series of articles detailing the deployment of video-surveillance technologies in Al Hoceïma, Agadir, Casablanca, Marrakech, and Meknes.Footnote ³³ In July 2022, the weekly magazine TelQuel also published an interview with a high-level police officer about the potential improvement achieved by the use of drones and AI in Casablanca, which is the only example of a public statement by a government official on the matter.Footnote ³⁴

With regard to border control, AI-enhanced technologies have been introduced by countries around the world not only to deter or stop irregular migration by surveilling borders, but also to serve as systems for tracking, controlling, and accelerating cross-border mobility more generally. In Morocco, for instance, this has resulted in the mounting of facial recognition cameras or procurement of thermal imaging cameras at its borders with the two Spanish enclave cities Ceuta and Melilla, largely funded by the EU.Footnote ³⁵ These borders are regularly mediatised as the main entry points for Sub-Saharan African migrants into Spain (and thus the EU), but they also experience a high daily flow of visitors and cross-border workers throughout the year. For Spain, controlling and preventing irregular migration into Ceuta and Melilla has been a hot topic since the 1990s, while Morocco considers Ceuta and Melilla as cities colonised by Spain, and they are therefore a regular cause of diplomatic crisis between the Moroccan and Spanish governments.Footnote ³⁶ Spanish media regularly accuse Morocco of attempts to put pressure on Spain by not effectively controlling borders, while Morocco invokes its unwillingness to play a gendarme role and calls for an integrated and participative approach to deal with the issue, including financial support from the EU.Footnote ³⁷

Managing the Spanish–Moroccan border is certainly big business given the considerable budgets allocated by the EU to ‘fight’ irregular migration and to ‘protect’ Ceuta and Melilla.Footnote ³⁸ For example, Indra Sistemas, a Spanish information technology and defence systems company, received at least 26.6 million euros across forty public contracts (twenty-eight without public tender), mostly from the Spanish Ministries of the Interior and Defence, for migratory control tasks including: the maintenance of the Integrated External Surveillance System (SIVE) of the Civil Guard, the installation of radars on the southern border and facial recognition at border posts, or the integration of the new ‘intelligent borders’ system.Footnote ³⁹ The French technology giant Atos also obtained at least twenty-six contracts from the Spanish Ministry of the Interior from 2014 to 2019, totalling more than 18.7 million euros, in order to repair and supply equipment for the SIVE. Similarly, from 2014 to 2019, the Government of Spain awarded French company Thales at least eleven migration control contracts (3.8 million euros in total), most of them to supply night vision systems and their respective maintenance services.Footnote ⁴⁰ The most recent deal over 4.8 million euros on the procurement of thermal surveillance cameras for the Moroccan Ministry of Interior has been concluded between the Spanish defence equipment company Etel 88 and the Spanish development agency FIIAPP.Footnote ⁴¹

In co-operation with mostly European tech companies, both Moroccan cities as well as Morocco’s northern border with Spain have thus seen the increasing deployment of and reliance on AI-enhanced technologies as governance tools. While Morocco’s legal framework around privacy and data protection has also been upgraded over the last decade, its limited implementation and the vast leverage of security and legal actors in interpreting the law, however, raise a host of challenges on the intersection between AI technologies and human rights. In the next section, we outline how we methodologically approached our research on urban and border surveillance, as well as some of the key limitations we faced.

19.3 Methodology

Francesco Colin and Issam Cherrat conducted the fieldwork for the urban and border surveillance cases, respectively, during the period March to August 2022. The fieldwork relied on extensive desk reviews, including various published and unpublished publications on the use of technology in urban and border governance, academic studies, grey literature from NGOs and state institutions, as well as the available press in French and English.

The other main component of the fieldwork was a set of nine semi-structured interviews on urban surveillance and twenty interviews on border surveillance with key stakeholders. Given the scarcity of information on the topic of the research, as well as the difficulty in accessing knowledgeable actors, we relied on snowball sampling. The interviewees included scholars, journalists, public officials, police officers, and civil society actors. Great care was taken during the research process to ensure the safety and anonymity of the interviewees and researchers. All interviewees were informed orally about the scope of the research and gave their consent prior to the interviews. Only two interviewees in the urban use case granted their permission to be recorded, while all of them asked to be quoted anonymously in the research outputs. Six interviewees contacted for the case study on border governance declined to have their interview mentioned in the study.

The study was severely limited by the broader security context in which the research was carried out. The sensitive nature of the topic – related to matters of national security and territorial integrity – as well as its relatively novel application in the Moroccan context made it complicated to access information and interviewees. All data revolving around surveillance practices is perceived to be the exclusive competence of Morocco’s security apparatus, and thus represents a subject on which one simply cannot ask too many questions. The limited information on the use of these technologies was extracted from calls for tender (CFT), to the extent that they were in the public domain. Such documents are accessible only when they get leaked by the press or in the case of privileged sources. In addition, many of these calls were still ongoing (or had been re-issued) at the time of writing. Finally, it was not possible to acquire first-hand information on the actual functioning of these technologies. Our attempts to establish direct contact with the staff of the intelligence services have been unsuccessful, and none of the interviewees had a direct knowledge of how these technologies are employed on the ground.

In addition to the issue of access, the overall general climate of repression (and associated fears of reprisals for speaking out) limited the fieldwork on both urban and border surveillance. Despite the precautions taken by the researchers, such as the exclusive use of secure platforms for communication, interviewees measured their words carefully when speaking about surveillance technologies. Representatives of private companies engaged in the deployment of these technologies were unwilling to participate in the research, saying that they did not want to jeopardise the relationship with the General Directorate for National Security (DGSN), the national police force, in the event of future tenders. In the case of border controls, several stakeholders refused to be interviewed once they learned about the topic, and access to information from government institutions was denied.

19.4 AI-Enhanced Technologies in Moroccan Cities

Across Moroccan cities, pilot experiences with AI-enhanced technologies are being developed for a plethora of applications – such as traffic management, monitoring of air quality, and energy efficiency, but also irrigation and waste collection.Footnote ⁴² However, generally speaking, in Morocco ‘there is still very little actual AI in smart cities’.Footnote ⁴³ As we noted earlier, most of the available data on the procurement of AI-enhanced technology such as facial recognition cameras is based on CFT documents, but information on its actual deployment, functioning, and use is extremely scarce.

Based on the desk review, it was possible to develop Table 19.1, which provides a schematic summary of the technology deployed in the urban context in Morocco.

Although Table 19.1 shows an impressive deployment of technology, especially for the city of Casablanca, these numbers still pale in comparison with other countries: while in Casablanca there are ‘only’ 0.74 cameras per 1,000 people, in the ten most surveilled cities of the world this ratio ranges from 62.52 to 8.77 cameras.Footnote ⁴⁴ Moreover, although the absolute lack of transparency surrounding these projects does not allow us to trace a clear timeline, we know that the deployment of high-tech surveillance technology has accelerated after the 2011 bombing at the Café Argana in Marrakech – as the attacked area was supposed to be covered by video surveillance, but apparently cameras were not working.Footnote ⁴⁵ Furthermore, Table 19.1 shows that the current wave of projects represents an extension of past deployments in some cities (Casablanca, Marrakech, Rabat) and the creation of new installations in others (Al Hoceïma, Agadir, Menkes). In any case, interviewees generally agree that these projects only represent the beginning, rather than the end, of such endeavours.Footnote ⁴⁶

However, there is no information available on the concrete way in which AI is employed in the analysis of images captured through these cameras. As one interviewee put it, ‘we know there is a computer at the central police station in Rabat, but god knows what goes on there’.Footnote ⁴⁷ CFT documents provide some useful information here: the CFT for the expansion of the surveillance system in the city of Casablanca runs to 389 pages and outlines the type of technology that needs to be provided, as well as its potential. It specifies that the cameras must be able to perform facial recognition tasks, that is, to be able to identify a target against a picture or a recorded photo, either in real time and on recorded footage (p.12 CFT’s Annex).Footnote ⁴⁸ The CFT for the new video surveillance project in the city of Al Hoceïma provides more details in terms of cameras’ (desired) capabilities: to compare the data collected via video surveillance against existing databases of pictures, to easily add pictures to databases based on live video feed, and to search for a specific person through an image added to the system.Footnote ⁴⁹ The capacity to identify a target on the basis of an image is the central function of face recognition systems in these contexts.Footnote ⁵⁰ However, the role of central security agents on the ground is still substantial: they would need to perform dynamic search functions (based on video metadata and on visual inputs), compile reports based on different data types and sources, and ensure co-ordination with police intervention on the ground.

In line with discursive shifts in the global surveillance trade,Footnote ⁵¹ the massive investment by Moroccan cities in AI-enhanced surveillance technology is presented as a shift from the ‘smart’ to the ‘safe’ city. Official discourse stresses the physical security purposes of the systems, such as catching accidents and thefts on camera. However, a small but increasing number of Moroccan civil society actors are raising concerns about the consequences of mass transmission of personal data to government entities in terms of privacy and individual liberties.Footnote ⁵² In addition, our study found that the legal framework attributes all control of local security issues to the local representatives of the Ministry of the Interior, rather than to elected local governments, limiting public oversight and accountability.

19.5 AI-Enhanced Technologies at Moroccan Borders

Unlike the unclear situation with regard to urban surveillance, it is known that the borders between Morocco and the two Spanish enclaves Ceuta and Melilla are progressively being transformed into ‘smart’ borders through the increasing deployment of AI-enhanced cameras and FRT. Yet, border control management is surrounded by secrecy in Morocco. Topics related to national security are treated with suspicion, and only fragmentary pieces of information are leaked to the press. It is almost impossible to know which firm or company has won a tender to install cameras or such equipment on the borders between Morocco and Spain, including the use of FRT cameras in its airports.Footnote ⁵³ In the words of an interviewee who did not want to be listed, ‘as a police officer, we do not ask about these things, I guess we do not have the right even, we are simply trained to use new technologies when they are deployed’. Therefore, for this section on the use of FRT at the Moroccan–Spanish border, we are relying on information from the Spanish side.

According to the Spanish Minister of Interior’s declaration in March 2022, Spain has modernised its entire technological systems at the border posts in Beni Enzar, Melilla, and in El Tarajal, Ceuta.Footnote ⁵⁴ This modernisation consists of the implementation of a fast-track system for cross-border workers, the installation of fifty-two posts for greater agility in the passage of people, and sixteen registration kiosks featuring the control and collection of biometric data. Furthermore, currently up to thirty-five cameras equipped with facial recognition systems are being installed between the entry and exit points of the borders of Ceuta and Melilla. The project is based on an entry control system with FRT, in which, in addition to the thirty-five cameras, there are four micro-domes,Footnote ⁵⁵ and a software platform to host the Live Face Identification System for the control of the closed-circuit television system. It is implemented by the company Gunnebo Iberia, a subsidiary of the Swedish world leader in security products, and Thales Spain, a subsidiary of the technological multi-national dedicated to the development of information systems for the aerospace, defence, and security markets.

Overall, the use of AI-enhanced cameras at the Ceuta and Melilla borders aims to shorten border control processing, enhance security at the crossings, and increase control over people and goods entering and exiting the border. The main problem at the Tarajal entry gate (Ceuta), for example, was that the poor existing infrastructures made it impossible to control the waiting line and to systematically track who enters and leaves through this passage. The poor infrastructure allowed for the smuggling of illegal goods and made it difficult to track whether minors entered Ceuta irregularly. The main objective of the deployment of new technologies is thus to monitor the number of people who enter and leave and to detect the number of people who do not return after a period of time. The technology used allows for flexible mobile facial scanning, that is, the inspection of people inside cars, trucks, buses, and on motorcycles or bicycles.Footnote ⁵⁶ It will also allow the implementation of ‘black lists’ during the passage of border control, showing personal information of the individual transiting through the border if they are registered on such a list. Deploying this technology, it is expected that some 40,000 facial readings per day can be carried out in Ceuta and 85,000 in Melilla.

Civil society actors have already drawn attention to the risks inherent to the use of those technologies at the Ceuta and Melilla borders: more than forty Spanish organisations and associations signed a statement rejecting the ‘smart borders’ project.Footnote ⁵⁷ They emphasised that the project’s ambition to ‘exercise greater security control through the use of artificial intelligence, by collecting biometrics, such as facial recognition, fingerprints, […] poses a risk of violating human rights’.Footnote ⁵⁸ They particularly highlighted that ‘the collection of biometric data for people who do not have a European passport is not in accordance with the principle of proportionality’.Footnote ⁵⁹ Indeed, as another civil society association highlighted, the Spanish–Moroccan borders risk being turned ‘into a laboratory for security practices’. They argue that with regards to the right to data privacy, ‘this will not happen at other borders such as Barajas-Madrid airport and will not happen with European citizens, but will happen at the borders where migrants cross in a state of extreme poverty, and it will happen with populations suffering from racism’.Footnote ⁶⁰

In terms of migration control, the deployment of FRT at the Moroccan–Spanish border is probably effective in controlling regular migration, for instance by facilitating and speeding up the circulation of individuals and cars, but less so when it comes to attempts at irregular migration. While it will inevitably make it harder for those migrants who need to reach Spanish territory in order to claim their rights to protection, that is, asylum seekers and unaccompanied minors, FRTs cannot predict when migrants will attempt to pass the fences of Ceuta and Melilla and are ineffective when thousands gather and decide to climb the fence simultaneously, such as the 2022 attempts in Melilla that caused the death of thirty-seven migrants.Footnote ⁶¹ Moreover, migrants adapt their border crossing strategies according to the technologies in place, for example in Fnideq,Footnote ⁶² where irregular migrants reach Ceuta’s shore by swimming when it is foggy and cameras cannot detect them. Lastly, the installation of ‘smart borders’ in the north of Morocco has (once again) redirected irregular migrants towards the longer, more costly and deadly migratory routes in the south of Morocco, where one can reach the (Spanish) Canary Islands through a perilous journey by boat.Footnote ⁶³

Despite Spain’s massive investment in these technologies, it remains unclear what the actual effects and outcomes of FRT used are on Moroccan–Spanish border dynamics since no official reports have been released yet. The impression from the field is that the new ‘smart’ border has slightly improved the quality of daily tasks, but that border crossings are still overwhelmed during periods of intense flux (during summer and national holidays). Furthermore, irregular migration dynamics seem to not have been affected by the use of new technologies, as migrants have adapted their strategies to cross the border.

19.6 Discussion

Despite the rapidly increasing use of FRT in Moroccan urban and border surveillance, public debate around these issues is still lacking in Morocco. In both cases, authorities justify the use of AI-enhanced technologies by the will to improve users’ experience and security.Footnote ⁶⁴ Between the high sensitivity of the data that is captured through these technologies and the generalised opacity with which it is treated, there are grounds to be concerned for the respect of citizens’ right to privacy.

From the two cases analysed, two cross-cutting issues emerge. The first one is the involvement of external actors, such as international donors, multi-national companies and foreign states, which makes AI-enhanced surveillance an inherently trans-national issue. External actors play a key role in the development and financing of FRTs, in their installation on the ground, but also in the (limited) monitoring of human rights protection frameworks. International donors provide funding for urban and border surveillance projects for instance, but they could also play a more active role in enforcing mechanisms for transparency in using such surveillance infrastructures. Yet, this is not always the case. For Casablanca’s first video surveillance projects (2015 and 2017), part of the funding came from a World Bank loan through a project to improve urban transportation. Although the World Bank raised concerns about the use of its funds, and demanded an audit that concluded the video surveillance system was not eligible for funding in the framework of their project, the project was still financed.Footnote ⁶⁵ Recently, the World Bank even approved an increase of 100 million dollars (in addition to the already committed 200 million dollars) to finance further development projects by the city of Casablanca.Footnote ⁶⁶

Similarly, the EU is extensively funding border control and surveillance technologies in Morocco, with little transparency concerning their use and few requirements in terms of associated human rights protection. For instance, the Moroccan DGSN acquired spying software from the Swedish firm MSAB and the US company Oxygen Forensic with funding from the Africa Emergency Fund, set up by the EU in 2015 for its ‘fight against irregular migration’.Footnote ⁶⁷ While this technology transfer project was implemented in the context of migration co-operation, the EU has no effective mechanism in place to prevent the misuse of such technologies for other repressive activities. More generally, although the EU has timidly tried to regulate the export of high-risk surveillance,Footnote ⁶⁸ it faces resistance from Members States.Footnote ⁶⁹ Additional rules were put in place in the revision of the EU’s Export Control Framework under EU Regulation 2021/821. But although export controls in the EU are becoming increasingly strict, EU Member States still often find ways to export these technologies that are deemed relevant for reasons of national security.Footnote ⁷⁰

The second cross-cutting issue that emerges from the analysis is that Morocco’s existing legal framework through which these projects are launched and implemented provides important obstacles to any kind of public oversight. Most of the tenders that accompany the development of these projects are circulated behind closed doors and not made public. Occasional leaks to the press are the main way in which these projects come to public knowledge. However, when tenders are unsuccessful, Moroccan law authorises the contracting authority to proceed through ‘over-the-counter’ contracts – which do not require any kind of publicity.Footnote ⁷¹ In other words, the companies that will implement these projects are selected directly by the contracting authority without a public tendering process, raising important questions in terms of transparency of the use of public funds. This will be the case for the 720 million Moroccan Dirhams project that will set up the new video surveillance system in the city of Casablanca.Footnote ⁷² Some interviewees also noted that when these tenders escape public scrutiny, they tend to be attributed to companies that have close ties to the regime.Footnote ⁷³

While the leaked CFTs provide some insights into which cameras are installed and how many, they leave Moroccan citizens and civil society in the dark as to how they will actually be used. Companies deploying these technologies argue that they have no control over their end-use. For instance, a source working for Huawei in Morocco highlighted: ‘if Huawei sells video surveillance products, it does not have access to what the final clients do with them, and does not participate in their installation’.Footnote ⁷⁴ Similarly, European companies seem impervious to ethical concerns for the eventual misuse of the technology provided. In the framework of these projects, they ‘do what they are asked to without too much resistance’.Footnote ⁷⁵

Other state institutions that should monitor the ethical implications of the use of AI-enabled surveillance technologies are not raising any concerns either. In its position paper on the digital transition, the Moroccan Conseil Économique, Social et Environmental defines AI development as a ‘national priority’, but it does not touch upon the use of AI in urban video surveillance. Similarly, the Conseil National des Droits Humains recently organised an international colloquium to discuss ethical implications of uses of AI, but it dealt with this topic from a purely academic perspective and avoided raising the issue of FRT-based surveillance by state authorities.Footnote ⁷⁶ Lastly, while raising the issue of the storage and analysis of personal data through facial recognition by private actors, the Commission nationale de contrôle de la protection des données à caractère personnel seemed untroubled by the use of the same technologies by security services and the exponential increase in the collection of personal data. In short, video surveillance is treated as the sole prerogative of the security apparatus, and so far public monitoring actors have avoided to engage directly with the topic. It seems that, implicitly and explicitly, public security should not be the public’s concern.

19.7 Conclusion

Our analysis shows that the umbrella argument of public security is applied not only to the use of AI-enhanced technologies in Moroccan urban spaces and at the Moroccan-Spanish border, but also to their deployment, oversight, and monitoring. As a result, the information on whether (and eventually how) high-tech surveillance technology is used is confidential, national security agencies are seemingly exempt from the monitoring of other state institutions, and independent actors are expected to trust that these institutions are acting in citizens’ best interest. This makes effective public oversight impossible, and amplifies the potential for it to be used for ‘surgical’ repression.

The lack of oversight is also nurtured by the absence of a public debate – and ostensibly of public interest – on the matter. An exemplary anecdote is that among the inhabitants of Casablanca, many think that the cameras around the city do not work, and are put there only to bring about an improvement in public behaviour.Footnote ⁷⁷ Kindling a public discussion on the securitisation of public spaces through high-tech surveillance was one of the ambitions of the TelQuel issue of July 2021, but so far this debate is still lacking.Footnote ⁷⁸ On the contrary, interviewees perceived Moroccans as being quite ill-informed about related issues of personal data protection.Footnote ⁷⁹

However, if the future plans inventoried in this chapter are indeed implemented, Morocco is rapidly advancing towards the implementation of AI-enabled technologies in urban and border surveillance, including FRT. It is clear that state institutions plan to use these technologies extensively, and the lack of (trans)national institutional oversight and public debate on the matter should raise concerns about the extent to which such implementation will affect citizens’ rights. Until the topic is picked up in public debate and diplomatic relations and reforms in the way this technologies are purchased and governed, Moroccan authorities will continue to conduct widespread AI-enabled surveillance without any oversight or accountability.Footnote ⁸⁰