2. The limited number of systems subject to the most publicised provisions of the AIA

As the administrative lawyer delves deeper into the AIA and understands its complex architecture, it is also possible that he or she will be somewhat disappointed by the small number of systems subject to its most publicised provisions.

From the outset, the AIA only applies to AI systems, not to any automated system used by public authorities. The finally adopted definition of AI system (Article 3(1) AIA), coming from the OECD,Footnote ¹⁰ is still very broad and includes not only machine learning systems, but also advanced symbolic or knowledge-based systems (expert systems). However, Recital 12 states that this definition excludes “systems that are based on the rules defined solely by natural persons to automatically execute operations.” In any case, it is necessary to await the Commission’s clarification of the concept of AI system in the guidelines on the practical implementation of the AIA to be drawn up in the coming months (Article 96(1)(f) AIA).

The AIA’s flagship measure, the prohibition of certain AI systems, concerns only the eight types of AI systems or practices listed in Article 5 of the AIA (twice as many as before the European Parliament’s intervention). And some of them are subject to quite a few exceptions.Footnote ¹¹ This is particularly the case for the controversial real-time remote biometric identification systems in publicly accessible spaces, the use of which is allowed in many cases (Article 3(41), (42) and (44), Article 5(1)(f), (2) and (3), and Recitals 17, 19 and 32–41 AIA).

At the same time, high-risk AI systems (Article 6 and Annexes I and III), to which almost all of the AIA is devoted, represent a very small part of the systems that exist or will be developed in the future. Not only that.

On the one hand, high-risk AI systems linked to motor vehicles in Section B of Annex I (motorbikes, cars, aircraft, trains, etc.) are in fact excluded from the scope of application of the AIA, being governed solely by their specific rules, as provided for in Articles 102 to 109 AIA for each of them. These provisions merely amend the various pieces of legislation governing them by stating that, when technical rules relating to AI systems which are safety components of such products are adopted under these pieces of legislation, (only) the requirements which Articles 8 to 15 AIA impose on high-risk AI systems “shall be taken into account.”

On the other hand, Article 6(3) of the AIA has included the controversial “additional layer” of classification of high-risk stand-alone AI systems proposed by the Council and maintained by the European Parliament.Footnote ¹² It states that an Annex III system “shall not be considered to be high-risk where it does not pose a significant risk of harm to the health, safety or fundamental rights of natural persons.” It goes on to clarify that this is also the case where the system does “not materially influenc[e] the outcome of decision making” and lists four conditions that must be met in isolation or in combination.Footnote ¹³ The application of this exception is no longer subject to administrative authorisation, as proposed by Parliament, but is left to the assessment of the system provider. In return, the provider is required to document the assessment and also to register the system in the database of high-risk systems under Article 71 AIA, which will be referred to later (Article 6(4) AIA).

The other AI systems (the vast majority), those which are not prohibited or classified as high risk, are also subject to the AIA, as is clear from Articles 1(2), 2(1) and 3(1) of the AIA.

However, such systems only have to comply, where applicable, with the (important) transparency obligations of Article 50 AIA (information on interaction with AI systems – eg a chatbot –, on the artificial nature of the generated content – eg deepfakes – or on the exposure to an emotion recognition or biometric categorisation system).

Otherwise, only the voluntary codes of conduct of Article 95 AIA will apply to them. These codes do not necessarily exist and are obviously not binding.

The AIA has therefore not finally incorporated the Article 4a suggested by the European Parliament to be added to the Commission’s Proposal, which listed and minimally specified a number of general principles applicable to all AI systems, regardless of their dangerousness (principles of human agency and oversight, technical robustness and safety, privacy and data governance, transparency, diversity, non-discrimination and fairness, and social and environmental well-being). These principles and their brief description were very reasonable, but it is true that their practical implementation could give rise to doubts (even if Article 4a(2) indicated that they did not create “new obligations under this Regulation”) and fuel all kinds of national restrictions on the free movement of any AI system.

The most important consequence of subjecting these systems, which do not merit a high risk rating, to the harmonised regulation of the AIA is precisely that they cannot be subject to additional restrictions by Member States. Member States may not subject their development to obligations restricting their free movement throughout the EU.Footnote ¹⁴ Therefore, in relation to this huge range of AI systems that exist and, above all, will be developed in the future, it is more important what the AIA does not say than what it says: by subjecting them only to voluntary codes of conduct, the AIA grants complete freedom to their development.

This “regulatory shield,” designed not to hinder the development of AI in Europe, is arguably one of the most important effects of the AIA. It is paradoxical that the European legislator is accused of being over-regulatory when the AIA will serve, above all, to prevent the imposition of obligations on the providers of the vast majority of AI systems that will come onto the market.Footnote ¹⁵

Finally, following the spectacular emergence of ChatGPT and its competitors at the end of 2022, the AIA has also incorporated some provisions applicable to general-purpose AI models. They are defined in Article 3(63) as those that display a significant generality and are capable of competently performing a wide range of distinct tasks. These provisions are found in Chapter V (Articles 51–56) and apply irrespective of the AI system into which the model is eventually incorporated, be it a general-purpose AI system (which can serve a variety of purposes, such as ChatGPT – Article 3(66) AIA) or a specific-purpose AI system. The AI system based on such a model must comply with the relevant rules of the AIA, depending on the risk it presents. The AIA imposes obligations only on developers (“providers”) of general purpose AI models. While it cannot be excluded that public authorities will eventually develop such models,Footnote ¹⁶ it does not appear that they will do so very often, at least for the time being. Therefore, the following pages will not refer to Chapter V and will focus on the development and use of AI systems by public authorities.

3. The delayed entry into application of the AIA, which in the case of public authorities may be extended until 2030

Having noted the above, the administrative lawyer may also be disappointed that it will be a long time before the provisions of the AIA become fully applicable to the AI systems currently used by national and EU public authorities.

Except for banned systems, which can no longer be marketed and used in Europe as of 2 February 2025, all other systems benefit from a very generous vacatio legis, which runs until 2 August 2030 for high-risk systems aimed at public authorities in general, and until 31 December of the same year for the crucial large-scale IT systems managed by the European agency eu-LISA, listed in Annex X. Moreover, this vacatio legis covers not only existing systems, but also those to be placed on the market or put into service before 2 August 2026 – public authorities in general – or 2 August 2027 – eu-LISA – (Art. 111(1) and (2) AIA).

This long and debatable adaptation period,Footnote ¹⁷ which is probably designed to allow for the amortisation of the substantial investments already made or committed by the various public authorities in AI systems (such investments have been very high, eg in the case of the large-scale systems managed by Eu-LISA),Footnote ¹⁸ is in any case stricter than for the private sector, where high-risk systems pre-existing or brought into service within this 24-month period only have to be adapted to the AIA when they are subject to significant changes in their design (Article 111(2) AIA).Footnote ¹⁹

1. Prohibition of AI systems characteristic of authoritarian regimes

Although few in number, the systems prohibited by Article 5 AIA include practices currently common in the actions of police forces in authoritarian countries, such as the social assessment of citizens (social scoring, Article 5(1)(c) and Recital 31 AIA); predictive policing that determines the risk of a natural person committing a crime without being based on objective and verifiable facts directly linked to a criminal activity (Article 5(1)(d) and Recital 42 AIA, underlining its incompatibility with the presumption of innocence);Footnote ²¹ and systems that massively scrap facial images from the internet and surveillance cameras in order to identify natural persons through facial recognition (Article 5(1)(e) and Recital 43 AIA). The latter is a system that is also widely used by US and even European police forces.

As for the aforementioned real-time remote biometric identification systems in publicly accessible spaces, their exceptional use by police forces is subject to many safeguards, which do not exist in authoritarian regimes, and which must serve to avoid the feeling of mass surveillance that they may provoke in citizens. These include the need for the use to be permitted and duly regulated by national law, the need for a fundamental rights impact assessment and registration of the use of the system in the EU database to be carried out (Article 5(2) AIA),Footnote ²² and the need for each specific use to be authorised (ex ante or, in urgent cases, within 24 hours) by an independent judicial or administrative authority (Article 5(3) AIA). Post remote biometric identification systems are themselves classified as high-risk (Annex III, point 1(a), Article 3(43) and Recitals 17 and 95 AIA), and their use by police forces is also subject to prior or subsequent judicial or administrative authorisation, with some exceptions and additional conditions (Article 26(10) AIA).Footnote ²³

2. The importance of high-risk AI systems used by public authorities

It has been shown above that high-risk AI systems represent only a small part of those that exist and will exist in the market. Particularly narrow is the list of stand-alone AI systems, which are those that are not linked to any product and relate to the use cases listed in Annex III. These are the types of AI systems that are usually in mind when discussing the risks of algorithms, in particular the risks of opacity, bias and discrimination on the basis of gender, race, income level, etc.

It is the case, however, that the vast majority of systems included in this list affect only or mainly public authorities and are or may be of great importance to them and to the persons affected by their use. From the latter’s point of view, which is the most obvious, such systems can have a decisive impact on their lives by affecting their basic subsistence conditions (determining their possibilities for migration and asylum, access to education, vocational training, employment, loans, life and health insurance, essential public services and benefits, or the availability of critical infrastructure) and other relevant fundamental rights such as liberty, protection of personal data, effective remedy or to vote and to stand as a candidate. Seen from the perspective of public authorities, such systems may also become essential for the pursuit of public interests and the protection of the (other) persons and legal interests entrusted to them.

It is therefore imperative that such high-risk systems are well regulated and supervised, and that they function flawlessly, without bias or error.

To ensure that this is the case, the AIA even provides for a detailed regulation of regulatory sandboxes, which allow for the testing of innovative high-risk systems such as those that may eventually be used by public authorities in their day-to-day work (Articles 57–59 AIA).Footnote ²⁴ It even makes it possible to authorise tests in real world conditions outside these sandboxes, with the informed consent of the persons concerned or – in police matters and other areas of law enforcement – without the results having any negative effect on them (Articles 60-61 AIA).

The AI systems included in Annex III may, moreover, increase (or decrease) over time. In order to provide adequate flexibility in the regulation of a set of technologies that will evolve significantly in the coming years, Article 97 AIA empowers the Commission to adopt delegated acts under Article 290 TFEU to amend some of its provisions and many of its annexes, subject to scrutiny by the Council and the European Parliament. These include Annex III, whose use cases the Commission may reduce, amend or extend (only in the framework of the eight existing areas) in accordance with the substantive criteria set out in Article 7 AIA. It can also modify the conditions of application of the exception of Article 6(3) AIA discussed above (Article 6(6), (7) and (8) AIA).

3. The imposition of detailed conditions for the development of high-risk systems by public authorities and their contractors

In order to ensure the proper functioning of these high-risk systems, the AIA subjects them to numerous detailed technical requirements in Articles 8 to 15, which, as mentioned above, are to be further specified by the European standardisation bodies.

These requirements must be fulfilled by the public authorities themselves when they are the ones developing these systems and therefore have the status of providers of these systems. Not only that. The broad concept of “provider” in Article 3(3) AIA implies that public authorities acquire such status not only when they develop the systems through their own IT services, but also when they commission an external party to develop them tailor-made, either against payment (as would be the case for a contractor with whom a service contractFootnote ²⁵ is concluded) or free of charge (eg a university or a public research centre). The latter is important because it means that the contracting authority, and not the contractor, is then responsible for the fulfilment of the many obligations imposed on providers. Public authorities will only occupy the other major role provided for in the AIA, that of users (“deployers,” Article 3(4) AIA) of the system, when they simply acquire AI systems available on the market (through a supply contractFootnote ²⁶ ) or released under free and open-source licences (Article 2(12) AIA)Footnote ²⁷ and use them when performing their duties.Footnote ²⁸ The AIA thus solves a major problem that can arise in the procurement of AI systems by public authorities, and avoids dilution of respective responsibilities.

This is not the appropriate place to study these requirements in depth, and they do not present particularities when the system is developed by or for an Administration. The most important obligations imposed – somewhat softened with respect to the Commission’s initial Proposal – are the following: to establish a risk management system (to assess and minimise the risks that the system may entail – Article 9 AIA –); to use quality data sets for the training, validation and testing of the system (Article 10 AIA); to prepare the mandatory technical documentation (Article 11 AIA); to configure the system so that it automatically records the events that occur during its operation (“logs,” Article 12 AIA) and that it operates with a sufficient level of transparency in order to allow the deployers to appropriately interpret and use the output of the system (Article 13 AIA, requiring this transparency to be reflected in clear and detailed instructions for use); to design the system so that it can be effectively overseen by humans during operation (Article 14 AIA) and achieve an adequate level of accuracy, robustness and cybersecurity (Article 15 AIA).

In addition to ensuring that their high-risk systems comply with the above requirements, providers must also observe the numerous additional obligations specifically imposed on them by Articles 16 to 21 AIA. These include the obligations to have a quality management system (Article 17 AIA), to keep documentation (Article 18 AIA) and automatically generated logs under their control (Article 19 AIA), to carry out the above mentioned conformity assessment, to draw up an EU declaration of conformity (Article 47 AIA), to affix the CE marking on the system (Article 48 AIA), to take the necessary corrective measures and to inform the competent authorities and operators in case of malfunctioning of the system (Article 20 AIA).

Special mention should be made of the obligation imposed on them by Articles 16(i) and 49(1) AIA to register the AI system in the database of Article 71 AIA, a database managed centrally by the Commission, which will be publicly accessible, and which must contain the information detailed in Annex VIII. This is an important and widely demanded transparency measure, which concerns only Annex III high-risk systems (not Annex I), and which will allow the public to know the basic aspects of the independent high-risk systems in place in the Union. As mentioned above, providers are also required to register in this database those systems which, in their view, do not merit a high-risk status even though they fall within one of the Annex III areas.

The finally adopted version of the AIA has reduced the scope of such transparency by creating a non-public section of the database in which the systems of points 1, 6 and 7 of Annex III, in the areas of law enforcement, migration, asylum and border control (Article 49(4) AIA), will be registered.Footnote ²⁹ This section will only be available for consultation by the Commission and the competent national (data protection) authorities, and contains less information than the public part of the database. Article 49(5) AIA has also ended up stating that the systems in point 2 of Annex III shall be registered at national level and no longer in the European database, which makes sense given the national-territorial basis of the critical infrastructure concerned.

4. The imposition of detailed conditions for the use of high-risk systems by public authorities

Particularly important for the purposes of automated administrative action are the many obligations imposed on public authorities when they use high-risk systems and are therefore in the role of deployers. These obligations are set out in Articles 26 and 27 AIA and are particularly intense in the case of Annex III systems. Some of them are only or almost exclusively required of public authorities (Article 26(8) and (10), Article 27 AIA).

The obligations apply whenever a high-risk system is used, irrespective of whether this is done in the context of a formalised administrative procedure aimed at producing a decision having legal effects (e.g. granting, refusing or revoking social benefits – Annex III, point 5(a) – or granting or refusing an asylum, visa or residence permit application – point 7(c) –), or in the context of a merely factual administrative action Footnote ³⁰ (eg when the dispatch of an ambulance or health care in the emergency department of a public hospital is refused – point 5(d) of Annex III – or the police are investigating a crime – point 6 –). In fact, the substantive approach of the AIA has the advantage that it is not necessary to qualify the type of administrative action in question, so that its provisions are applicable even if doubts may arise as to the legal or factual nature of that action (eg when a border control officer refuses entry to a foreign national).

On the other hand, the obligations of Articles 26 and 27 AIA must be observed both when the administrative action is fully automated and when it is only partially automated. As has been seen, the – correct – criterion used by the AIA (Article 6(3)) is that the AI system used materially influences the decisions referred to in Annex III, irrespective of whether they are taken by a human or a machine. This is crucial and deserves to be viewed positively, in view of the numerous studies confirming the tendency of humans to rely excessively on the results suggested by machines (automation bias).Footnote ³¹ By the way, Recital 61 states, in relation to the systems in point 8(a) of Annex III, that judicial decision-making “must remain a human-driven activity.”

The examination of such obligations, which raise many issues of interest, deserves a specific paper.Footnote ³² Here they will only be listed and minimally commented upon following the typical sequence of administrative decision-making.

Before using for the first time an AI system listed in Annex III (except those relating to critical infrastructure), the public authority must carry out an assessment of the impact that its use may have on fundamental rights (Article 27 AIA). This obligation, widely called for as a measure to prevent algorithmic discrimination, has been introduced at the request of the European Parliament, but with a much reduced scope compared to amendment 413 it adopted. In addition to greatly simplifying the content of the assessment to be carried out, the obligation to consult, for a period of six weeks, the competent authorities and representatives of persons or groups of persons likely to be affected by the system in question has been removed.Footnote ³³

Then, also before the first use of these same Annex III systems, public authorities (only public authorities) are obliged to register such use in the European database of Article 71 mentioned above (Article 26(8) AIA). This obligation also derives from the European Parliament’s amendments, although it has again been watered down. It is still necessary to include in the database a summary of the conclusions of the fundamental rights impact assessment previously carried out, but no longer to describe the aspects relating to the use of the system as required by the Parliament. The publicity of the systems of points 1, 6 and 7 of Annex III is also restricted here, in the same terms as set out above in relation to the registration to be carried out by their providers.

During the use of any high-risk system, public authorities, like any other deployers, must use it in accordance with the instructions for use and inform the provider and the competent authority about episodes of malfunctioning (Article 26(1) and (5) AIA). They must also entrust the necessary human oversight of the system to natural persons who “have the necessary competence, training and authority” (Article 26(2) AIA), which is particularly important in the case of public authorities – which have significant shortcomings on this point – and links to the essential AI literacy duty imposed on them by Article 4 AIA. They must also ensure that the system’s input data are “relevant and sufficiently representative,” keep the logs generated automatically by the system for at least six months, and inform workers that they will be exposed to an AI system, where this is the case (Article 26(4), (6) and (7) AIA).

It is particularly relevant that the version finally approved has also incorporated the obligation proposed by the European Parliament to inform in general natural persons exposed to Annex III systems that take or assist in taking decisions affecting them (Article 26(11) AIA). Although Article 26(11), unlike the provision proposed by the Parliament, does not specify the information to be provided, Recital 93 does: such information should include the intended purpose of the system and the type of decisions it takes or assists in taking, as well as the right to obtain a more detailed explanation.Footnote ³⁴

The latter right to explanation is contained in Article 86 AIA, in the section on remedies, and has also been included at the request of the European Parliament. According to paragraph 1 of this provision, any person who is affected by a decision which produces legal effects or significantly affects that person in a similar way (by having an adverse impact on their health, safety or fundamental rights), and which is based on the output of any of the systems in Annex III (except, again, those relating to critical infrastructure), has the right to obtain from the deployer “clear and meaningful explanations about the role of the AI system in the decision-making procedure and the main elements of the decision taken.”

Unlike the right to information, this right exists only in relation to particularly important individual decisions, similar to those which Article 22(1) of the General Data Protection Regulation (GDPR)Footnote ³⁵ prohibits (with many exceptions) from being taken in a fully automated manner. However, the fact that both the right to information and the right to explanation are linked to individual decisions taken in relation to Annex III use cases will mean that both rights will normally exist together, since most such use cases have a significant impact, as seen above, on the health, safety or fundamental rights of persons. Unlike the right to information, the right to explanation is no longer limited to natural persons and also applies to legal persons.Footnote ³⁶ Another important difference between the version of the provision proposed by the European Parliament and the version finally approved is that the latter no longer refers to the “right to request,” but to the “right to obtain” the explanation. The nuance is important and implies that Article 86 AIA obliges to provide the corresponding explanations “ex officio,” and not only when requested by the person exposed to the AI system.

Importantly, this right to explanation has to be related to the duty to state reasons for administrative decisions deriving from the fundamental right to good administration in Article 41(2)(c) of the Charter of Fundamental Rights of the EU (CFREU), from Article 296(2) TFEU and from the case law of the Court of Justice. When the public authority adopts a decision in one of the areas listed in Annex III, fully or partially automated, using an AI system, it will have to include in the obligatory statement of reasons for the decision a “clear and meaningful explanation” of the role that the AI system has played in that decision. This shall be the case whether or not the person concerned so requests.

The most relevant aspect of Article 86 AIA is, in general, that it presupposes that the high-risk AI systems used by deployers (in our case, public authorities) in the areas listed in Annex III must be properly explainable. This requirement, which already derives in general from Article 13 AIA – in particular from subparagraphs 3(b)(iv) and (vii) –, is very important and prevents or at least limits the use of opaque machine learning systems (known as “black box” systems), whose operation is not understood even by their own programmers, and which are the ones that have raised the most objections due to their lack of transparency.

Finally, Article 85 AIA, again introduced at the initiative of the European Parliament, recognises the right of any natural or legal person to lodge a complaint with the relevant market surveillance authority. This right is recognised “without prejudice to other administrative or judicial remedies,” such as the administrative and judicial remedies available at national and EU level against administrative action. This right of complaint is thus conceived as a complementary means of monitoring compliance with the AIA, which can be used by persons other than those affected by the use of the system, and which seeks the control of such systems and not the legal protection of the latter. It should be stressed that the version of the AIA finally adopted has not extended the high-risk qualification to AI systems used in the resolution of administrative appeals, as Parliament intended by including them in point 8(a) of Annex III (amendments 71 and 738).

5. Other relevant issues

Other aspects of the AIA that will also exert a powerful influence on adm-ADM include the following:

• The definition of the key concepts in this matter (Article 3 AIA, with 68 points).

• The design of the governance system responsible for ensuring the proper functioning of AI systems and models, merging the control system for product safety with that of data protection, and adding new specialised bodies at EU level such as the new European AI Office, already created within the CommissionFootnote ³⁷ (and lacking the independence demanded by the European Parliament), the European Artificial Intelligence Board made up of representatives of the Member States, the Advisory Forum and the Scientific panel of independent experts (Chapter VII AIA).

• The establishment of supervisory mechanisms (Chapter IX AIA).

• The delimitation of the significant penalties that may be imposed in this matter, of up to EUR 35 million or, if the offender is an undertaking, 7 per cent of its total worldwide turnover for the previous financial yearFootnote ³⁸ (Chapter XII AIA).

• The great importance given to the duty of confidentiality, for the protection of both public and private interests (Articles 74(13) and 78 AIA, limiting access to the source code even vis-à-vis market surveillance authorities and thereby conditioning the application of European and national rules on access to public documents).