Artificial intelligence (AI) and automated decision-making (ADM) tools promise money and unmatched power to banks and governments alike. As the saying goes, they will know everything about their citizens and customers and will also be able to predict their behaviour, preferences, and opinions. Global consulting firm McKinsey estimates that AI technologies will unlock $1 trillion in additional value for the global banking industry every year.Footnote ¹ Governments around the world are getting on the AI bandwagon, expecting increased efficiency, reduced costs, and better insights into their populations.

AI, apart from being a fashionable term that many organisations and researchers like using, denotes a set of related techniques and tools, ranging from machine learning, natural language processing, and computer vision to speech recognition and robotics.Footnote ² AI systems, incorporating these tools and techniques, on their own or combined, into hardware and software, have been described as ‘systems that display intelligent behaviour by analysing their environment and taking actions – with some degree of autonomy – to achieve specific goals’.Footnote ³ In this book, we are mostly interested in AI tools, which are a subset of ADM. The ADM technology, including AI, is used to make decisions that affect individuals as citizens or consumers. The degree to which humans are involved in such decisions may vary, but the ‘autonomy’ of the AI tools should not be overestimated. Ultimately, they are only tools, or means to achieve certain goals, which are set by humans.

Many of the ADM and AI tools, which governments are eagerly applying today, have been developed and experimented with for decades in the private sector. Technology tools, along with the broader managerial culture, are often transferred from corporations to government departments. Governments also often fund the initial development of these tools, which are later commercialised by corporations. These interactions are often shielded from public eye with the help of legal rules and market practices, which prevent us from knowing how the latest technology is used by the industry and what new tools are being developed, let alone how to regulate their use. For example, China’s Social Credit System has roots in automated credit scoring in the financial industry. Similarly, data-enabled fraud detection, used in the Australian ‘Robo-debt’ system, has long been a common practice in the banking industry. However, governments are not just the copycats in this relationship; they typically fund for the development of such ADM technologies, which are often shielded with trade secrecy laws.

This book explores the use of AI and ADM tools in the financial industry and public administration. Designing and applying AI and ADM tools in a close and mutually reinforcing relationship between the financial industry and governments – or what we call Automated Banks and Automated States – pose new threats to the accountability of public institutions and the regulation of financial industry alike. We understand Automated Banks as financial institutions investing in, and using, new technologies for increased efficiency and profits. Public administration, established to serve communities, closely follows the ‘banks’, adopting similar procedures, aims, and technology, resulting in Automated States. With an increasingly blurred line between public and private authority, this book aims at identifying new safeguards to ensure the rule of law, the protection of fundamental rights, and corporate and state accountability in the age of AI.

Financial industry, which has always been concerned with collecting data and analysing the information to be able to predict the future as accurately as possible, thus maximising their wealth (what we refer to as ‘money’), has traditionally been regarded as private actors, as opposed to public governments. But the power that financial industry has over people in most societies can be compared to that of governments. Governments and financial industry have always been collaborating closely, engaging in a mutually reinforcing causal relationship, exchanging information and managerial culture, and participating in policy-making. That relationship is now evidenced in development and deployment of the AI and ADM technologies, which is at the core of this book.

The aim of this book is to encourage a dialogue between ‘public’ and ‘private’ legal scholars on accountability, better regulation, new safeguards, and scrutiny of AI applications in the financial industry and public administration. Drawing on socio-legal and critical studies, the book provides a platform for discussion of the use of AI and ADM tools by financial industry and government agencies and, importantly, their close interaction in this space. With its conceptual focus, not being tied to a specific jurisdiction, and diverse authors from Australia, Asia, the United States, and Europe, the book will appeal to wide audiences in research, policy and regulatory spheres, as well as general readers interested in knowing the new dynamics of power and wealth enabled by AI.

The book is organised into three main parts.

Titled Automated Banks, Part I examines how AI and ADM are used in the financial industry. The four chapters in Part I analyse the benefits, challenges, and opacity brought about by the use of AI and ADM tools, exploring how legal systems and market practices in financial industry often prevent effective control, scrutiny, and accountability of Automated Banks. In Chapter 1, Associate Professor Teresa Rodríguez de las Heras Ballell sets the scene for the discussion of AI in financial sector, discussing the trends in AI regulation using the example of the most recent developments in the European Union. The author argues that increasingly extensive automation of the financial industry flourishes under technology-neutral regulation. At the same time, the application of existing rules may not always lead to desired outcomes such as prevention of misconduct and resulting harms. Professor Jeannie Paterson, Professor Tim Miller, and Henrietta Lyons then analyse in Chapter 2 the notion of ‘fintech innovation’. The authors demystify the kinds of capacities that are possible through the fintech technologies being offered to consumers, exploring the methods deployed by fintech solutions and interests behind them, in particular challenging a popular assumption that fintech innovation is of great benefit to marginalised communities with lower socio-economic backgrounds. Dr Doron Goldbarsht’s analysis in Chapter 3 shows how legal rules aimed at preventing wrongdoing in the financial system and the use of AI tools by inter-governmental bodies fighting money laundering and terrorism financing vest the industry with unprecedented power. The author sheds light on the benefits and challenges of adopting AI to mitigate risks of financial crimes. Part I ends with Chapter 4 which analyses how the opacity surrounding the use of AI and ADM tools by financial entities is enabled, and even encouraged by the law. The co-editor of the book, Dr Zofia Bednarz, and Associate Professor Linda Przhedetsky unpack how financial entities often rely on rules and market practices protecting corporate secrecy, as well as those incentivising the use of AI and ADM tools, showing how the legal systems allow the technology to become a shield behind which corporations can hide their consumer scoring and rating practices. The authors then explore potential regulatory solutions that could break the opacity and ensure transparency, introducing direct accountability and scrutiny of ADM and AI tools, and reducing the control of financial corporations over people’s data. Together, the chapters in Part I reveal the trends in the use of ADM and AI by the Automated Banks, how they are intertwined with the legal system, and lay the foundations for understanding their close interactions with the public sector, discussed in Part II.

Titled Automated States, Part II examines how AI and ADM tools are used in the public sector. In Chapter 5, Professor Terry Carney looks at the use of AI and ADM tools in welfare administration, and examines new challenges to the fundamental rights of the most vulnerable. Carney argues that existing safeguards for deployment of automated tools in public administration do not ensure decision-making values of transparency, quality, and responsiveness to the interests of citizens and communities. In Chapter 6, Paul Miller, an ombudsperson for community services in the Australian state of New South Wales, explores how the use of AI and ADM tools is shaping public administration, its impact on citizens, and how it affects scrutiny of public administration from a regulator’s perspective. Dr José Miguel Bello y Villarino then focuses on legal challenges that incorporation of AI tools will bring in the Automated State in Chapter 7. The author discusses the distinct nature of AI technology through an exploration of the dual role of public administration: a state that executes policy and a state that designs policy. In the final chapter of this part, Chapter 8, Dr Aitor Jiménez and Ainhoa Douhaibi analyse the use of AI and ADM tools in welfare and surveillance through the lens of critical race studies. The authors use the example of Catalonia (Spain) to argue that AI and ADM technologies employed to control and monitor immigrant populations are rooted in colonial punitive governmental strategies. Together, the chapters in Part II explore the origins of the use of AI by public administration, the challenges it poses to fundamental rights of the vulnerable and marginalised, and the role of the administrative law in the Automated State. Part II lays the foundations for critical discussion and regulatory proposals for future regulation in Part III.

Titled Synergies and Safeguards, Part III asks how money, power, and AI tools are entwined and what new safeguards could ensure that Automated Banks and Automated States are accountable to their customers, citizens, and communities. This part is opened by Professor Cary Coglianese, with his Chapter 9 focusing on AI tools fulfilling administrative law’s core values of expert decision-making and democratic accountability. Using the example of the US administrative law, the author points to a new challenge posed by a large-scale shift to the use of AI tools by government, ensuring that an Automated State is also an empathic one. Chapter 10 by Professor Ching-Fu Lin explores the blurred line between public and private authority in designing and applying AI tools. The author refers to important consequences resulting from ADM tools sorting individuals out, and citing the US case of Houston Federation of Teachers v. Houston Independent School District as a starting point, asks critical questions about the role of judicial review in scrutinising the use of ADM and AI tools. In Chapter 11, Associate Professor Tatiana Cutts critiques the broad consensus that human supervision holds the key to sound ADM and the resulting focus of academic and judicial spheres on ensuring that humans are equipped and willing to wield this ultimate decision-making power. The author argues that opaque ADM tools obscure the reasons for any given prediction, thus depriving the human decision-makers of appropriately weighing that prediction in their reasoning process and making a policy of using such opaque tools unjustified, however involved humans are along the way. In the concluding chapter of the book, Chapter 12, co-editor Dr Monika Zalnieriute offers a counter-perspective in arguing that the traditional emphasis on procedural safeguards alone – or what she calls procedural fetishism – is insufficient to confront the unprecedented power of the Automated States. The author argues that only by shifting our perspective from procedural to substantive, can we search for new ways to regulate the future of Automated States and keep them accountable to their citizens and communities.

Collectively, the chapters in the book challenge the ‘AI novelty’ discourse, prevalent in both the financial industry and public administration. The authors look at the Automated Banks and Automated States – rather than the technology itself – to specifically emphasise the interests and actors behind the ADM and AI technology. The common theme of the contributions is the focus placed on practices or behaviours, of both government administration and private corporations, that technology enables or encourages, pointing to the recent socio-technological developments being a continuation of, rather than a radical departure from, earlier practices and technologies used. The innovation, so often cited by financial industry and governments, is neither really new nor that beneficial, especially from the point of view of the end-users subjected to it.

At the intersection of money, power, and technology, it becomes clear how the systematic use of ADM tools, which are neither reliable nor transparent, widens the gap in power asymmetry between the Automated Banks and Automated States on the one hand, and their customers, citizens, and communities on the other. Opacity and proneness to bias emerge as the most prominent characteristics of AI tools, impeding scrutiny of the practices of public administration and industry, and their accountability. The chapters in the book suggest that public and private collaboration becomes a black box barrier to enforcement where proprietary ADM systems are used.

The artificiality of divisions between private and public sectors, as well as public and private law disciplines, is at the heart of this book, which brings together different disciplines, different points of view, different arguments and jurisdictions. The book illustrates how money, power, and AI lead to blurred distinction between private and public sectors. And while the technology and the behaviour it enables are not new per se, the authors convincingly argue that new rules, frameworks, and approaches are necessary to prevent harms that increasingly common deployment of AI and ADM tools ultimately leads to.