3.3.1 Society-in-the-Loop

SITL (Rahwan Reference Rahwan2018) is a key concept upon which the MOM framework builds. We view SITL as the backbone of the framework for infusing the human-centric aspect into it. As we focus on evaluating – often subjective – unpredictable algorithmic behavior, we argue that human perception is the key to inspecting the effects of its generated outcomes while minimizing the possible risks of such systems and the harmful impact they might have on society. In addition, with SITL, we might identify at least some of the unknown unknowns we cannot determine prior to the investigation for biased and discriminatory behavior. Thus, individuals’ contradicting views and values are crucial.

Algorithmic social contract. Rahwan (Reference Rahwan2018) advocated for an algorithmic social contract derived from the notion of social contract. In the political philosophy domain, social contract is largely perceived as a contract by which the society, government or moral principles rely on agreements between voluntary agents for their existence (Seabright et al. Reference Seabright, Stieglitz and Van der Straeten2021). Influential political philosophers such as Hobbes (Reference Hobbes1651), Rousseau (Reference Rousseau2003, Reference Rousseau1964) and Locke (Reference Locke2013) have heavily contributed to the fundamental notions of the social contract theory. Although they have various disagreements, they also have their commonalities (Seabright et al. Reference Seabright, Stieglitz and Van der Straeten2021). Based on Seabright et al. (Reference Seabright, Stieglitz and Van der Straeten2021), some important commonalities are the quality of being human (referred to as the state of nature, revolving different qualities and fundamental rights [e.g., being equal]), the existence of a contract by which humans agree with each other forming a united force to decide in a way that is being considered representative to “the will of all” (referred to as the contract of association), and the humans’ voluntary surrendering of some individual liberty and the promise to obey the government (referred to as the government contract).

While many researchers and theorists claim that diverse values, ideals and settings would be beneficial in the context of the social contract, others debate this from the lens of its practical implications where their adoption is “unlikely to reflect the direct participation and consent of all concerned” (Jos Reference Jos2006). Thus, the existing literature on the social contract is still fuzzy due to the magnitude of disagreements within the community (Campos Reference Campos2019). Furthermore, Campos (Reference Campos2019) realized that such a contract “does not merely determine which acts are right and wrong but it also establishes what reasons and forms of reasoning are justifiable,” which is a fundamental point, especially in the context of weaving a dedicated social contract for exercising human oversight of algorithms.

Therefore, Rahwan (Reference Rahwan2018) described an algorithmic social contract as the contract by which an SITL can scrutinize algorithmic behavior by embedding the general will. As already stated, SITL consists of involved key stakeholders who share – sometimes conflicting – values and ideals to agree upon the tradeoffs between the different values that system behavior is expected to perpetuate. As a result, the algorithmic social contract consists of the aforementioned expectations along with the particular benefits the stakeholders should be focused on achieving with the use of the system and the corresponding costs they should pay for, in the face of the implications emanated by incidents of unexpected algorithmic behavior. Based on Rahwan’s suggestion to take into consideration the types of behaviors people expect from these systems, we argue that there is a need to examine these depending on the context and application of use. In addition, Rahwan (Reference Rahwan2018) emphasized the urgency of developing new mediums such as methods, metrics and tools to enable the public to articulate any expectations based on goals, ethics, norms and the social contract itself into these systems and, by the necessary programming, debugging and monitoring, enforce the algorithmic social contract between humans and algorithms.

Further to this, we envision an algorithmic social contract in which some core elements would be considered vital for its composition in our SITL-inspired framework. Although we elaborate further on those core elements in the next sections of this work, we sketch out the basic structure of the respective contract. To begin with, depending on the context and application of use, we need to be sure that diversity is in place, to mediate the fair and equal representation of a crowdsourced society of workers via DITL. In this sense, we not only need to consider national and international ethics and laws but also basic societal values, ideals and norms targeting the actions and effects of the algorithmic behavior. Developers must shift towards a human-centric worldview consisting of the aforementioned social artifacts. As Weizenbaum (Reference Weizenbaum1972) suggested, “the [algorithmic] revolution need not and ought not to call [hu]man’s dignity and autonomy into question that is a kind of pathology that moves [humans] to wring from it unwarranted, enormously damaging interpretations.”

Microtask crowdsourcing

There is a growing research community that strives to provide solutions to real-world problems by recruiting people (a “crowd”) to contribute to defined tasks with the aid of technology. Estellés-Arolas and de Guevara (Reference Estellés-Arolas and de Guevara2012) systematically analyzed the scholarly literature for different interpretations of crowdsourcing and concluded that:

Adding to that, Brabham lists the three key ingredients of crowdsourcing:

1. 1. An organization that has a task it needs [to be] performed,

2. 2. A community (crowd) that is willing to perform the task voluntarily,

3. 3. An online environment that allows the work to take place and the community to interact with the organization, and

4. 4. Mutual benefit for the organization and the community.

Moreover, in a review of the key design elements of a crowdsourcing initiative, Karachiwalla and Pinkow (Reference Karachiwalla and Pinkow2021) identified four dimensions, namely, the (a) task, (b) crowd, (c) platform and (d) crowdsourcer, which is aligned with Brabham’s list of the key ingredients of crowdsourcing (Brabham Reference Brabham2013). Karachiwalla & Pinkow’s main contribution is to deliver a blueprint for practitioners to utilize when designing and executing crowdsourcing projects.

Crowdsourcing falls into two main categories: microtask and macrotask crowdsourcing. Microtask crowdsourcing is a helpful instrument for combining human and machine intelligence (i.e., for improving the accuracy of an ML model). “Microtasks are relatively quick, simple, and repeatable activities that can be – and often are – completed by volunteers in parallel, without the need for specific training or specialist knowledge” (Ibáñez et al. Reference Ibáñez, Reeves and Simperl2020). On the other hand, macrotask crowdsourcing is a more complex procedure that requires much more time (even hours) to complete by individuals who are specialized in the corresponding domain and context of the task (Haas et al. Reference Haas, Ansel, Gu and Marcus2015; Ibáñez et al. Reference Ibáñez, Reeves and Simperl2020). To support our framework, we focus on microtask crowdsourcing, involving the public in the scrutinization of algorithms. In the following sections, we argue that recruiting individuals with specific characteristics (e.g., current occupation) might be helpful for the context and application of the use of the corresponding task. This maintains a more sophisticated crowd during a microtask crowdsourcing oversight process without eliminating the nature of microtask crowdsourcing per se (i.e., the speed of completing a task).

Numerous works on microtask crowdsourcing have contributed to this domain from various perspectives. The work of Gadiraju et al. (Reference Gadiraju, Demartini, Kawase and Dietze2015) focuses on distinguishing between trustworthy and untrustworthy workers’ behavior. Others focused on the psychological and sociological reasons behind workers’ participation in crowdsourcing (Deng & Joshi Reference Deng and Joshi2016). From an SE perspective, Zhen et al. (Reference Zhen, Khan, Nazir, Huiqi, Alharbi and Khan2021) investigated the effects of microtask crowdsourcing to enhance or support SE. In another systematic review, Zulfiqar et al. (Reference Zulfiqar, Malik and Khan2022) found a number of microtask activities from the workers’ perspective.

During the past years, there is an increased interest in crowdsourcing platforms (CPs) (Brabham Reference Brabham2013; Liu Reference Liu2020; Zhen et al. Reference Zhen, Khan, Nazir, Huiqi, Alharbi and Khan2021) as many commercial solutions have emerged. Some of the most common ones – that are also highly cited and used by the research community – are Amazon Mechanical Turk (MTurk),Footnote ¹² Prolific,Footnote ¹³ ClickworkerFootnote ¹⁴ and Appen’s Crowdsourcing Solutions.Footnote ¹⁵ These platforms offer a set of features that can aid in various tasks, ranging from simple customer discovery to labeling datasets and training ML models. As mentioned, previous work in microtask crowdsourcing combined HITL approaches to assess, monitor and improve systems’ performance. The benefits of HITL are reported in the crowdsourcing literature, especially when using microtasks, and particularly when ethical concerns present challenges from a technological perspective that cannot easily be addressed by automated solutions. The controversial work of Awad et al. (Reference Awad, Dsouza, Bonnefon, Shariff and Rahwan2020) is a representative example of crowdsourcing moral judgments on machines via HITL. The authors created a web platform called “The Moral Machine (MM)” to gather data on human perception of the moral acceptability of decisions made by automated vehicles faced with choosing which humans to harm and which to save. Moreover, they emphasized their belief that “social scientists and computational social scientists have a pivotal role to play as intermediaries between engineers and humanities scholars in order to help them articulate the ethical principles and priorities that society wishes to embed into intelligent machines” (Awad et al. Reference Awad, Dsouza, Bonnefon, Shariff and Rahwan2020). In a similar line of work, Nakao (Reference Nakao2022) used MTurk to study workers’ perceptions of fairness in a hiring process scenario where an HR department uses an AI tool found to discriminate against female job applicants. Nakao’s work underscores the need to embed HITL AI fairness perception and metrics when AI is used in diverse decision-making processes.

Microtask crowdsourcing is a vital ingredient for the MOM framework. By utilizing it, we can simulate SITL by convening a diverse crowd for exercising oversight in algorithms and monitoring its diversity in various aspects through DITL. Crowdsourcing could bridge the gap in time and cost when recruiting a diverse crowd, according to the needs of the oversight process. The ease of crowd pool availability and the capabilities and task types that can be facilitated make the MOM framework an important control tool for developers to assess their systems. Bearing in mind that each CP provides a different crowd community (i.e., based on region, ethnicity, age, occupation and more), the idea of combining those communities into a unified solution – under the MOM framework – will further empower developers’ toolbox.

Diversity

Another core component of the MOM framework is diversity. We should observe diversity from various perspectives, concerning different levels and aspects of it.

Although diversity can be depicted differently depending on the discipline and there is little consensus on the precise terminology, we likewise focus on the socio-technical notion of diversity derived by Drosou et al. (Reference Drosou, Jagadish, Pitoura and Stoyanovich2017). Drosou et al. (Reference Drosou, Jagadish, Pitoura and Stoyanovich2017) defined diversity as a concept able to characterize the quality of a collection of items and consists of various interpretations depending on the context and application of use. Another crucial aspect we need to consider depending on the context of use is the notion of novelty, which as the same authors (Drosou et al. Reference Drosou, Jagadish, Pitoura and Stoyanovich2017) mention is usually being used in specific applications to reduce redundancy. As a result, novelty-based diversity is defined with respect to what has been observed in the past, up to the current point in time.

An essential element of using the MOM framework is the diversity of workers, and as a result, the diversity of the crowdsourced SITL. Particularly, we are concerned with how personal characteristics correlate to or affect workers’ perceptions of algorithmic behaviors and their responses to the given tasks during the execution of the MOM framework when exercising human oversight. Therefore, we adopt the two types of diversity described in Giunchiglia et al. (Reference Giunchiglia, Bison, Busso, Chenu-Abente, Rodas, Zeni, Gunel, Veltri, De Götzen, Kun and Ganbold2021), Giunchiglia and Fumagalli (Reference Giunchiglia, Fumagalli, Mayr, Guizzardi, Ma and Pastor2017) and, Schelenz et al. (Reference Schelenz, Bison, Busso, de G¨otzen, Gatica-Perez, Giunchiglia, Meegahapola and Ruiz-Correa2021). Namely, we distinguish those two types into objectual and functional diversity.

Objectual diversity – referred to as “observable diversity” or “surface diversity” (Giunchiglia et al. Reference Giunchiglia, Bison, Busso, Chenu-Abente, Rodas, Zeni, Gunel, Veltri, De Götzen, Kun and Ganbold2021) – is the facet of diversity that applies to “observable” characteristics such as sex, race, ethnicity, national origin, age, membership in formal organizations (e.g., religious, political) or physical features. It can be seen as the set of common observable attributes that describe an individual actor at a superficial level. On the contrary, functional diversity (Giunchiglia et al. Reference Giunchiglia, Bison, Busso, Chenu-Abente, Rodas, Zeni, Gunel, Veltri, De Götzen, Kun and Ganbold2021) applies to less observable characteristics. Such characteristics include one’s technical abilities, role in an organization, socioeconomic and cultural background, personality traits, cognitive abilities and values. As these less observable attributes tend to characterize an individual actor at a deeper level of understanding, this type of diversity takes longer to recognize in others. Despite that it is usually what we exploit when considering online contexts – where prolonged interactions are not always supported – identifying functional diversity becomes even more challenging and complex. Considering both types of diversity is crucial for navigating through different views and values during the crowdsourced SITL oversight process.

The benefits of considering diversity during an oversight task have been reflected in previous research. For instance, in their crowdsourcing study for investigating with the inclusion of a number of predictors in algorithmic decision-making in the context of recidivism, van Berkel et al. (Reference Van Berkel, Goncalves, Hettiachchi, Wijenayake, Kelly and Kostakos2019) found evidence that more diverse groups tend to more closely align with the majority agreement. In fact, some researchers (van Berkel et al. Reference Van Berkel, Goncalves, Hettiachchi, Wijenayake, Kelly and Kostakos2019; Suresh et al. Reference Suresh, Gomez, Nam and Satyanarayan2021; Grgić-Hlača et al. Reference Grgić-Hlača, Lima, Weller and Redmiles2022) suggested that it is crucial to include workers with diverse interests and perspectives, as to consider the range of social consequences across domains of algorithmic processes, while others (Nakao Reference Nakao2022) argued that using crowdsourcing techniques can aid in recruiting diverse stakeholders and aggregating their views (e.g., about fairness).

Unfortunately, the literature lacks methods, metrics and tools that can aid in determining the diversity factors in such oversight tasks, especially those that might influence the workers’ perception of the observed algorithmic behavior. In this context, the scientific community has to prioritize understanding these diversity factors that could be crucial in fairly and adequately assessing algorithmic behavior in a beneficial manner that promotes the maintenance, co-existence and interaction with algorithmic systems in a societal environment.

Diversity factors. When we refer to diversity factors from the worker’s perspective, we refer to a set of attributes that characterize an individual in an objective (e.g., physical characteristics) or subjective (e.g., fairness perception) manner that can be distinguished into two main types, objectual and functional diversities. We argue that these factors are able to influence the workers’ perception during an oversight process in various ways. For instance, Grgić-Hlača et al. (Reference Grgić-Hlača, Lima, Weller and Redmiles2022) found that individuals with personal experiences that are closely related to the decision-making setting, assess algorithmic fairness differently compared to those who did not have such experiences. More specifically, the authors provide the example of having attended a bail hearing, which negatively correlates to the perceived fairness when using defendants’ juvenile criminal history information for making bail decisions. In this case, personal experiences might be a diversity factor worth considering and investigating. Of course, further work is needed to define a comprehensive set of diversity factors.

It is evident the diversity factors can have different effects and influences depending on the context and application of use. This is why, when designing an oversight task using the MOM framework, we suggest carefully choosing the factors that could be important for the respective oversight application, and the aims of the developer or organization that exercise oversight. We believe that their careful usage and monitoring would bring only benefits, allowing the parties to successfully and effectively assess algorithmic behavior. That said, the use of diversity factors is not necessarily easy; sometimes it can be complex and other times even controversial. Thus, these factors should be seen as a set of tools that are equipped in the MOM framework to sufficiently facilitate oversight of algorithms, rather than a “golden solution” that can aid in all circumstances.

Finally, we have to point out the interrelated nature of diversity factors. They should not be interpreted as individual elements that are mutually exclusive, but rather as a set of factors that might influence each other. This is another challenge the literature must address when it comes to the actual effects of the set of diversity factors in this domain of knowledge. While our current focus is on describing the high-level oversight framework (i.e., an investigation of the specific diversity factors that could be involved in MOM is out of our current scope), future work must address the identification of the key diversity factors for a given oversight context (e.g., which diversity factors should be considered when crafting an oversight process to be used for assessing the behaviors of algorithmic decision-making in the banking context, etc.)

DITL. True SITL is a noble but impractical goal; we will never be able to involve an entire society in oversight. Rather, we aim to approximate SITL through diversity management, a process we call DITL. Utilizing DITL, we monitor the diversity factors of interest to manage the composition of the crowdsourced oversight force during the entire time of the oversight process; from the design (preparation phase) and execution of the process (recruitment and inspection phase) to the review of the results and the final decision (advise the phases of the MOM framework depicted in Figure 2). DITL can be a crucial tool for managing diversity by which we can maximize the utilization of its benefits in the oversight context and leverage a diverse set of views and values derived from the “wisdom of the crowd.” In other words, DITL is a tool approaching the notion of SITL, by using a predefined set of diversity factors of interest to monitor the composition of the crowdsourced society during a human oversight process, depending on the context and application of use.

Figure 2. A blueprint of the MOM framework to advise software developers during a human oversight process to scrutinize algorithmic behavior. The five phases are explained in a synoptic visual representation.

Figure 3. The applicable timeframes of MOM framework, based on the summarized SDLC practices in SE by Lu, Zhu, Xu, Whittle and Xing (Reference Lu, Zhu, Xu, Whittle and Xing2022).

Figure 4. An example of a third-party computer vision component to be integrated into a system for choosing among job candidates. We illustrate the oversight involvement during each timeframe as presented in Figure 3.

Ethics

Because of the human-centric nature of the MOM framework, ethics are an integral part of it. Numerous discussions reflect the need for considering human values and ethical principles in an oversight process, especially during the past few years. From an EU setting, the EC’s proposed ethics guidelines on Trustworthy AI⁶ reflect four basic ethical principles, such as (a) respect for human autonomy, (b) prevention of harm, (c) fairness and (d) explicability. In another EU initiative called AI4People, Floridi et al. (Reference Floridi, Cowls, Beltrametti, Chatila, Chazerand, Dignum, Luetge, Madelin, Pagallo, Rossi, Schafer, Valcke and Vayena2021) synthesized five ethical principles in the context of algorithmic systems that have to be considered during the SDLC. More specifically, the authors based their recommendations on the existing principles in bioethics to redefine a similar set of principles such as beneficence, nonmaleficence, autonomy, justice and explicability. Others (Mittelstadt et al. Reference Mittelstadt, Allo, Taddeo, Wachter and Floridi2016) reviewed the discussion around algorithmic ethics, providing a perspective map to organize the debate and assess the literature to identify areas for improvement toward the development of algorithmic ethics. Social norms are another aspect worth considering when dealing with ethics. For example, when parallelizing the algorithmic with human behavior, we should ask a few key questions: How do people act in X situations? What are the norms behind driving this act? Which of those norms should we consider applying in or assessing by, the current algorithmic solution?

Of course, ethical considerations in this core component are not limited to the aforementioned. Further work is needed, to provide a comprehensive set of ethical principles that are commonly agreed upon at least by the scientific literature, key organizations (i.e., EU) and initiatives. Moreover, we should be aware that as technology evolves, it is quite possible to deal with upcoming and emerging ethical concerns that are yet to come. This is why we need a modular framework such as MOM, to operate and facilitate human oversight and assessment of algorithmic behavior in a dynamic manner.

Guidelines and legal framework

Depending on the context of use – for instance, not only the country where the MOM framework is being used but also, where the developed system will be operating – regulations might apply differently. Many such regulations and guidelines aim to protect individuals’ freedom and core values, especially in the context of algorithmic governance. Some consider human values and ethical principles that are important to the scope of use. Last but not least, both national and international laws might apply depending on the occasion.

We mention once more the practices in the EU setting as a paradigm. The EC drafted the first regulatory framework for AI, the EU AI Act, where different rules apply for different risk levels of the AI application.Footnote ¹⁶ The initial plan was to reach an agreement on the final form of the law by the end of 2023. Despite the concerns about the complexity of the recent introduction of generative AI, the member states came into an agreement on December 2023Footnote ¹⁷ that will gradually be put into force during the next couple of years.Footnote ¹⁸ Thus, software developers and, more broadly, organizations that develop or use software that could pose risks will be enforced to comply with this law, as appropriate per the risk level of their systems. Particularly, the EC defines two classes of risks: unacceptable risk and high-risk. ¹⁶ AI systems categorized under the unacceptable risk are considered to be a threat to people and will be banned. On the other hand, AI systems that could negatively impact safety or fundamental rights will be considered high risk. Nevertheless, there is a fine line between these two classes, where a gray area exists, which encompasses systems that cannot be classified explicitly and transparently. Further work is required to provide methods for determining the risk type of those systems accordingly.

Despite the evolution speed of AI and technology in general, the legal framework is still ill-prepared with what currently exists and how is applied to the real world. The works of Floridi et al. (Reference Floridi, Cowls, Beltrametti, Chatila, Chazerand, Dignum, Luetge, Madelin, Pagallo, Rossi, Schafer, Valcke and Vayena2021), Mittelstadt et al. (Reference Mittelstadt, Allo, Taddeo, Wachter and Floridi2016) and Kaur et al. (Reference Kaur, Uslu, Durresi, Barolli, Li, Enokido and Takizawa2021) mirror the gap of retaining the necessary mechanisms, instruments, regulation and other schemes (i.e., government financial incentivization) that at the same time will both support and enforce organizations comply with the relevant regulations for algorithmic systems.

To sum up, the respective ethical guidelines and legal framework are core components of the MOM framework. They should be integrated into the process, in the way that MOM itself and the respective target algorithmic system of focus, stay compliant with any national or international laws that apply.

Modularity

To ensure a framework that can be dynamically adapted to different contexts and applications of use, it needs to be modular at the core. Consequently, we refer to modularity of the framework having three main characteristics in mind: (a) the framework consists of distinct important modules that facilitate its utilization (i.e., the core components), (b) a framework that is expandable towards new required modules in the future and (c) the ability of the framework to be altered and adapted into different contexts and applications of use. This enables the framework to operate in different situations and types of applications.

Modularity is vital to the MOM framework, as it is involved in various key aspects. Below we expand on five main critical aspects:

• • Domain of application: The framework has to consider different application contexts and uses. For example, it should be applicable in a diverse set of domains (e.g., not only in financial systems). The type of the target system under scrutiny is also essential and the framework should have the ability to facilitate human oversight for various types of applications ranging from simple algorithmic implementations to complex computer vision and natural language processing components. Finally, it should be able to be appended with new modules of upcoming emerging domains, leaving space for future improvement.

• • Human factors or entities: Different oversight executions might require a different pool of stakeholders depending on the context and application of use. Therefore, the framework should provide the capability to assemble the crowdsourced simulated society accordingly. This is where microtask crowdsourcing would be also valuable for recruiting the necessary people in the loop. Looking further into the crowdsourced society, some domains might require several additional stakeholders as experts in the fields (e.g., doctors in the health domain, when exercising oversight over a medical system). These expert stakeholders will be teaming with the crowdsourced society, both as part of a SITL to scrutinize the target algorithmic behavior.

• • Technological strategies: Various technological strategies for integration with the target systems to operate or assess them would be required for applying the MOM framework to exercise oversight. The framework itself should be flexible in terms of the way it can be integrated (e.g., using RESTFul services and library APIs). Also, there are various auditing strategies (Sandvig et al. Reference Sandvig, Hamilton, Karahalios and Langbort2014) that the developer who uses the MOM framework might want to apply.

• • Guidelines and laws: As both the MOM framework and the system of focus would operate in distinct contexts that might be affected by different legal perspectives, the proposed framework should be able to facilitate this kind of compliance. National and international guidelines and laws that apply should be considered for inclusion in the framework.

• • Diversity: The notion of diversity has a modular nature by definition. As described, modularity in diversity is a crucial factor for monitoring SITL through DITL, by recruiting a crowdsourced society of workers who bear diverse views and values on the target matter during the oversight of algorithmic behavior. The framework has to facilitate a DITL approach by providing insights during the execution of the oversight process and the means for altering the crowdsourced society based on the aforementioned diversity factors chosen, to maximize the benefits of such process.

Phase 1: Preparation

This preparation phase is the most important step for executing a comprehensive human oversight process using the MOM framework. During this step, all the necessary preparation decisions and actions should be made to tweak the modules of the framework depending on the context and application of use. More specifically, this step can be distinguished into two main consequent sub-categories: (a) the Definitions and then (b) Integration. Following, we elaborate further on these two subcategories.

Definitions. The developers, along with any other crucial involved stakeholders [hereon: the overseers], have to come to a consensus on the definitions of four main aspects, depending on the context and application of use. First, they should conclude with the ethics and norms involved, the aims of the target system, and the regulations that apply. These would structure an algorithmic social contract where both the overseers and the SITL would follow to assess the target algorithmic behavior. Second, they must agree on choosing the CPs they will exploit based on their capabilities, and as a result, the pool of workers each platform provides. In addition, they have to decide the number of workers in the crowdsourced society that would be formed and the corresponding budget they will be compensated for their contributions, depending on the complexity and duration of the task. Third, specific diversity factors of focus should be determined prior to the execution along with a set of quantitative and qualitative metrics. These will aid in the monitoring of DITL throughout the oversight process and continuously improve aspects of the crowdsourced society to provide observations of higher quality and scrutiny. Fourth, designing the stimulus is vital (i.e., the crowdsourcing task). More specifically, they have to define clear task instructions, the task flow, and any assets that would be involved considering also the ones that have to be prepared prior to the oversight task (e.g., a predefined dataset of diverse people images). We suggest designing micro-tasks – rather than macro-tasks – because of their simple, quick and efficient nature that enables them to be repeatable and parallel when necessary.

Integration. After the definitions are formed, the developers should begin the necessary actions to integrate both the system of focus and the selected CPs in the MOM framework. Particularly, the system of focus would be integrated via APIs (e.g., RESTful APIs) or other interfaces needed. In addition, the developer should be aware of the clear objectives of the system and follow a behavioral risk management plan for the components that are susceptible to generating unexpected behavior that might cause harm to groups of people and individuals. Finally, the developer has to integrate and configure the chosen CPs either through their provided interfaces or using their offered features to incorporate the flow of the task into the MOM framework for human oversight execution.

Phase 2: Recruitment

When the necessary preparation is done, the developer can begin proceeding in this recruitment step to start forming the simulated SITL via micro-task crowdsourcing by utilizing the chosen CPs. In this phase, the crowdsourcing task is executed and the recruitment of the crowdsourced society begins, always based on the characteristics derived from the chosen diversity factors. Also, the workers’ compensation begins. Finally, it is important to point out that this is the beginning of a continuous monitoring process of SITL through DITL that will be active also during the next two phases of inspection and review. This would enable an agile approach to improving the oversight as facilitated by the MOM framework through the adjustment of SITL through DITL. When the crowdsourced observations are not sufficient or representative of the aims of the task, focus diversity factors can be altered accordingly to enrich or make more specific the pool of workers collected so far. This phase acts as the anchor for the following ones to step back and repeat the necessary actions to improve the quality of the oversight process.

Phase 3: Inspection

The next phase is the inspection phase, where live monitoring and oversight are active during the inspection of algorithmic behavior by SITL and the overseers (i.e., the developer and involved stakeholders). During this phase, a set of predefined quantitative and qualitative metrics (from the preparation phase) are generated based on the SITL observations. The SITL/DITL monitoring continues to play an active role in this phase. The overseers can modify SITL through DITL by altering the diversity factors of focus at any given time to re-recruit and re-execute the crowdsourced oversight task by going back to the second phase when observations are not sufficient.

Phase 4: Review

When the execution of the crowdsourcing part is done, the overseers should review the observations gathered by SITL. More specifically, they should conduct postquantitative and postqualitative assessments to examine the target algorithmic behavior. Again, when observations are insufficient or nonrepresentative of the aims of the system or the oversight task, the overseers can roll back to the second phase where they could adjust SITL through DITL and re-execute the crowdsourcing part to improve the results. This is the last phase where the monitoring of SITL/DITL is still active and ends by proceeding to the next phase.

Phase 5: Decision

The final phase is focused on decision-making after the necessary inspection and review of the target system is done. To be exact, the overseers come to a decision on the assessment of the target algorithmic behavior and the implications or issues perpetuated. At the same time, a set of reports is generated based on the reporting metrics and other forms of assessments that have been initially agreed upon (in the preparation phase). Optionally, this process can aid in initiating a developer intervention to fix or mitigate the issue if possible. Otherwise, the user of the system can contest the decision of the generated output and override it to mitigate any harmful effects on groups of people or individuals.