Evaluating the algorithmic behavior of interactive systems is complex and time-consuming. Developers increasingly recognize the importance of accountability for their algorithmic creations’ unanticipated behavior and resulting implications. To mitigate this phenomenon, developers not only need to concentrate on the observable inaccuracies that can be measured quantitatively but also the more subjective outcomes that can perpetuate social bias, which are challenging to identify. We require a new approach that involves humans in scrutinizing algorithmic behavior. It leverages a combination of quantitative and qualitative methods to support an ethical, value-aligned design and a system’s lifecycle, informed by users’ perception and values. To date, the literature lacks an agreed-upon framework for such an approach. Consequently, we propose an oversight framework, Modular Oversight Methodology (MOM), which aids developers in assessing the behavior of their systems by involving a carefully crowdsourced society-in-the-loop. The framework facilitates the development and execution of an oversight process and can be tweaked according to the domain and application of use. Through such an oversight process, developers can assess the human perception of the algorithmic behavior under inspection, and extract valuable insights that will aid in assessing its implications. We present the MOM framework, as a first step toward tailoring more robust, domain-specific solutions to exercise human oversight over algorithms, as a means for software developers to keep the generated output of their solutions fair and trustworthy.

Health technology assessment (HTA) programs inform decision making about the value and reimbursement of new and existing health technologies; however, they are under increasing pressure to demonstrate that they are a cost-effective use of finite healthcare resources themselves. The 2023 HTAi Global Policy Forum (GPF) discussed the value and impact of HTA, including how it is assessed and communicated, and how it could be enhanced in the future. This article summarizes the discussions held at the 2023 HTAi GPF, where the challenges and opportunities related to the value and impact of HTA were debated. Core themes and recommendations identified that defining the purpose of value and impact assessment is an essential first step prior to undertaking it, and that it can be done through the use and expansion of existing tools. Further work around aligning HTA programs with underlying societal values is needed to ensure the long-term value and impact of HTA. HTA could also have a role in assessing the efficiency of the wider health system by applying HTA methods or concepts to broader budgetary allocations and organizational aspects of health care. Stakeholders (particularly patients, industry, and clinicians but also payers, wider society, and the media) should ideally be actively engaged when undertaking the value and impact assessment of HTA. More concerted efforts in communicating the role and remit of HTA bodies would also help stakeholders to better understand the value and impact of HTA, which in turn could improve the implementation of HTA recommendations and application to future actions in the lifecycle of technologies.

The SUERC Radiocarbon Laboratory reports approximately 3000 unknown samples per year with an additional 1200 samples processed for quality assurance purposes. In addition to the primary OxII standard (SRM-4990C) required for AMS batch normalization, secondary “known-age” standards have been used over many years to evaluate individual batch quality. These have included wood (as prepared alpha-cellulose), barley mash, humic acid, a background mammoth bone, known-age bones, and a whisky sample. In this paper, we present some of the results gathered over routine laboratory operation (for more than 10 years) and examine the results illustrating how they are being used to monitor and quality assure performance. Since many of these samples have also been used in the Glasgow intercomparisons, we will also reflect on the results, as well as the actual and potential uses of such samples.

It is not enough to study animal welfare; our responsibility is to promote it. To this end, we need to step out of our laboratories and develop robust protocols for assessing welfare in groups of animals on farms or in the laboratory. While these protocols must incorporate principles derived from detailed scientific study they will, in practice, need to be based on clinical observations and records that an assessor can acquire at a single visit. Such assessments must also be comprehensive: approaches based only on behaviour, or motivational state, or physical appearance, or performance records, can never tell the full story. The ‘Five Freedoms’ provide a comprehensive template that incorporates the different elements that define welfare state. However they only describe welfare at the time of observation. Protocols, for example, for on-farm assessment need to include measurements and records that provide evidence of long-term consequences of the quality of husbandry practices (eg body condition, chronic injuries). This paper reviews approaches developed at Bristol for the on-farm evaluation of welfare in dairy cows and free-range hens. The criteria used to assess welfare from clinical evidence were developed through expert consultation using the ‘Delphi’ review process. Experts were also asked to make value judgements as to the seriousness of different elements of poor welfare by indicating the point at which intervention would be necessary to resolve specific problems. This study identified a serious limitation of Quality Assurance schemes that seek to encompass many different elements of welfare into a single index that ranks overall welfare as acceptable or unacceptable. Specific farms had specific welfare problems and these required specific solutions.

This paper briefly reviews the development of monitoring procedures used for the assessment of husbandry and welfare within Farm Quality Assurance (QA) schemes. Most current protocols are based on measures of the resources, records and management provisions necessary to promote good husbandry. However, it is now generally accepted that monitoring protocols should be largely or wholly made up of direct, animal-based measurements of animal welfare. Whether based on provisions or outcomes, many current schemes lack impact, partly because they do not necessarily lead to effective action on-farm and partly through lack of public awareness or trust in the claimed benefits of the scheme. This paper proposes the concept of the ‘Virtuous Bicycle’ as a delivery vehicle for improvements in farm animal welfare through simultaneous operation of two virtuous cycles, one on-farm, involving assessment, action and review, the other at the retailer level, involving assurance and promotion, based on proof of compliance.

The objective of the present retrospective analysis was to review between- and within-sector variations in an outcome-based measure of animal welfare throughout slaughterhouses that currently supply to Tesco Stores Ltd, UK. Non-conformances in relation to individual scheme standards were designated a specific level in terms of severity and frequency and from this a single outcome status, based on a ‘traffic-light’ system is assigned to the site (which informs both subsequent corrective action and future inspection frequency). Sector-specific, country and time differences were found and underlying contributory factors and associated commercial implications are reviewed.

This paper describes the implementation and simultaneous promotion of an action plan designed to ensure animal welfare standards on-farm that exceed the requirements for acceptability in law. The approach is based on two action cycles, the producer and retailer cycles, The producer cycle, involving welfare audit and the implementation of an action plan for welfare has four stages: self-assessment; independent audit; creation of an action plan based on identification of principal hazards and critical control points; review; and revision of the action plan depending upon assessment of outcomes. The retailer cycle is designed to set quality standards for animal welfare, demonstrate compliance, promote proven high welfare products and reward producers. The paper reviews some incentives and constraints to action for both farmers and retailers and presents encouraging examples of the extent to which both producers and retailers have responded to increased public demand for high welfare products.

This paper focuses on quality assurance in language massive open online courses (LMOOCs). It is a qualitative study that adopts the grounded theory method and analyses evaluative comments on the quality of LMOOCs from learners’ perspectives. With the data collected from 1,000 evaluations from English as a second language (ESL) learners on China’s biggest MOOC platform “iCourse”, this study examines what has influenced learners’ perceptions of LMOOCs and identifies the specific quality criteria of five types of them, including ESL courses for speaking, reading, writing, cultural studies, and integrated skills. The results of the study will lay a foundation for the establishment of a quality criteria framework for LMOOCs and provide insights into design principles for effective online language courses tailored to the diverse needs of a massive number of language learners.