Health visiting in England is a universal service that aims to promote the healthy development of children aged under five years and safeguard their welfare. We consulted stakeholders about their priorities for research into health visiting and also used these consultations and a literature review to generate a logic model. Parents wanted research to explore how health visiting teams can provide a caring, responsive, accessible service (the mechanisms of change). Policymakers, commissioners, and clinical service leads wanted descriptions and evaluations of currently implemented and ‘gold standard’ health visiting. The challenges to evaluating health visiting (data quality, defining the intervention, measuring appropriate outcomes, and estimating causal effects) mean that quasi-experimental studies that rely on administrative data will likely underestimate impact or even fail to detect impact where it exists. Prospective and experimental studies are needed to understand how health visiting influences infant–parent attachments, breastfeeding, childhood accidents, family nutrition, school readiness, and mental health and well-being.

Health technology assessment (HTA) agencies assess evidence to support decision making about which technologies to provide and pay for in the health system. HTA impact is understood as the influence that HTA report findings can have in the health system, including impacts on reimbursement decisions, changes to health outcomes, or broader system or societal impacts. The International Network of Agencies for Health Technology Assessment (INAHTA) is a global network of publicly funded HTA agencies. INAHTA’s mission, in part, is to advance the impact of HTA to support reimbursement decisions and the optimal use of health system resources. Each year, INAHTA awards the David Hailey Award for Best Impact Story to the member agency that shares the best story, as voted by fellow members, about HTA impact. The impact story sharing program in INAHTA contributes to a deeper understanding of what works well (or not so well) in achieving HTA impact. This paper provides six impact stories from agencies that were finalists for the 2021 and 2022 David Hailey Impact Award for Best Impact Story: the Institut national d’excellence en santé et en services sociaux, the Malaysian Health Technology Assessment Section, Ontario Health, the Center for Drug Evaluation, the National Institute for Health and Care Excellence, and Health Technology Wales. These stories demonstrate that HTA agencies can, in differing ways, effectively support governments in their efforts to place evidence at the centre of decision making.

Creating a sustainable residency research program is necessary to develop a sustainable research pipeline, as highlighted by the recent Society for Academic Emergency Medicine 2024 Consensus Conference. We sought to describe the implementation of a novel, immersive research program for first-year emergency medicine residents. We describe the curriculum development, rationale, implementation process, and lessons learned from the implementation of a year-long research curriculum for first-year residents. We further evaluated resident perception of confidence in research methodology, interest in research, and the importance of their research experience through a 32-item survey. In two cohorts, 25 first-year residents completed the program. All residents met their scholarly project requirements by the end of their first year. Two conference abstracts and one peer-reviewed publication were accepted for publication, and one is currently under review. Survey responses indicated that there was an increase in residents’ perceived confidence in research methodology, but this was limited by the small sample size. In summary, this novel resident research curriculum demonstrated a standardized, reproducible, and sustainable approach to provide residents with an immersive research program.

Novel prediction methods should always be compared to a baseline to determine their performance. Without this frame of reference, the performance score of a model is basically meaningless. What does it mean when a model achieves an $F_1$ of 0.8 on a test set? A proper baseline is, therefore, required to evaluate the ‘goodness’ of a performance score. Comparing results with the latest state-of-the-art model is usually insightful. However, being state-of-the-art is dynamic, as newer models are continuously developed. Contrary to an advanced model, it is also possible to use a simple dummy classifier. However, the latter model could be beaten too easily, making the comparison less valuable. Furthermore, most existing baselines are stochastic and need to be computed repeatedly to get a reliable expected performance, which could be computationally expensive. We present a universal baseline method for all binary classification models, named the Dutch Draw (DD). This approach weighs simple classifiers and determines the best classifier to use as a baseline. Theoretically, we derive the DD baseline for many commonly used evaluation measures and show that in most situations it reduces to (almost) always predicting either zero or one. Summarizing, the DD baseline is general, as it is applicable to any binary classification problem; simple, as it can be quickly determined without training or parameter tuning; and informative, as insightful conclusions can be drawn from the results. The DD baseline serves two purposes. First, it is a robust and universal baseline that enables comparisons across research papers. Second, it provides a sanity check during the prediction model’s development process. When a model does not outperform the DD baseline, it is a major warning sign.

The US biomedical research workforce suffers from systemic barriers causing insufficient diversity and perpetuating inequity. To inform programming enhancing graduate program access, we implemented a formative mixed-method study to identify needed supports for program applications and graduate program success. Overall, results indicate value in added supports for understanding application needs, network development, critical thinking, time management, and reading academic/scientific literature. We find selected differences for underrepresented minority (URM) students compared to others, including in the value of psychosocial supports. This work can inform broader efforts to enhance graduate school access and provides foundation for further understanding of URM students’ experiences.

The purpose of this paper is to answer the research question: How may methods of Benefit–Cost Analysis (BCA) influence policy decision-makers? The new research is a continuation of the previous research, and it is updated with new data and new examples. As it is well-known by BCA experts and practitioners, BCA is a relatively simple and widely used technique for deciding whether to make a change. BCA is very important in political and governmental decisions. Governments are policy creators, and Governments are usually formed of politicians. How much do experts inside Governments influence the policymakers? Do politicians allow enough experts to influence policymaking? The research regarding the work of the Croatian Government’s Office for Legislation and its Unit for the system and coordination of the Assessment of the Effects of Regulations (Unit) showed that the analysis of that Unit was not very good in some very important laws. These laws were changed very often because in practice was seen that the application of some laws was not good, and had caused some problems for the citizens, caused legal uncertainty for them, and in some cases had caused a lot of damage.

Organizations supporting translational research and translational science, including Clinical and Translational Science Award (CTSA) hubs, provide a diverse and often changing array of resources, support, and services to a myriad of researchers and research efforts. While a wide-ranging scope of programs is essential to the advancement of translational research and science, it also complicates a systematic and unified process for tracking activities, studying research processes, and examining impact. To overcome these challenges, the Duke University School of Medicine’s CTSA hub created a data platform, Translational Research Accomplishment Cataloguer (TRACER), that provides capacity to enhance strategic decision-making, impact assessment, and equitable resource distribution. This article reviews TRACER development processes, provides an overview of the TRACER platform, addresses challenges in the development process, and describes avenues for addressing or overcoming these challenges. TRACER development allowed our hub to conceptually identify key processes and goals within programs and linkages between programs, and it sets the stage for advancing evidence-based improvement across our hub. This platform development provides key insight into facilitators that can inform other initiatives seeking to collect and align organizational data for strategic decision-making and impact assessment. TRACER or similar platforms are additionally well positioned to advance the study of translational science.

With assisted dying becoming increasingly available to people suffering from somatic diseases, the question arises whether those suffering from mental illnesses should also have access. At the heart of this difficult and complex matter are values such as equality and parity of esteem. These issues require humane deliberation.

Human evaluation of machine translation is considered the “gold standard” for evaluation, but it remains a challenging task for which to define best practices. Recent work has focused on incorporating intersentential context into human evaluation, to better distinguish between high-performing machine translation systems and human translations. In this work, we examine several ways that such context influences evaluation and evaluation protocols. We take a close look at annotator variation through the lens of calibration sets and focus on the implications for context-aware evaluation protocols. We then demonstrate one way in which degraded target-side intersentential context can influence annotator scores of individual sentences, a finding that supports the context-aware approach to evaluation and which also has implications for best practices in evaluation protocols.

Structured processes to improve the quality and impact of clinical and translational research are a required element of the Clinical and Translational Sciences Awards (CTSA) program and are central to awardees’ strategic management efforts. Quality improvement is often assumed to be an ordinary consequence of evaluation programs, in which standardized metrics are tabulated and reported externally. Yet evaluation programs may not actually be very effective at driving quality improvement: required metrics may lack direct relevance; they lack incentive to improve on areas of relative strength; and the validity of inter-site comparability may be limited. In this article, we describe how we convened leaders at our CTSA hub in an iterative planning process to improve the quality of our CTSA program by intentionally focusing on how data collection activities can primarily advance continuous quality improvement (CQI) rather than strictly serve as evaluative tools. We describe our CQI process, which consists of three key components: (1) Logic models outlining goals and associated mechanisms; (2) relevant metrics to evaluate performance improvement opportunities; and (3) an interconnected and collaborative CQI framework that defines actions and timelines to enhance performance.