Background
Translating scientific discoveries from bench to bedside is often challenging for new healthcare technologies (Reference Seyhan1), particularly for medical tests. Despite extensive research efforts, many innovative tests ultimately fail to enter clinical practice (Reference Wang and Kricka2). This is in part due to their failure to address a clear unmet clinical need (Reference Engel, Wachter, Pai, Gallarda, Boehme and Celentano3), as well as a common lack of understanding as to how tests will fit into existing clinical pathways (Reference Abel, Shinkins, Smith, Sutton, Sagoo and Uchegbu4). This is further exacerbated by the slow adoption of new tests by end users (Reference François, Carmen, Yves and Yves5). Therefore, there is a need for new methods that ensure research efforts are focused on technologies that fulfill a specific unmet clinical need and have the potential to be cost-effective.
Target Product Profiles (TPPs), typically drafted in the early stages of technology development, describe the necessary properties of a new technology to meet an unmet clinical need (Reference Lambert6). TPP development is ideally an iterative process, with initial technology requirements being updated as additional evidence becomes available (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7). The aim of TPPs is to focus innovation on “fit-for-purpose” technologies that address a specific clinical need. Although past TPPs of tests have focused on addressing infectious diseases in low- and middle-income countries (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7), the broader utility of TPPs is being increasingly recognized (8). In the wake of the COVID-19 pandemic, for example, the UK Medicines & Healthcare products Regulatory Agency (MHRA) developed a series of TPPs for COVID-19 tests to promote the development of clinically useful diagnostics (9).
In a recent systematic review, we described the methodology that is currently being used to develop TPPs of medical tests (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7). Existing TPPs largely focus on summarizing necessary specifications across five key evidence domains: (i) analytical performance (e.g., sample requirements), (ii) clinical validity (e.g., diagnostic sensitivity and specificity), (iii) infrastructural requirements (e.g., transport and storage conditions), (iv) human factors (e.g., device format and complexity), and (v) costs (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7). Several methodology limitations were also identified. These concerned: (i) the subjectivity of data informing test requirements (e.g., heavy reliance on expert opinion); (ii) poor transparency in reporting the methodology underpinning TPP documents; (iii) a lack of explicit consideration for clinical utility when defining test characteristics; and (iv) an oversight of cost-effectiveness considerations.
Economic evaluation compares the costs and benefits of alternative healthcare strategies to support technology adoption and reimbursement decisions. Whilst economic evaluation is usually conducted at the later stages of the evaluation pathway, early economic evaluation (EEE) is increasingly performed at the proof-of-concept or pre-regulatory-approval stage as a means of directing stop/go decisions and informing the optimal trajectory of future research (Reference Buisman, Rutten-van Mölken, Postmus, Luime, Uyl-de Groot and Redekop10;Reference Frempong, Sutton, Davenport and Barton11). A standard approach used in economic evaluations—early or otherwise—is decision analytic modelling: a framework of analysis that uses mathematical tools and schematics to provide a simplified representation of the decision problem, as a series of uncertain events.
In this opinion paper, we discuss how EEE could strengthen the methodological rigor of TPP development for tests. With TPP development being framed around unmet clinical needs, EEE in this context should be regarded as a technology-agnostic methodology that is done independently of any new or existing technology. Based on our collective experience, we describe the possible advantages that EEE methods could afford across four core activities of TPP development: scoping (see Section "Scoping: Pathway Mapping to Define Test Role, Purpose, and Potential Impact"), drafting and consensus building (Section "Drafting and Agreeing Test Specifications"), and updating (Section "Updating TPP specifications"). By placing EEE at the heart of TPP development, we describe how the methodological limitations of TPPs (highlighted above) could be effectively addressed. The key arguments raised below are summarized in Figure 1.
Figure 1. Overview of some expected benefits that early economic evaluation (EEE) can bring to each Target Product Profile (TPP) development activity, alongside relevant methods.
Integrating EEE Methodology within TPP Development
Scoping: Pathway Mapping to Define Test Role, Purpose, and Potential Impact
The scoping phase defines the focus of a TPP in terms of the unmet clinical need being addressed and which test characteristics should be included (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7). Typically, this is achieved via literature reviews and consultations with experts and stakeholders (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7). Based on current practice, it is unclear if any formal pathway analysis and/or assessment of a test's placement and impact is typically undertaken at this stage (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7); this risks overlooking key mechanisms of impact on patient outcomes.
The first step in economic modelling is to map the care pathway in which the new technology will sit. This typically involves reviewing clinical guidelines and consulting clinicians, patients, and carers on their individual experiences of the care pathway. The result is a clear schematic of the existing clinical pathway, detailing the main activities and events that may occur for the patient population under consideration. This ensures that the processes involved in standard care are clearly understood and also forces test developers to specify a test role (i.e., whether a test is a replacement, triage, or add-on) and purpose (e.g., screening, diagnosis, prognosis, prediction, or monitoring) within the specified pathway. This is crucial in the context of TPP development, because selecting a different test role and/or purpose can result in significantly different test requirements (Reference Frempong, Sutton, Davenport and Barton12). Where the optimal placement of a new test is unclear, additional comparators can be added to an EEE to explore alternative options (Reference Frempong, Sutton, Davenport and Barton12).
Pathway mapping further helps to identify the possible downstream consequences (harms and benefits) of a new test, including expected impacts on: decision making (e.g., change in treatment decisions), patient health outcomes, clinical workflow (e.g., time-to-treatment), and economic outcomes (Reference Miller, Atrzadeh, Burnham, Cavalieri, Dunn and Jones13). This is vital for TPPs, because the expected harms and benefits associated with a test should directly influence which test specifications are included within a TPP. For EEEs, this step further helps to identify which test properties should be captured and varied within the economic model (diagnostic accuracy, test price, turnaround time, etc.). Although pathway mapping is a core process of economic modelling, it should be noted that this activity can be conducted independently of any formal economic evaluation.
Drafting and Agreeing Test Specifications
The TPP drafting phase estimates the specifications for new tests, whereas consensus building aims to attain consensus on those requirements through stakeholder surveys and consensus meetings. Typically, test specifications are presented at two levels (“desirable” and “acceptable”) based on expert consultations and literature findings, and without explicitly considering the dependencies between different test specifications (e.g., the dependence between diagnostic sensitivity and specificity), or other factors related to the clinical context (e.g., the required sensitivity and specificity will be dependent on disease prevalence) that could jointly impact the overall utility of a given testing strategy (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7). This apparent failure to consider parameter dependencies risks over- or underestimating test requirements.
Here, we outline how EEE can: (i) allow exploration of dependencies between test specifications and features of the clinical context; (ii) facilitate optimization of quantitative test specifications based on a willingness-to-pay (WTP) threshold; and (iii) increase the objectivity of data informing TPP requirements.
Exploring Dependencies Between Test Specifications and Features of the Clinical Context
Desirable ranges for test specifications may vary when considering different factors underlying the care pathway or other test properties. It might, therefore, be helpful to distinguish upfront between:
• Properties inherent to a test—test properties (e.g., diagnostic accuracy and test turnaround time) that could have a direct/indirect effect on the utility of a test.
• Factors relating to the clinical context—aspects exogenous to a test that could influence how a test impacts upon relevant outcomes (e.g., prevalence of disease, natural disease progression, efficacy, and cost of treatment).
With EEE, by synthesizing multiple test attributes and factors relating to the clinical context within a single modeling framework, the dependencies between different parameters can be effectively captured and explored. One-way or multiway sensitivity analyses can be used to explore the impact of one or more parameter(s) on the clinical utility or cost-effectiveness (Reference Abel, Dakin, Roberts, Ashdown, Butler and Hayward14–Reference Trentham-Dietz, Ergun, Alagoz, Stout, Gangnon and Hampton19). Scenario analyses can also be used to explore the impact of a group of parameter changes that represent a specific clinical scenario (Reference Abel, Dakin, Roberts, Ashdown, Butler and Hayward14;Reference Lansdorp-Vogelaar, Goede, Bosch, Melotte, Carvalho and van Engeland16).
Estimating Ranges for Quantitative Test Characteristics
Outside the context of TPPs, EEE has previously been used to identify acceptable ranges for test characteristics or components (Reference Frempong, Sutton, Davenport and Barton12;Reference Abel, Dakin, Roberts, Ashdown, Butler and Hayward14–Reference Dowdy, O'Brien and Bishai20). In the absence of evidence for a new test, hypothetical values for test properties can be assigned and, for each specification or combinations of specifications, the downstream costs and benefits can be computed. Based on a specified WTP threshold, it is possible to then back-calculate the maximum costs and minimum specifications for a new test to be cost-effective (Reference Buisman, Rutten-van Mölken, Postmus, Luime, Uyl-de Groot and Redekop10).
Key relevant methods that may be useful in this phase are:
• Headroom analysis—The headroom price of new tests represents the maximum price at which a test is considered cost-effective, assuming perfect clinical accuracy. This method can be used to determine the maximum cost for a new test, while calculating the financial room for improvement for a hypothetical test assuming perfect conditions. See (Reference Frempong, Sutton, Davenport and Barton12;Reference Kluytmans, Deinum, Jenniskens, van Herwaarden Antonius, Gloerich and van Gool Alain15) for examples.
• Threshold analysis—Identifies the critical value for input parameters above or below which a reimbursement decision is expected to change. This approach could be used within TPPs to estimate the maximum price or minimum diagnostic accuracy at which a test remains cost-effective given a certain WTP threshold. See (Reference Frempong, Sutton, Davenport and Barton12;Reference Kluytmans, Deinum, Jenniskens, van Herwaarden Antonius, Gloerich and van Gool Alain15–Reference Sutton, Lamont, Evans, Williamson, O'Rourke and Duggan18;Reference Dowdy, O'Brien and Bishai20) for examples.
• Sensitivity analysis—In the absence of a clear WTP threshold, sensitivity analysis could be used to identify ranges of minimal test properties, while exploring changes in cost-effectiveness results due to altering one or more model parameters. See (Reference Abel, Dakin, Roberts, Ashdown, Butler and Hayward14;Reference Kluytmans, Deinum, Jenniskens, van Herwaarden Antonius, Gloerich and van Gool Alain15;Reference Trentham-Dietz, Ergun, Alagoz, Stout, Gangnon and Hampton19) for examples.
Objective Derivation of Test Specifications
The early stages of technology development are characterized by high uncertainty, making it challenging for experts to form accurate judgments around the desirable specifications that a new test should possess.
EEE methodology facilitates the generation of more objective data, which can be combined with subjective evidence (i.e., clinical and stakeholder judgment) to inform TPP specifications. EEE results can also support the elicitation of expert opinion, with clinical experts being asked to review and discuss the modelling findings. Presenting the results of EEE modelling within consensus meetings could further help to narrow down the ranges for performance benchmarks that a new test should hit to be both clinically effective and cost-effective.
Were EEEs to be integrated into TPP development, the methods, model code, and analysis underpinning TPPs should be clearly reported and made freely available, in line with international modelling guidelines (Reference Husereau, Drummond, Petrou, Carswell, Moher and Greenberg21). This would provide a more transparent and objective approach to setting performance benchmarks, while, in turn, allowing others to inspect what data have informed TPP development.
Updating TPP Specifications
Although TPPs are dynamic documents that should be updated as new evidence is found (Reference Lambert6), no formal or standardized approach for updating TPP specifications is currently available (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7). We believe that this activity is important as a means of ensuring that test development is driven by accurate, up-to-date information. Here, we discuss some expected advantages that EEE can bring to this core activity.
Incorporating New Evidence
EEEs can be designed so as to be sufficiently flexible to allow for the model to be iteratively updated as evidence is accrued (Reference Abel, Shinkins, Smith, Sutton, Sagoo and Uchegbu4). This means that different TPPs could be developed based on the level of evidence available: early TPPs could focus on defining the unmet clinical need and key requirements for a new test to address that need, whereas later iterations could identify more precise test specifications.
Abel et al. (Reference Abel, Dakin, Roberts, Ashdown, Butler and Hayward14), for example, published their model code to allow others to examine and explore additional questions. For TPPs, this approach would help to ensure that performance specifications could be validated with each iteration of a TPP, thus further increasing the methodological rigor underpinning TPPs.
Highlighting Areas for Future Research
Findings from sensitivity analyses could be used to communicate areas of greatest uncertainty requiring further research. Alongside this, when more evidence and information about a test is available (i.e., when parameter distributions reflect uncertainties rather than “unknowns”), Value of Information (VOI) analysis may be useful to assess the value of gathering more information on model parameters to reduce current decision uncertainty (Reference Frempong, Sutton, Davenport and Barton12;Reference Sutton, Lamont, Evans, Williamson, O'Rourke and Duggan18).
Conclusion
TPPs are increasingly of interest as a means of focusing the innovation pipeline for new tests on areas of clinical need (8;22). Here, we argue that some of the methodological limitations with TPPs (Reference Cocco, Ayaz-Shah, Messenger, West and Shinkins7) could be addressed by integrating EEE into TPP development. In particular, pathway mapping would provide clarity on the mechanisms by which a test could impact on patient health, and decision analytic modelling could increase the objectivity and transparency of the data informing TPP specifications. Transparent and flexible early economic models would also allow test specifications to be updated as new evidence becomes available, helping to prioritize areas for future research.
There are, however, potential limitations and complexities with integrating EEE into TPP development. For example, conducting a wide range of sensitivity analyses with EEE risks overwhelming TPP developers with information, making the estimation of clearly defined test performance requirements challenging. Such analyses should always be kept within the confines of clinical plausibility, and clinical experts can support the identification of clinically relevant scenarios. Nevertheless, given that uncertainty in the technology development process is unavoidable, we believe that the value in such analyses lies in explicitly communicating those key areas of uncertainty, identifying trade-offs between test characteristics, and highlighting priorities for future research.
Economic evaluation also requires the adoption of a particular jurisdiction (e.g., UK NHS), which dictates what country-specific clinical pathways and costs are included in the model and what prespecified decision criterion is used to inform the analysis (e.g., the UK WTP threshold per quality-adjusted life year). In case a particular decision criterion is not based on cost-effectiveness, the EEE methods discussed here may still be useful to explore scenarios around clinical utility or cost. The applicability of modelling findings to other jurisdictions is, therefore, often limited, and multiple model versions may be required to derive test specifications across different jurisdictions. This is true of any formal evaluation of downstream clinical or cost outcomes, however (where country-specific clinical pathways and costs will come in to play), and is not specific to EEE.
Several of the issues raised here (transparency, uncertainty, notion of value, etc.) have been discussed in previous papers (Reference Grutters, Govers, Nijboer, Tummers, van der Wilt and Rovers23–Reference Teljeur and Ryan25); interested readers may wish to consult these commentaries for further aspects to consider when using EEEs to inform the technology development process.
Although we have argued that EEE is a valuable methodology to support TPP development, we recommend this approach as an adjunct rather than a replacement for the existing TPP development methodology. Expert input and consensus-building discussions with relevant stakeholders will be key to ensuring that EEEs capture the nuances of a clinical context, and that the results are clinically meaningful. Ultimately this will ensure the efficient development of tests that provide greater utility for both patients and healthcare services.
Acknowledgments
The authors thank Dr. Kerrie Davies (Leeds Teaching Hospitals NHS Trust) for reading the manuscript and providing insightful feedback.
Funding Statement
This work was carried out as a part of the full-time School of Medicine PhD Scholarship awarded to PC by the University of Leeds. This work is also supported by the “Antimicrobial Resistance Cross Council Initiative” (Grant Number MR/N029976/1). Funding Partners: The Biotechnology and Biological Sciences Research Council, the Engineering and Physical Sciences Research Council, and the Medical Research Council. BS and AS are supported by NIHR Leeds In Vitro Diagnostic Co-operative; BS is also part-funded by Cancer Research UK via the CanTest Collaborative.
Conflicts of Interest
PC, AS, RW, and BS have nothing to disclose. MM is an employee of the Medicines and Healthcare Products Regulatory Agency (MHRA). He has previously been a paid consultant/advisor to PinPoint Data Science, UK Department of Health and Social Care, European Union, Cepheid Inc, Boston Healthcare, and Simon-Kucher & Partners.
Open access
