EVIDENCE ON DEVICES IS TYPICALLY SPARSE

A fundamental problem in making decisions about adopting new technologies is lack of evidence. Regulatory systems require quite small amounts of evidence for market approval of devices (Reference McCulloch1). This requirement varies depending on the class of device, but even at its most stringent does not match the demand for evidence for new pharmaceutical products (Reference McCulloch1). There have been suggestions that the same levels of evidence should be required for devices as for drugs (Reference McCulloch1;Reference Hofmann2). However, this does not take into account the research capacity of the device industry (many small companies rather than a few multinationals) (Reference Chapman, Taylor and Girling3), the critical dependence of devices on clinical pathways and the context of their use (Reference Drummond, Griffin and Tarricone4), the short “market life” and gradual evolution of many medical devices (Reference Gelijns, Russo and Hong5), and their typically lower financial returns. All these factors mean that demanding similar evidence for devices as for pharmaceuticals is unrealistic. Some of these factors may change over time (e.g., consolidation in the medical technology sector may lead to fewer, larger companies), but some are differences in principle (e.g., the dependence of devices on the context of their use and who is using them) that are unlikely to change.

MAKING RECOMMENDATIONS ABOUT ADOPTION

The typically sparse evidence base makes health technology assessment (HTA) of new devices a real challenge. It is particularly difficult for local organizers of health care to make judgements about adopting new technologies, on the basis of limited capacity and information. In the United Kingdom, the National Institute for Health and Care Excellence (NICE) has established a system for evaluating devices and diagnostics notified by manufacturers (the Medical Technology Evaluation Program) (6;Reference Campbell and Campbell7). It considers claims made for their advantages to patients and to the health service and models the cost consequences of introducing them, in place of current management. This “value proposition” forms the basis of recommendations about their adoption into the UK National Health Service. Based on our experience from 5 years of evaluating new technologies, we believe that this kind of approach represents a valuable and necessary compromise between piecemeal decision making by local healthcare providers and conventional HTA, which requires a more mature evidence base.

WORKING WITH LIMITED EVIDENCE

When high quality published evidence from clinical trials is lacking, then decisions need to be made about the type and quality of evidence that is sufficiently reliable and useful: for example, data from registers, unpublished conference abstracts, surveys, and local audits. If assessments are done early in the life cycle of technologies, then all of these need to be considered. Advice from experts is particularly helpful when published evidence is sparse or poor: clinical and scientific experts can help in sifting and interpreting the available information (Reference Oyebode, Patrick, Walker, Campbell and Powell8). They can provide valuable insights into current practice, the possible consequences introducing a new technology, and the likely obstacles to its adoption.

We have previously described some of the issues in working with limited evidence to make judgements about new devices, including a departure from conventional HTA by considering the concepts of promise and plausibility (Reference Campbell9). These require qualitative judgements about the way a device works, the nature and size of its proposed benefits, and the plausibility that its adoption will provide the claimed improvements for patients and the health service, in everyday practice. Experience has supported our impression that the “promise” of a technology influences the amount and quality of evidence needed for a favorable judgement about adopting it into clinical practice. Here, we describe the balance between promise and other key considerations, and the available evidence, in making decisions about adopting new devices and diagnostic technologies.

PROMISE AND PLAUSIBILITY

The concept of promise comprises several considerations which link the operation of a device to the benefits it might deliver. A simple, well-designed device with an easily understood mechanism of action helps to underpin a sense of promise, especially if its concept seems to be clever, novel, or unique. If a device or diagnostic appears radically to improve a pathway of care, reducing its complexity for patients or healthcare staff, or reducing the need for resources, that will contribute to its promise. The numbers of patients likely to derive benefit and the degree to which it could transform their health are further considerations (Reference Campbell9). Expectation of a large improvement in the health outcomes of a large number of patients equates to high promise. The range of clinicians and scientists on NICE's Medical Technologies Advisory Committee are well placed to make judgements about whether or not technologies appear to have promise. Committee members are selected for their practical experience as well as their track records in research and health technology assessment.

The concept of promise suggests that decisions are being made against a backdrop of sparse evidence. If plentiful data from large, well-conducted trials are available then conventional HTA methods can be confidently applied, without the need for promise to be an important element of decision making about adoption. Promise could perhaps be described as a positive judgement about effectiveness when evidence is limited.

Plausibility is also important: this means an informed judgment about whether various elements of promise are likely to be delivered by “real-world” use of a device or diagnostic test. Claims are often made for the benefits of new devices and diagnostics to patients and to health services which seem reasonable, but which, after critical scrutiny, are judged not to be plausible. This kind of scrutiny requires the collective view of a variety of well-informed people (like the NICE Medical Technologies Advisory Committee) with input from a range of experts, aimed at avoiding bias from just one or two individuals about what is plausible and what is not.

RISK

Safety is a fundamental consideration in making judgements about adoption of technologies. Regulatory systems in Europe, the United States, and elsewhere classify devices into different risk categories which influence the quality and quantity of evidence which is required for regulatory approval (Reference Santos, Gazelle, Rocha and Tavares10;11).

The fact that devices have been subject to regulatory scrutiny before coming onto the market means that safety concerns are seldom a major issue when making decisions about their adoption. Nevertheless, safety is an essential element which needs to be balanced with other considerations. If risks are high, then more evidence is likely to be required for a positive decision about adoption, and there needs to be confidence about the collection of data in the long term for implantable devices (Reference Campbell12). Any judgements about safety and risk need to be made in the context of the severity and impact of the condition which a device is intended to treat.

COST

From a healthcare provider's perspective, high cost drives expectation of large benefits; therefore, evidence for those benefits needs to be convincing before limited resources are deployed. Less evidence may be considered adequate for a favorable decision about adopting a technology when its purchase cost is low and particularly when potentially large savings could be realized by its successful implementation.

There are various ways of estimating cost, ranging from a sum of capital plus revenue costs to complex calculations of incremental cost effectiveness. For adoption of new devices and diagnostics, a cost consequences approach seems appropriate. This means estimating the cost of adopting a new technology in place of a clearly defined existing method of clinical management. This allows consideration of a wide range of data, including both direct costs and indirect costs (or savings) relating to less time in hospital, lower rates of complications, and any other issues which seem relevant. In this context, plausibility is again important: well-informed advice and judgements are needed to decide whether assumptions about each aspect of cost are reasonable and realistic. All this needs to be considered in the context of whether there is an expectation of reducing costs or whether a decision on adoption allows for increased benefit at some agreed level of increased cost.

BALANCING PROMISE, RISK, AND COST

Promise, risk, and cost all influence the amount of evidence likely to be required to make a positive decision about adopting a technology. We have described above the way that each of these interacts with the amount of evidence available and illustrate this graphically in Figure 1a–c.

Figure 1. Illustration of the relationship between (a) Promise, (b) Risk, and (c) Cost and the level of evidence (“Evidence”) likely to be required to support the adoption of a medical device or technology. Note that the orientation of the relationship for promise (a) is opposite that of both risk and cost (b,c). (d) Cost and risk are combined and plotted against promise because, in reality, these three factors influence each other in determining the level of evidence likely to be required to show that a given technology or device should be adopted for use. Thus, a high promise/low cost/low risk device is likely to require less evidence for adoption than a low promise/high cost/high risk one.

Note that while a low cost/low risk device might be adopted with a lower level of evidence than one which is high cost/high risk, a technology with a high level of promise might be so attractive to patients and the healthcare system that it is adopted despite sparse evidence. Decisions to adopt a technology with a low level of promise will require more evidence because any benefits are likely to be marginal (Figure 1a). We have limited the influence of promise on the amount of evidence required by a shallower “slope” in its relation to the evidence than the more conventional and tangible criteria of risk and cost.

In practice, judgements about promise, risk, and cost are not made in isolation; all three interact (Figure 1d). At the extremes, a high promise/low risk/low cost technology is likely to require less evidence for adoption than a low promise/high risk/high cost one.

PRESENTING A CASE FOR ADOPTION

The stimulus to introduce any technology into clinical practice naturally emanates from its manufacturer (or distributor). In the system used by NICE, manufacturers need to provide a clear list of claims for the advantages of their device or diagnostic to patients and to the health service, in place of current management. These claims are evaluated against available evidence by an independent committee, with a wide range of members, as described above. Our experience in receiving many claims of benefit from industry has been very mixed. In part, this is a reflection of the varying expertise and research capacity of the medical technology sector (Reference Craven, Allsop, Morgan and Martin13;Reference Sprange and Clift14). We recognize that evidence is often sparse but there does need to be at least some relevant and credible evidence to support claims of benefit for patients and health services.

Carefully considered claims of benefit often suggest the type of evidence that might support them. For example, a claim that a diagnostic technology will improve detection of a condition could be supported by research demonstrating improved sensitivity or specificity compared with current practice; and a really promising case will describe how improvement in diagnosis leads to improved outcomes. In general, if it is unclear even what type of evidence is required to support a claim (at least in principle) then the claim is probably too poorly constructed to be useful.

A plausible case for adopting a technology is often dependent (in contrast to pharmaceuticals) on the clinical and service context of its intended use, so clarity is needed about the patients and pathways being targeted and precisely what current practice the technology would replace or augment (Reference Gelijns, Russo and Hong5;Reference Taylor and Iglesias15). Unless these are well defined, it is very difficult to reach a judgement about the level of promise and whether the case for adoption is plausible.