The US Food and Drug Administration (FDA) has embraced “real-world evidence” (RWE) to evaluate the safety and efficacy of medical devices and drugs. However, the turn towards RWE remains controversial. Securing high-quality evidence after market entry can be a significant challenge. And concerns about the safety of several medical devices – discovered only after real-world use – have renewed calls for more rigorous pre and postmarket evaluation. Here, we discuss the shift toward RWE and the attendant challenges and concerns. Then, through a historical examination of the “Drug Efficacy Study Implementation” program (DESI), we argue that changing how RWE studies are conducted and who evaluates them might mitigate some concerns. Distributing the responsibility for designing, conducting, and assessing RWE beyond industry sponsors and the FDA is critical to producing – and acting upon – more clinically useful information about these products. We explore how the DESI program, which used third parties to examine the effectiveness of more than 3,000 drugs between 1963–1984, coupled with existing flexibilities in the law governing medical devices, provide both the inspiration and necessary conditions to support a DESI 2.0.
10.1 Introduction
A defining dilemma in regulating health products is balancing upfront scrutiny of safety and effectiveness prior to marketing with ongoing oversight during everyday use. Reliable evidence from both the premarket and postmarket phases is essential for both informed regulation and optimal clinical use. Yet the standards for evaluating this evidence are underspecified by law, challenged by innovation, and contested by a range of actors. In this chapter, we bring into conversation two types of products that have traditionally been subject to divergent regulation – drugs and devices – to illustrate both the challenges of, and potential opportunities associated with, increasing reliance upon what can be learned from the real world, so-called “real-world evidence” (RWE).
Since the mid-20th century, the US Food and Drug Administration (FDA) has relied on premarket data collection to demonstrate that products are safe and effective. In recent years, however, the agency has gradually relied more on evidence gathered on the postmarket side of the equation, typically under the auspices of expedited reviews designed to speed up market access to promising (if yet unproven) drugs.Footnote 1 Moreover, since the 1976 Medical Device Amendments, the vast majority of devices have gained entry into the US market by demonstrating substantial equivalence to a previously marketed device,Footnote 2 thus inviting the FDA to infer safety and efficacy on the basis of previous clinical use of older devices. Nevertheless, after a number of high-profile cases in which devices entered the market as substantially equivalent to older devices but later proved to carry significant risks,Footnote 3 the agency is under some pressure to revisit how it sets the evidentiary bar.
The FDA’s evidentiary standards, particularly the important balance between pre and postmarket evidence, are in flux. Section 10.2 details this shift and develops an argument that medical devices are especially ripe for regulatory experimentation. In Section 10.3 we pull from historical experience with drugs to describe key features of a new regulatory approach for devices. We theorize that the “Drug Efficacy Study Implementation” (DESI) program, which the FDA initiated in the 1960s, could be refashioned to improve both the quality of the evidence and the regulatory decisions made about medical devices. We see particular promise in expanding both evidence gathering and evidence evaluation to third parties outside both the FDA and industry. In concluding, Section 10.4 outlines potential stumbling blocks for a “DESI 2.0” for devices, which we hope will guide further development of this idea.
10.2 Evolving Standards for Drugs and Devices
10.2.1 Lifecycle Regulation and Real-World Evidence at the FDA
The idea that a product’s safety or efficacy profile might change significantly once used widely in the “real world” is far from new. Although controlled experiments help evaluate the safety and efficacy of a product in a target population, they may also mask important risks or exaggerate benefits that become apparent when the product is used over longer periods, in larger populations, and beyond the confines of strict trial protocols. Thus, the FDA has always been somewhat alert to how pre and postmarket experience with a product might differ. Even so, the FDA’s recent shift from pre- to postmarket data gathering and evaluation is both marked and remarkable. New sources of postmarket data are influencing the FDA’s upstream decisions about whether, and on what terms, to approve health products.Footnote 4
Of course, postmarket studies that are required by the FDA, or voluntarily undertaken by the sponsor, may still take the form of a randomized clinical trial (RCT). That is not what the FDA and others mean when they refer to real-world evidence (RWE) and real-world data (RWD). The former is essentially any evidence generated outside typical clinical research settings. The latter comes in multiple forms, including “electronic health records (EHRs), claims and billing activities, product and disease registries, patient-generated data including in home-use settings, and data gathered from other sources that can inform on health status, such as mobile devices.”Footnote 5 Some researchers are trying to replicate RCT findings using RWD,Footnote 6 and within at least some corners of the FDA, including recent FDA Commissioners,Footnote 7 the appetite for RWE is growing.Footnote 8
One linear account of this change is that Congress and others outside the agency have pushed the FDA toward a “lifecycle” approach to regulation that incorporates RWE. While the process has been mostly gradual, dating back to the HIV/AIDS crisis of the 1980s, the 21st Century Cures Act of 2016 marked a tipping point. The landmark legislation directed the FDA to consider nontraditional study designs and data analysis to streamline drug reviews;Footnote 9 apply the “least burdensome means” of approving devices, for instance, by factoring in the likelihood of RWD clarifying safety and effectiveness;Footnote 10 remove certain medical software from medical devices subject to FDA oversight;Footnote 11 establish an expedited regulatory pathway for “breakthrough” devices;Footnote 12 and develop guidance to incorporate RWE and patient experience data into its decision making for drugs and devices alike.Footnote 13
Another reading of this shift toward lifecycle regulation and RWE suggests that the FDA itself has shaped this regulatory arc. For example, the FDA established the accelerated approval process for drugs and the breakthrough program for devices years before Congressional direction – perhaps to safeguard the agency’s central role in pharmaceutical governance.Footnote 14
Whatever the motivations, the shift toward lifecycle regulation and RWE remains a work in progress. Postmarket studies regularly take years to completeFootnote 15 and seldom improve the evidence already gathered.Footnote 16 The FDA rarely threatens to impose fines or withdraw authorization when postmarket studies are delayed or the evidence does not confirm efficacy.Footnote 17 Moreover, the FDA’s legal authorities and resources to enforce postmarketing requirements are inadequate and the continuing dominance of the FDA’s reviewing divisions over its postmarket monitoring divisions compromises the agency’s ability to revisit initial decisions.Footnote 18 Meanwhile, numerous studies show – notwithstanding agency claims to the contrary – that the FDA has been applying a lower regulatory bar for approval of drugs, and the vast majority of medical devices escape formal scrutiny of safety and efficacy.Footnote 19
In sum, the FDA’s capacity to spur sponsors to generate reliable information about their productsFootnote 20 and to adjust regulatory decisions as the evidence evolves are each in serious question. It is time to consider new mechanisms to counter these shortfalls. The remainder of Section 10.2 details why medical devices, especially digital health products, offer an opportunity for the FDA to pursue this very sort of regulatory experimentation.
10.2.2 Signs of Regulatory Experimentation in Digital Health and Beyond
The FDA’s framework for regulating devices has not changed much since the 1976 Medical Device Amendments, despite radical technological advances.Footnote 21 Although a consensus now favors reform,Footnote 22 Congress has done little apart from calling for task force recommendations for how to regulate health IT products,Footnote 23 and trying to clarify which products fall within FDA jurisdiction.Footnote 24
In the absence of reform, the FDA itself has begun to experiment with new approaches. The agency’s 2017 Digital Health Innovation Action PlanFootnote 25 articulates three key departures from the FDA’s longstanding framework for devices: 1) shifting evidence gathering and evaluation from the premarket to the postmarket phase; 2) scrutinizing firms rather than products, using a new “Software Pre-Certification Program” to evaluate companies offering products; and 3) outsourcing market certification to independent, third-party reviewers, moving away from centralized agency review. While the first departure mirrors the agency’s lifecycle approach to drug regulation, the other two departures are unique, as centralized, product-specific reviews have been the lodestar of FDA regulation for roughly a century.Footnote 26,Footnote 27
Although the details of these new approaches are still in flux, they revolve around a few core ideas. First, shifting evidence gathering to the postmarket setting effectively grants sponsors a kind of conditional or phased authorization, with the expectation that postmarket evidence might confirm the device’s safety and efficacy.Footnote 28 The FDA has assigned the task of gathering such evidence to NEST, the National Evaluation System for health Technology,Footnote 29 a public-private initiative led by the FDA’s Center for Devices and Radiological Health (CDRH).Footnote 30 NEST is charged with collecting RWE from multiple sources, including electronic health records, insurance claims, pharmacy records, device registries, and patient-generated data (PGD).Footnote 31 As of 2019, the NEST network includes over 195 hospitals, 3,942 outpatient clinics, and fifteen coordinated registry networks that curate and analyze data.Footnote 32 NEST will organize data “into several standardized common data models (including domains such as demographics, diagnoses, procedures, and laboratory tests).”Footnote 33 Importantly, while NEST was originally proposed as a way to conduct postmarket surveillance to identify safety issues early,Footnote 34 it has broadened its focus to collecting data throughout the entire product lifecycle, using it not only for postmarket surveillance, but also for premarket review.Footnote 35 For example, the FDA said such evidence could be used to support a sponsor’s petition for device reclassification.Footnote 36 The data could also be used, ideally, to inform insurance coverage and reimbursement decisions, clinical practice, and patient adoption.Footnote 37
In 2018 and 2019, NEST solicited proposals for test cases to evaluate how well such data can be used to answer specific questions.Footnote 38 The latest round includes, for example, a study using insurance claims data to evaluate whether to expand the label for cardiac devices in children with congenital heart disease, and a trial using electronic health records and patient data to evaluate how well the Apple Watch ECG can detect irregular heart rhythms, to inform premarket submissions and postmarket surveillance.Footnote 39 The twenty approved test cases span a range of therapeutic devices (oncology, cardiology, vascular, orthopedic, etc.), a range of risk profiles (from low-risk 510(k) devices to higher-risk PMA devices), a range of data (retrospective and prospective), and a range of proposed uses (premarket, postmarket, and coverage decisions).Footnote 40 The test cases will also allow NEST to address concerns over the validity of studies using RWE, with expert committees focusing on the quality of the source data and designing appropriate methodologies for data analysis.Footnote 41
While the FDA’s shift toward lifecycle regulation has been the subject of growing critique,Footnote 42 NEST can add significant scientific rigor to the process of collecting and analyzing RWE to the benefit of “regulatory, clinical, and coverage decision making” not to mention “the health and the quality of life of patients.”Footnote 43 However, there are also reasons to be skeptical that this will occur, underscoring the need for even more radical experimentation, which we describe in Section 10.3.
10.3 A DESI for Devices?
Though the FDA has experimented with medical device regulation in recent years, mounting evidence that the move toward lifecycle regulation and RWE carries serious tradeoffs suggests more radical changes may be required. We draw inspiration from a historical program, designed and implemented by the FDA in the wake of 1962 legislative reforms, to envision even more modern advances to medical device regulation. We first describe the DESI program, then argue that existing legal authorities can and should be repurposed to support a DESI 2.0 for devices.
10.3.1 The “Drug Efficacy Study Implementation” Experiment
Although the 1962 Kefauver-Harris Amendments are widely considered to be foundational, the requirement that drug manufacturers show “substantial evidence” of effectiveness were preconfigured by agency practice. Safety, the sole criterion for market entry from 1938 to 1962, was understood by the FDA to encompass clinical utility or effectiveness beginning in the early 1950s.Footnote 44 Administrative innovation pre-staged congressional legislation.
The 1962 amendments likely emboldened the FDA, not only in terms of justifying heightened expectations for evidence of efficacy, but also in terms of using its administrative discretion to fashion solutions to problems perceived in the marketplace. Central among them was the question of what to do about the thousands of “old drugs” that had entered the market between 1938 and 1962, which were not formally evaluated for effectiveness prior to Kefauver-Harris. Congress did not explicitly require the FDA to review these old drugs,Footnote 45 but the agency read multiple sections of the legislation as all the mandate they needed.Footnote 46 Within a few years “DESI” was born.
DESI would come to evaluate some 3,400 old drugs for over 16,000 therapeutic indications over twenty-plus years.Footnote 47 To accomplish that feat, the agency understood that it needed a remarkable new structure. In 1966, under the leadership of Commissioner James Goddard, the FDA contracted the work to the National Academy of Sciences (NAS) and National Research Council (NRC).Footnote 48 The FDA not only lacked sufficient personnel for the task, but its personnel lacked the clout that NAS/NRC experts could command if and when difficult decisions had to be made to pull products from the market. The FDA created a centralized Policy Advisory Committee to define DESI’s procedures, which in turn spawned thirty review panels assigned to the therapeutic categories of the day. Each panel was comprised of a chair and approximately six NAS/NRC experts. They worked in confidence, delivering recommendations to the FDA about whether a given drug was “effective,” “probably effective,” “possibly effective,” or “ineffective.” Even though the panels did not conduct new research, each panel reviewed the medical literature for roughly 150 drugs, requiring 10,000 hours of expert scientific labor.Footnote 49
DESI drew lawsuits from industry as the FDA followed through on panel recommendations, announcing hundreds of drug withdrawals via the Federal Register.Footnote 50 The litigation was less about the involvement of outside NAS/NRC experts, and more to do with the summary-type procedures that the FDA had adopted in the name of efficiency. Notwithstanding firms’ legal challenges, the litigation ultimately failed. The Supreme Court largely validated the agency’s approach in the 1973 “Hynson quartet” of cases involving challenges to NDA withdrawals for preamendment drugs.Footnote 51 Even without explicit statutory authorization, in Hynson the Supreme Court refers to DESI as a “statutory mandate,”Footnote 52 and a Senate Report from 1972 refers to DESI as being “required by the Drug Amendments of 1962.”Footnote 53 Further, the Supreme Court upheld the FDA’s power to use summary procedures, ruling that firms’ expectations of a full administrative hearing to decide the fate of a drug was conditional upon having first produced “substantial evidence” of effectiveness. Where such evidence is lacking, the Court held, a full hearing need not follow.
The implications of DESI are manifold. But the move to engage outside actors in the decision-making process is underexamined. If the FDA’s inability to encourage high-quality evidence production are ultimately reflective of a kind of incumbency – both in terms of who is involved in producing and how it is appraisedFootnote 54 – then regulation may take as its inspiration DESI’s disruptive move to bring outside actors into the regulatory fold. In the realm of medical devices, recent FDA initiatives such as NEST show some willingness to do this.
But the success of a DESI 2.0 for devices may depend on coupling 1) third-party evidence generation with 2) third-party reviews of that evidence – two functions which neither the original DESI nor more recent initiatives like NEST have sought to combine. Third-party evidence generation and reviews might significantly strengthen the use of “real-world” signals beyond what the FDA and/or industry is either capable or willing to do, making more meaningful recent odes to total lifecycle regulation and postmarket surveillance.
10.3.2 Repurposing Existing Legal Authorities to Support a “DESI 2.0”
Allowing third parties to both generate evidence and conduct rigorous product reviews is a less radical idea than we might think. Data are now available through many different sources, including massive device registries.Footnote 55 And the sheer volume of devices introduced into the market, particularly in digital health, augers in favor of outsourcing some portion of review of safety and efficacy. It is the joining of these two functions and empowering third parties to fulfill them that is crucial.
A threshold question is whether a DESI 2.0 would be legally permissible. Just as the Kefauver-Harris Amendments were interpreted by the FDA as authorizing the original DESI, the current statute is flexible enough to support both device reviews and evidence generation by third parties. First, the statute very broadly requires the FDA to “consider, in consultation with the applicant, the least burdensome appropriate means of evaluating device effectiveness that would have a reasonable likelihood of resulting in approval,” unless “contrary to public health.”Footnote 56 Moreover, as with DESI, the statute entrusted initial review and classification of pre-1976 devices to expert panels, and envisioned that the FDA could turn to expert panels to review classification petitions.Footnote 57 In both cases, panel decisions are recommendations published and reviewed by the FDA.Footnote 58 Likewise, the statute authorizes the FDA to withdraw or suspend PMA approvals, particularly when “new information” is presented.Footnote 59 FDA rules make clear that the agency “may seek advice on scientific matters from any appropriate FDA advisory committee” and “may use information other than that submitted by the applicant” in deciding whether to withdraw approval of a PMA.Footnote 60
Section 523 of the FDCA provides more specific authority for the kind of dual-function third-party mechanism we imagine. In 1997 Congress amended the statute to codify a five-year pilot program to allow the FDA to accredit third parties to review 510(k)s and make nonbinding recommendations to the agency.Footnote 61 The FDA initiated the pilot in 1996, before it received statutory authorization.Footnote 62 The idea was to provide manufacturers of certain devices “an alternative 510(k) review process that could yield more rapid marketing clearance decisions” and preserve FDA review for higher-risk devices.Footnote 63 The FDA published accreditation criteria in 1998,Footnote 64 and the pilot has been renewed by Congress every five years since 2002.Footnote 65 Currently, only eight entities are accredited for the renamed “3P Review Program.”Footnote 66 Although 510(k) user fees are waived and FDA clearance is 29 percent faster when recommended by an accredited third party, the program remains underutilized.Footnote 67
Despite possessing sufficient legal authority to create a DESI 2.0, the FDA might seek more clear statutory authorization from Congress in order to act upon third-party evidence and recommendations. Although the FDA’s authority to adopt summary-type procedures for devices would be supported by the Hynson quartet of Supreme Court decisions, more aggressive reliance on third-party reviews might need clearer statutory support. To wit, after the FDA announced its Digital Health Action Plan and software precertification program, several Senators sent a letter to the FDA questioning the agency’s statutory authority to do so.Footnote 68 Indeed, the FDA’s announcement of the Action Plan itself acknowledged that it may lack statutory authority for third-party precertification.Footnote 69 However, there is a long history of the FDA relying on panels and advisory committees to make nonbinding recommendations regarding product approvals, classifications, and withdrawals. A similar system, whereby NEST (or some other third party) would be empowered not only to analyze newly collected data but also make recommendations to the agency about appropriate regulatory actions in light of that evidence – ranging from label changes to product withdrawal – would seem to be within FDA authority. Advisories and recommendations are, by their very nature, nonbinding, though their publication would force the FDA to offer compelling justifications for making decisions contrary to the recommendations.
Thus, the stars seem well aligned for a DESI 2.0 for devices. Agency practice presages it. Intense cooperation with third parties to develop new sources of RWE and deploy them for regulatory decisions presages it. The statutory authority remains broad and arcs in that direction. And, perhaps most importantly, the need is clear. The FDA itself remains unable to give adequate attention to the sheer volume and variety of new devices. If vogue ideas like RWE, RWD, and total product lifecycles are to gain real traction, formalizing and inviting third-party participation seems crucial.
10.4 Potential Obstacles and Future Research
While the law may not be an immediate obstacle to creating a DESI 2.0, industry, institutional, and scientific obstacles remain. For starters, medical device manufacturers are likely to challenge any such initiative in Court. In Hynson, industry contested the agency’s authority to adopt summary procedures, which the Supreme Court upheld. In recent years, lower courts have, at times, endorsed exceedingly low standards for what constitutes “substantial evidence.”Footnote 70 If a mere scintilla of evidence was sufficient to trigger a formal evidentiary hearing before any decision to withdraw a device from the market, any efficiencies to be gained from third-party reviews would be seriously undermined.
A second set of obstacles is more institutional in nature. On one hand, the external academic researchers affiliated with NEST have incredible credentials, but it is not obvious that they command the level of deference from the FDA that the NAS/NRC once did. Elite universities have established relationships with the FDA;Footnote 71 how critically these academic units would approach the task of generating robust new evidence and, when warranted, reversing prior agency decisions, is not known. In this regard, potential conflicts of interest (especially financial conflicts) are a potential concern. The work of FDA advisory committees has been plagued by conflicts, so ensuring that a DESI 2.0 retains a strong independenceFootnote 72 with respect to each device evaluated may prove critical to the initiative’s success. More generally, nongovernment certification has a spotty track record, from longstanding critiques of Joint Commission accreditation of hospitals for Medicare,Footnote 73 to familiar critiques of third-party certification of “meaningful use” for electronic health records (EHRs),Footnote 74 to more recent critiques of the Federal Aviation Administration (FAA) allowing Boeing to self-certify its 737 Max aircraft (later recalled after multiple crashes).Footnote 75 These examples demonstrate the need for traditional regulatory compliance monitoring and enforcement as a backstop to any third-party recommendations.Footnote 76
Thirdly, there are also scientific obstacles to implementing a DESI 2.0. As noted above, standards for generating RWE from a variety of real-world data are a work in progress. There is serious scientific debate about the strength of different kinds of RWE for different types of health interventions, not to mention when such evidence should motivate regulatory action. Anticipating these debates, the major trade associations like AdvaMed and BIO have commented on the FDA’s use of RWE.Footnote 77 Committing DESI 2.0 to transparency – in terms of the data it generates, its analyses, and recommendations – can serve not only to enhance trust, but also to refine scientific standards.Footnote 78
DESI was a watershed moment in the history of medical product regulation, using outside review panels to evaluate evidence of clinical efficacy for thousands of products. Although DESI was encouraged by watershed legislation, the Kefauver-Harris Amendments did not clearly authorize it. Today, the FDA is being pushed toward lifecycle regulation and reliance on so-called “real world evidence” to evaluate products. Whether this shift is successful or not depends, we think, on whether the FDA can learn important lessons from the DESI experiment with pharmaceuticals in the 1960s–80s. In particular, third parties may be useful not only in generating RWE on specific products but also evaluating such evidence to support the FDA’s regulatory decision making.
Open access