Step 1a: Define the Causal Question and Estimand

The question for all study designs was what would the difference in risk of MACE (defined as death from any cause, nonfatal MI, or nonfatal stroke) within one year be if all patients in a target population consistent with the PIONEER 6 inclusion/exclusion criteria and timeframe [Reference Husain, Birkenfeld and Donsmark16], and with similar healthcare engagement, were prescribed oral semaglutide plus standard-of-care compared to if all patients were prescribed standard-of-care alone, and if censoring had been prevented for all patients? The outcome for this case study includes all-cause death (rather than cardiovascular death as in PIONEER 6) because the RWD does not include cause of death. The target population for all designs is a population of patients that not only meet the trial eligibility criteria but also are likely to enroll in a trial, based on their disease status and healthcare engagement. We do not aim to generalize the results of the trial to a new population. See Appendix 1 (Supplementary material 1) for the causal estimand.

Step 1b: Specify a Causal Model

Next, we specify a causal model for each design describing what variables might affect treatment, censoring, or outcomes using the causal graphs [Reference Pearl25] shown in Fig. 2. For the RCTs, only the randomization procedure affects treatment assignment. As depicted in Fig. 2a, health status, socioeconomic status, and related issues of healthcare access and engagement, collectively referred to as U, might affect both censoring and MACE. Measured pre-baseline covariates, including age, sex, race, HbA1c, high-density lipoprotein cholesterol, low-density lipoprotein cholesterol, estimated glomerular filtration rate (a marker of kidney function), prior MI, prior stroke or transient ischemic attack (TIA), prior heart failure, morbid obesity, and use of glucose-lowering medications, insulin, and CV medications, may account for some aspects of these underlying factors.

Figure 2. Causal graphs for Designs 1–3. DPP4i=dipeptidyl peptidase-4 inhibitor; MACE=major adverse cardiovascular events; NCO=negative control outcome; RCT=randomized controlled trial; RWD=real-world data; SOC=standard-of-care; SES=socioeconomic status.

In hybrid Design 3 (Fig. 2b), participation in the RCT versus the real-world system affects treatment because RCT participation is required to receive oral semaglutide if the RWD is concurrent with the pre-approval RCT. RWD controls receiving standard-of-care are selected based on initiation of a dipeptidyl peptidase-4 inhibitor (DPP4i), where conditioning on DPP4i initiation in the RWD is represented by the box around treatment in Fig. 2b. This choice is discussed in Step 3 and Appendix 2 (Supplementary material 1).

Being in the RCT could also modify the effect of treatment or directly affect measured outcomes for reasons including closer monitoring, encouragement of adherence, variation in standard-of-care or placebo effect, or more accurate outcome measurement [Reference Pocock26–Reference Chow, Habermann and Abraham28]. The RCT inclusion and exclusion criteria, the timeframe of RCT recruitment, health status, socioeconomic status (SES), and healthcare engagement or access may also affect trial participation and MACE. Type of clinical site also affects trial participation – because the RCTs are conducted at research sites globally while the RWD comes from the United States and includes primary care practices – and may affect baseline characteristics, the definition of standard-of-care, censoring, and/or MACE.

Fig. 2b also includes fractures as a negative control outcome (NCO). NCOs are sometimes used to detect bias in observational studies [Reference Lipsitch, Tchetgen Tchetgen and Cohen29,Reference Arnold, Ercumen, Benjamin-Chung and Colford30]. An appropriate NCO is an outcome that is not affected by treatment but that is affected by the same sources of bias in the treatment effect estimate as the primary outcome. An observed association between the treatment and the NCO is then due either to finite sample variability or due to bias from sources including confounding, selection bias, measurement error, etc. [Reference Lipsitch, Tchetgen Tchetgen and Cohen29,Reference Arnold, Ercumen, Benjamin-Chung and Colford30]. From the available options, we chose fractures as an NCO because this outcome is generally serious enough to require medical attention for those with access, is associated with SES [Reference Valentin, Ravn and Jensen31], and is recorded in a manner similar to the primary outcome. Fractures is not an ideal NCO, however; although closer monitoring in the RCT might improve cardiovascular outcomes, we do not expect closer monitoring to lead to decreased numbers of fractures (no yellow arrow from RCT v. RWD to the NCO in Fig. 2b). Studies prospectively designed to include an NCO could consider alternate choices to more comprehensively capture potential sources of bias.

Step 2: Consider the Observed Data

Next, we consider the data that will be observed. Designs 1–2 propose to use data from one or both of the following RCTs: PIONEER 6 [Reference Husain, Birkenfeld and Donsmark16] and SOUL [Reference McGuire, Busui and Deanfield17]. While PIONEER 6 has ended, the data from SOUL are not yet available. Both trials randomized participants to receive semaglutide or placebo plus standard-of-care. The inclusion and exclusion criteria for both trials targeted patients with T2DM and high cardiovascular risk but without unstable disease or recent use of a GLP-1RA, while PIONEER 6 also excluded recent users of pramlintide and DPP4is [Reference McGuire, Busui and Deanfield17,Reference Bain, Mosenzon and Arechavaleta19]. Participants were regularly evaluated in person or by phone. Outcomes were adjudicated. The timeframe for the outcome of one year after baseline was selected because no administrative censoring occurred before that time in PIONEER 6.

The external control arm considered in Design 3 could come from multiple sources, including the control arm of a previous trial such as SUSTAIN-6. While we would expect controls enrolled in a previous trial to be more similar to the current trial controls than participants in RWD, in this case study, we chose to consider RWD controls in order to discuss the many challenges that might arise in a hybrid study when a past trial’s control arm is not available. The RWD also has the advantage of being contemporaneous with the RCT, avoiding bias due to temporal differences between the RCT and external control arms. Specifically, the RWD considered in Design 3 came from Optum’s de-identified Clinformatics® Data Mart Database (CDM) (2007–2022), which is derived from administrative health claims for a geographically diverse population, spanning all 50 of the United States.

Consistent with recommendations from the RCT DUPLICATE study, we used naive initiation of a DPP4i (defined as a new prescription following at least 90 days without a previous prescription based on AHFS codes) to enhance comparability of healthcare access and engagement among RWD compared to RCT controls [Reference Franklin, Patorno and Desai32]. Time zero was defined as the first time a participant met the eligibility criteria for PIONEER 6, during a calendar time window contemporaneous with PIONEER 6 recruitment, and was prescribed a DPP4i.

We applied as many of the PIONEER 6 inclusion/exclusion criteria as possible given the available CDM data (Supplementary material 1, Appendix 7). Medical history variables that were part of these criteria or the baseline characteristics discussed in Step 1b were determined based on ICD-9/ICD-10 diagnosis and procedure codes. Medication use was identified through AHFS codes, where treatment at baseline corresponded to at least one prescription in the 180 days preceding time zero. To identify GLP1-RA and pramlintide usage for the exclusion criteria, we defined continuous treatment eras as consecutive prescriptions based on AHFS codes with no more than 90-day gaps between them. Laboratory measurements were identified through LOINC codes, where baseline values were identified as the most recent measurement within 180 days prior to time zero. For the primary and negative control outcomes, we identified nonfatal MI/stroke and fractures using ICD-9/ICD-10 codes for inpatient visits in the first diagnosis position (Supplementary material 1, Appendix 7). All-cause death was identified from external sources as provided by Optum.

Steps 3-4: Assess Identifiability and Specify a Statistical Estimand

We now discuss whether the causal effect of interest (Step 1a) is identifiable from the observed data (Step 2). When using RCT data alone (Designs 1–2), we assume no unmeasured common causes of treatment or censoring and MACE and adequate data support (positivity) [Reference Hernan33,Reference Petersen, Porter, Gruber, Wang and van der Laan34]. These assumptions are highly likely to hold by design; while it is possible that there are unmeasured common causes of censoring and MACE because censoring was negligible (0.3% in PIONEER 6), this would be unlikely to impact the results. An alternative design (distinct from Designs 1–3) in which we committed to augmenting the RCT data with external control data would require additional assumptions: (1) no unmeasured common causes of MACE and either selection into the RCT versus the RWD cohort, definition of standard-of-care in the RCT versus RWD, or censoring (no U in Fig. 2b), (2) no direct effect of RWD versus RCT participation on MACE or outcome measurement, and (3) adequate data support (positivity [Reference Hernan33,Reference Petersen, Porter, Gruber, Wang and van der Laan34]).

In contrast, rather than rely on these assumptions holding, Design 3 estimates the magnitude of bias that would be introduced by RWD before deciding whether to integrate RWD with RCT data. If violation of these assumptions results in meaningful bias, Design 3 is likely to reject the RWD controls. In selecting candidate external controls for Design 3, the objective is to satisfy these assumptions to the extent possible, thereby increasing the probability that RWD will be integrated into the hybrid analysis, decreasing the probability that a follow-up superiority trial will be necessary, and minimizing risk of lower than nominal confidence interval coverage.

To improve the plausibility of identification assumptions, we selected RWD controls with a similar disease stage and healthcare engagement compared to the RCT controls based on those who were prescribed an active comparator medication (DPP4i) and had relevant baseline labs and medical history recorded. The aim was to minimize the impact of factors like health status and healthcare engagement on selection into the RCT versus the external cohort and on the definition of standard of care in the two studies. Yet this step also restricts the target population for the hybrid analysis to patients similar to those who actually enrolled in the RCT. Because DPP4i use was an exclusion criterion in PIONEER 6, we also require the assumption, supported by data, that DPP4is do not influence cardiovascular outcomes [Reference Franklin, Patorno and Desai32]. We also restricted RWD controls to the time period of PIONEER 6 recruitment to make standard-of-care more similar, and we selected RWD controls whose baseline characteristics were shared by at least some RCT participants. This last step was necessary to avoid a violation of the positivity assumption and further restricted the target population.

As is generally true when observational data are considered, it remained unlikely that all causal identification assumptions held completely. Yet with the above considerations, it was plausible that the bias from integrating RWD would be small. Design 3 evaluates this claim empirically. Appendix 1 (Supplementary material 1) provides statistical estimands that are equivalent to the treatment effects of interest, or causal estimands, under identification assumptions. Appendix 2 (Supplementary Material 1) provides further discussion of these assumptions.

Step 5: Choose a Statistical Model and Estimator

Next, we chose a statistical estimator. In Designs 1–2, because censoring was negligible, we estimated the unadjusted risk difference between arms among persons with follow-up through one year. For Design 3, we used an estimator designed to analyze hybrid studies.

Multiple hybrid-design estimators have been developed in recent years, with the common goal of evaluating whether to include (or how to weight) external data in a hybrid analysis based on the difference in mean outcomes, conditional mean outcomes, or effect estimates estimated using external data or estimated using RCT data alone. These estimators differ based on whether they take a Bayesian [Reference Hobbs, Carlin, Mandrekar and Sargent35–Reference Lin and Evans38] or Frequentist [Reference Dang, Tarp and Abrahamsen5,Reference Chen, Zhang and Ye6,Reference Yang, Zeng and Wang24,Reference Cheng and Cai39,Reference Oberst, D’Amour, Chen, Wang, Sontag and Yadlowsky40] approach, their criteria for deciding whether to include or how to weight external data, the estimator used for effect estimation, and the method of confidence interval construction. We refer the interested reader to Lim et al. (2018) [Reference Lim, Walley and Yuan4] and Oberst et al. (2022) [Reference Oberst, D’Amour, Chen, Wang, Sontag and Yadlowsky40] for more detailed explanations of alternative methods.

In this case study, we use the experiment-selector cross-validated targeted maximum likelihood estimator (ES-CVTMLE) [Reference Dang, Tarp and Abrahamsen5] as the estimator for Design 3. Cross-validated Targeted Maximum Likelihood Estimation (cv-TMLE) is a robust, efficient approach that incorporates machine learning using cross-validation [Reference van der Laan and Rose7,Reference van der Laan, Polley and Hubbard41–Reference Hubbard, Kherad-Pajouh and van der Laan43]. This allows it to flexibly adjust for covariates without introducing new assumptions, improving precision and potentially reducing bias while preserving inference [Reference van der Laan and Rose7,Reference van der Laan, Polley and Hubbard41–Reference Hubbard, Kherad-Pajouh and van der Laan43]. Censoring is handled by incorporating inverse probability of censoring weights into the TMLE targeting procedure [Reference Gruber and van der Laan44].

The ES-CVTMLE extends the cv-TMLE method to evaluate and integrate external data [Reference Dang, Tarp and Abrahamsen5]. First, the ES-CVTMLE estimates the bias introduced by augmenting the RCT with RWD by comparing the conditional mean outcomes for RCT and RWD controls (see Supplementary material 1, Appendix 3) [Reference Dang, Tarp and Abrahamsen5]. ES-CVTMLE also estimates bias as the estimated average treatment effect on the NCO. Dang et al. (2022) describe how these two bias estimates are used to decide whether to integrate RWD or to analyze the RCT data alone [Reference Dang, Tarp and Abrahamsen5].

In this case study, we chose the ES-CVTMLE for Design 3 because it relies on few statistical assumptions, incorporates an estimate of bias based on an NCO, and adjusts confidence interval widths based on the estimated magnitude of bias. While the focus of this case study is on using simulations to compare study designs, similar simulations could also be used to compare different potential estimators. Appendices 3–6 provide further details about estimation.

Step 6: Specify a Procedure for Sensitivity Analysis

Causal Roadmap Step 6 is to conduct a sensitivity analysis to evaluate how violations of identification assumptions might impact conclusions. Because the topic of sensitivity analysis is complex, we do not cover this step in this case study but instead refer readers to the companion paper that reviews sensitivity analysis in detail.

Step 7: Compare Study Designs Using Simulations

We compare Designs 1–3 using simulations that mimic our true study designs (Supplementary Material 1, Appendix 4). The code to run these simulations may be found in Supplementary Material 2. For Designs 1 and 3, we simulate data from a small RCT aiming to mimic PIONEER 6 (RCT1). For Design 1, we use an unadjusted estimate of the difference in risk between arms of RCT1. For Design 3, we consider simulated RCT1 and a simulated “real-world” dataset aiming to mimic CDM, and we use the ES-CVTMLE to estimate the risk difference. For the sake of comparison to a more standard estimator, we also report the results of simulated Design 3 if a cv-TMLE with no assessment of bias had been used to analyze the pooled RCT and RWD. In both Designs 1 and 3, if the null hypothesis is rejected at this first stage, we use this initial estimate as our final effect estimate. If the null hypothesis is not rejected, then we simulate data from a larger trial aiming to mimic a superiority trial (RCT2) and estimate the risk difference using an unadjusted estimate. In Design 2, we use the unadjusted effect estimate from RCT2.

Because Design 3 considers non-randomized data, it is possible that confidence interval coverage will be lower than in Designs 1–2 if both the causal identification assumptions (Step 3) are violated and the empirical bias estimate based on the available finite sample is smaller than the true bias (leading to inappropriate inclusion of external controls). Whether this risk is acceptable will depend on the context, including the magnitude of benefit to patients of Design 3 over Designs 1–2. While there are many potential ways to quantify benefit to patients, here we estimate the number of patient-years during which participants are precluded from starting a GLP1-RA (by being in an RCT control arm), averaged over 1000 iterations of this simulation. We also report power to detect the true effect using α = 0.05. Because participation in an RCT requires acceptance of potential risks and commitment of time to the study, it is important that any design that involves an RCT be adequately powered to answer the question of interest [Reference Hernán45].

We evaluate Design 3 when the magnitude of bias introduced by including simulated RWD is zero and when it is one of ten potential magnitudes in either direction up to ± 2.1%. In this primary simulation, the effect of unmeasured factors causing bias is the same on the relationship between semaglutide and MACE as it is on the relationship between semaglutide and the NCO. Appendix 5 (Supplementary material 1) shows the results of the same simulation both when the NCO is not considered and when the NCO is considered but the unmeasured factors causing bias for the relationship between semaglutide and MACE have no effect on the NCO, mimicking a worst-case scenario for violations of both the causal identification assumptions and the assumptions needed to measure bias using an NCO.