Introduction
Although research comparing the effectiveness of healthcare interventions and strategies has been conducted for more than a century, the term comparative effectiveness research has taken on new meaning (What Is Comparative Effectiveness Research?, 2009). According to the Agency for HealthCare Research and Quality (AHRQ), CER is designed to guide healthcare decisions by providing evidence about the effectiveness, benefits, and harms of different treatment options. The evidence is generated from research studies that compare drugs, medical devices, tests, surgeries, or ways to deliver healthcare (Agency for HealthCare Research and Quality, 2013). Although CER has traditionally been used to compare treatment options, clinical trials, tests, and procedures, the opportunities for chronic disease prevention and control programs are only now being recognised. This paper provides the methodology used to conduct CER in tobacco cessation interventions.
Tobacco use is the leading cause of preventable disease and death in the United States, causing approximately 443,000 deaths annually (Smoking-attributable mortality, years of potential life lost, and productivity losses–United States, 2008). Tobacco use contributes to pulmonary disease, infertility, birth defects, and cardiovascular disease (Department of Health and Human Services, 2010). Furthermore, tobacco use is a risk factor for cancers of the bladder, cervix, oesophagus, kidney, larynx, lung, oral cavity, pancreas, stomach, and acute myelogenous leukaemia (The 2014 United States Surgeon General's Report, 2014).
Two public health programs funded by the Centers for Disease Control and Prevention (CDC) have activities and interventions aimed at reducing tobacco use. The NCCCP funds 65 state, tribal, and territorial cancer control programs throughout the country. These programs focus on six priority areas: emphasis of primary prevention of cancer; support of early detection and treatment activities; addressing public health needs of cancer survivors; implementing policy, systems, and environmental changes to guide cancer control; promoting health equity; and demonstrating outcomes through evaluation (www.cdc.gov/cancer/ncccp). CDC established the NTCP to reduce disease, disability, and death related to tobacco use. The goals for comprehensive tobacco control include the promotion of quitting among adults and youths, prevention of initiation among youth and young adults, elimination of exposure to second-hand smoke, and elimination of tobacco-related disparities among different populations (www.cdc.gov/tobacco/tobacco_control_programs/ntcp/index.htm) (CDC, 2007). This comprehensive approach combines regulatory, economic, educational, clinical, and social strategies (CDC, 2011). The NTCP provides funding for state quitlines, which provide telephone-based smoking cessation services, including individualised telephone counselling and self-help material to help smokers quit. Quitlines are cost-effective, population-based interventions that increase successful smoking cessation (Task Force on Community Preventive Services, 2001), and have the potential of reaching underserved populations (Fiore et al., Reference Fiore, Bailey and Cohen2008; Lichtenstein, Zhu, & Tedeschi, Reference Lichtenstein, Zhu and Tedeschi2010). Quitline services in all states can be accessed through a toll-free number, 1-800-QUIT-NOW.
This project was developed to compare existing and traditional tobacco cessation interventions with innovative Web-based interventions. This study includes three components (i.e., cessation, promotion, partnership) and was designed with unique study questions and methods for each component of the study.
Methods
Cessation Component 1: Quantitative Study
Study questions
The cessation component assessed the effectiveness of telephone quitlines versus Web-based tobacco cessation programs by comparing the prevalence of 30-day abstinence that was measured seven months after enrolment. To date, four studies have specifically compared the two intervention modalities in either insured populations or a single state (An et al., Reference An, Betzner, Schillo, Luxenberg, Christenson and Wendling2010; Graham et al., Reference Graham, Cobb, Papandonatos, Moreno, Kang and Tinkelman2011; Swan et al., Reference Swan, McClure, Jack, Zbikowski, Javitz and Catz2010; Zbikowski et al., Reference Zbikowski, Hapgood, Barnwell and McAfee2008). This study's large sample size and multistate collaboration will provide much needed data to compare state-funded telephone quitlines and Web-based tobacco cessation programs. The study questions were as follows: (a) What are the demographic profiles of participants who enrol in a specific intervention (e.g., telephone-based counselling or Web-based intervention)? (b) Do baseline smoking and quitting behaviours of participants differ between those who enrol in telephone-based counselling versus a Web-based cessation program? (c) What are the predictors of successful quitting (e.g., prevalence of 30-day abstinence) for each intervention? and (d) Does quitting success differ significantly among participants who enrol in telephone-based counselling versus those who enrol in a Web-based cessation program?
Study sample size
The study aimed to recruit 8,000 participants aged 18 years or older (4,000 quitline users and 4,000 Web-based intervention users) from Alabama, Arizona, Florida, and Vermont. Two studies helped guide the power calculation (An et al., Reference An, Betzner, Schillo, Luxenberg, Christenson and Wendling2010; Zbikowski et al., Reference Zbikowski, Hapgood, Barnwell and McAfee2008). An and colleagues reported a 16% point difference between the 30-day point prevalence abstinence (29% for telephone users versus 13% for Web-based users) at the six-month follow-up among a sample of Minnesota's QUITPLAN program (n = 7,049; 4,968 Web site users and 2,351 telephone users). Similarly, Zbikowski and colleagues reported an 11% point difference between 30-day quit rates at the six-month follow-up among 11,143 quitline users who were using an employee or healthcare plan cessation program that included integrated phone and Web-based services. The quit rates for each intervention were 18% for telephone and 7% for Web. We used power and sample size calculations based on methods published by Rosner (Reference Rosner1995). Calculations assumed an alpha of 0.05, power of 0.80, equal sample sizes, and an oversample of 50% (anticipating an approximate participation rate of 50%) to determine a 3.0% difference between interventions. This approach was conservative because loss to follow-up rates varied greatly among previous phone-based cessation studies, from 4% to 55% (Stead, Perera, & Lancaster, Reference Stead, Hartmann-Boyce, Perera and Lancaster2006).
State programs varied for the contracted program vendor, tobacco control interventions, and quitline call volume to meet sample size requirements. All states that had separate telephone and Web-based interventions, that conformed to National Quitline Data Warehouse standards for intake and follow-up data collection, were eligible to participate.
Data collection methods
The study was reviewed by CDC's Human Subjects Research Protection Office, and Institutional Review Board (IRB) approval was obtained. Selected states received IRB approval before data collection. In compliance with the Paperwork Reduction Act, the data collection was approved by the Office of Management and Budget (Control #0920-0917). We obtained permission for follow-up contact when the participant first registered for tobacco cessation services. Participants who completed the follow-up survey were offered a $40 incentive as compensation for their time. Intake and follow-up data were provided by state-contracted cessation service companies that collected information for users of both telephone and Web-based interventions as part of their normal monitoring and evaluation activities, or the data were collected by the CDC contractor for this study.
Intake data conformed to the North American Quitline Consortium's (NAQC’s) Minimum Data Set (MDS) requirements to ensure uniform information collection (North American Quitline Consortium (NAQC), 2009). Intake measures for quitline users include demographics, current smoking status, smoking history, current smoking behaviours, previous quit history, self-reported source of referral to the program, and services requested from the program (North American Quitline Consortium (NAQC), 2009). Similar Web-based intake questionnaires use comparable measures formatted for Web administration (Civljak et al., Reference Civljak, Sheikh, Stead and Car2010).
This study component used the standard measure of cessation program effectiveness by recontacting participants seven months after intake to ask about tobacco use during the past 30 days (An et al., Reference An, Betzner, Schillo, Luxenberg, Christenson and Wendling2010; Civljak et al., Reference Civljak, Sheikh, Stead and Car2010). Follow-up was conducted that entailed contact attempts by e-mail, standard mail, and phone (Dillman, Reference Dillman2007). To increase the response of Web-based participants, investigators delivered recruitment follow-up letters by express mail service at the midpoint of the study.
This survey component was based upon the NAQC requirements, and specific questions were added to address study questions. This approach collected information about participants’ satisfaction with the cessation intervention, current smoking status, quit attempts made since their enrolment, duration of quit (if applicable), use of nicotine replacement therapies or medication to help quit, use of various behavioural smoking cessation strategies, and use of technology and various forms of media. The main outcome of interest was prevalence of 30-day abstinence, defined as not having smoked any cigarettes, even a puff, during the past 30 days.
Service providers were asked to provide information about the number and length of interactions (e.g., telephone calls or signing into the Web site) completed by telephone and Web-based intervention users. Variables collected are listed in Table 1.
Table 1 Cessation component 1 variables
Analysis and outcomes of interest
Descriptive statistics were used to summarise sample characteristics of users who enrolled in each type of intervention (telephone vs. Web-based program), their levels of program satisfaction, and use rates (e.g., number of calls, Web entries, use of program features). Bivariate and multivariate regression modelling was used to assess the association between mode of intervention and successful quitting. The main outcome measured was prevalence of 30-day abstinence (i.e., quitting success) with select secondary outcomes as noted in Table 1. These models adjusted for demographic and baseline smoking characteristics. In addition, we included state indicators in each regression model to capture any latent differences across states that were not explicitly accounted for by the covariates. We used a model-driven approach to identify whether variables significantly affected the outcome while controlling for other factors. Confounding, effect modification, and collinearity were considered where relevant.
Promotional Strategies Component 2: Quantitative Study
Study Questions
The primary objective of the promotional component was to compare the effectiveness of traditional versus innovative promotional strategies to increase telephone quitline call volume. Given the level of funding available for media promotions, as well as the changing media landscape, it is important to understand how states’ usage of innovative promotion efforts, such as digital advertising and social media, affect telephone quitline call volume. This study component addresses the following research questions: (a) What innovative promotional and educational activities are states implementing for cessation in general and for the quitline? (b) What is the reach and use of states’ cessation Web sites and social media platforms? (c) How do audiences reached via innovative media platforms compare with audiences reached via traditional media platforms? (d) What is the comparative effectiveness of traditional media versus online advertising in driving calls to the quitline?
Study Sample Size
We recruited TCPs from the 50 states and U.S. territories to participate. Twenty-four states agreed to participate in the study component. These states were as follows: Alabama, Arizona, Arkansas, California, Delaware, Florida, Indiana, Iowa, Louisiana, Massachusetts, Michigan, Missouri, Nebraska, Nevada, New Hampshire, New York, North Carolina, Oregon, Rhode Island, Texas, Utah, Vermont, Wisconsin, and Wyoming.
Data Collection Methods
Media purchase and placement information for smoking cessation promotional strategies and quarterly quitline call volume data were obtained from each participating state TCP for up to 21 months during October 2011–June 2012. Investigators analysed this data submitted by the 24 states to identify and describe both traditional and innovative promotional efforts, and to comparatively assess the impact of these efforts on quitline call volume and caller characteristics. There were no eligibility requirements for participation, and states were allowed to select the data items that they submitted to CDC.
Analysis and Outcomes of Interest
Traditional Media Purchase and Placement Data
Each promotional strategy was classified as either traditional (i.e., television, radio, print, or out-of-home) or innovative (i.e., digital, paid search, or social media). Appropriate metrics were identified to measure the level of exposure or reach of each type of advertising. For instance, Gross Rating Points (GRPs), the product of reach times frequency, was used to measure level of exposure to television and radio advertising. The number of impressions, (the estimated number of times a print advertisement or billboard was viewed), was used to measure level of exposure for print advertising. Finally, Click through Rates (CTRs), (the proportion of viewers who clicked on an advertisement out of the total impressions delivered), were used to measure exposure to online banner ads.
Innovative Quitline Call Volume Data and Online Activity
Investigators worked with participating state TCP staff and telephone and Web-based vendors to gather media activities data, Web analytics, and social media metrics. De-identified data on quitline services were obtained from the National Quitline Data Warehouse. Similar to the cessation intervention component, the promotional intervention component used the quitline intake MDS variables to describe caller characteristics and call volume trends associated with specific promotional efforts. For TCPs that promoted their quitline with a state Web site, the reach and use were measured by using the state's preferred Web analytics platform. If the state did not monitor Web traffic, assistance was provided by Google Analytics™ (a free Web analytic tool) on their site.
Google Analytics™ relies on a string of JavaScript code that is inserted into each Web page to track unique visitors on a Web site (Google Analytics). Aggregate traffic metrics for designated periods were then obtained from the platform. Metrics included the number of visits, number of unique visitors, average time spent on the site, number of pages viewed, and geographic region of visitors. For states that use social media to promote the quitline, activity on media platforms (e.g., YouTube™, Facebook™, Twitter™) was tracked by using free public programs, such as Facebook Insight™ (Facebook). These social media tracking programs allow groups or organisations to create a user-friendly dashboard of metrics, including the number of posts on a wall, number of comments, and number of likes, subscribers, fans, and friends for a particular time.
Other contextual factors, such as smoking prevalence for the state, level of smoke-free ordinance coverage (in terms of both local vs. state coverage, as well as types of public spaces covered), and cigarette taxes, were obtained by investigators in aggregate form from the appropriate government sources. Descriptive statistics were used to summarise the overall reach and use of each promotional activity. Bivariate statistics were used to describe callers to the quitline by type of self-reported referral source. In addition, call volume trends were described by promotional activity. A model-drive approach was used to identify significant promotional activity contributions to call volume by using multivariable models. Confounding, effect modification, and collinearity were considered where relevant.
Partnership Component 3: Qualitative Study
Study Questions
The Partnership Component was an effort to document the extent of collaboration among six NCCCP and NTCP states: Alabama, Arkansas, Delaware, Florida, Nebraska, and Vermont. This study questions included the following: (a) What is the level of integration between CCC and TCP programs? (b) In what ways do the CCC and TCP programs collaborate? (c) What are the key factors that facilitate (and hinder) collaboration? and (d) What additional opportunities for collaboration could be used?
Study Sample Size
Six states were selected on the basis of the following inclusion criteria: (a) ability to effectively carry out activities under the CCC Program Cooperative agreement; (b) existing relationships with the TCP program in their states; (c) ability to effectively carry out activities under the TCP cooperative agreement; (d) a history of conducting research; (e) capacity to designate an epidemiologist who can participate in these study activities; (f) existing innovative tobacco cessation activities; and (g) experience collecting data for the National Tobacco Clearinghouse or states with state-wide quitline registries. The states selected represented all geographic regions in the continental United States with the exception of the Pacific.
Data Collection Methods
To develop a thorough description of the NCCCP and NTCP programs and their collaborative efforts, data was incorporated into NVivo (Version 10.0) from a variety of sources. These include a document review, which included a systematic review of available organisational charts, Web sites, and state cancer plans. Finally, key stakeholder interviews (via site visits) were conducted by using a semistructured protocol with key health department, CCC, and TCP leadership staff in each of the six states to better understand aspects of each program's organisational structure, activities, and collaborative efforts around cancer and tobacco control.
Qualitative methods provide flexible in-depth exploration of the participants’ perceptions and experience, and the interviews yield descriptions in the participants’ own words. They allowed the interviewer flexibility to pursue relevant and important issues as they arose during the discussion. By using a grounded theory approach (Glaser, & Strauss, Reference Glaser and Strauss1967), the researcher becomes immersed in the data, thus allowing for openness to non-forced and non-preconceived discovery of emergent themes (Glaser, Reference Glaser2005) and generation of theories based on interpretive procedures (Haig, Reference Haig1996). The discussion guide included probes to ensure that input was obtained on specific items of interest, and open-ended questions ensured that participants’ responses and perceptions were fully captured. Site visits were conducted from March to July 2012. With the permission of the respondent, interviews were digitally recorded to supplement any information missed by the interviewer's notes. Individual responses were not linked to participants.
Analysis and Outcomes of Interest
Qualitative data collected from the key stakeholder interviews were organised and analysed by using NVivo (Version 10.0) software to facilitate the cross-referencing of qualitative data from multiple sources, coding by multiple researchers, and the development of findings for reports. A code list was developed on the basis of the prioritised research domains (i.e., infrastructure, priority of cancer and tobacco control activities, collaborative efforts, cross promotion) and applied to the qualitative data collected. Once codes were developed and all coders were in agreement on what each meant, additional steps were taken to ensure consistent coding and to enhance reliability, including pilot-testing of codes, double-coding, and training of project staff to reliably collect, enter, and analyse the data. To support triangulation, information obtained from each data source (interviews and observations, document review, secondary data sources) was used to verify findings and provide a more accurate description of each program.
Results and Discussion
Quitlines have gained prominence because they have provided evidence of their clinical efficacy, their effectiveness in real-world settings, and because of their potential to make cessation services more universally available (Telephone Quitlines: A Resource for Development, Implementation, and Evaluation). However, research is still needed on the effect of the promotion and use of quitlines on the prevalence of tobacco use in states that have them. Quitlines currently reach only 1% to 2% of the tobacco users in their states per year (Ossip-Klein, & McIntosh, Reference Ossip-Klein and McIntosh2003). An increase in utilisation rates may have a substantial population impact on decreasing smoking prevalence. The large spikes in call volume commonly experienced during promotional campaigns also indicate a large untapped demand for services (Telephone Quitlines: A Resource for Development, Implementation, and Evaluation). Consequently, as tobacco control programs try to increase their population impact, new ways to increase public awareness, the use of evidence-based services, and quit rates may all be needed. This study will provide evidence-based information needed by State, Tribal, and Territorial health departments regarding effective cessation modalities and promotional strategies. The findings will help public health agencies further develop and tailor their specific cancer and tobacco programs.
The overall goal for the investment in CER is to promote high-quality healthcare through broad availability of information that helps clinicians and clients match the best science to individual needs and preferences (Comparative Effectiveness Research Funding). However, one of the major challenges in research is ensuring that evidence-based interventions are disseminated to various key audiences, including, but not limited to decision-makers, practitioners, or the public. Therefore, methods for this study include the use of diffusion principles to guide dissemination of the key findings. A dissemination plan based on these principles will be developed to target key audiences such as staff from both the NCCCPs and the NTCP’s, key decision-makers, clinicians, as well as underserved populations.
The strengths of this study include the development of a mixed-methods design utilising both quantitative and qualitative techniques. The study examines multiple factors that are associated with cessation interventions, and builds upon existing data sources. However, some limitations exist. CER is limited by the intrinsic methodologies it utilises (What Is Comparative Effectiveness Research?, 2009). And as with many studies, self-reported bias or confounders may limit the reliability of data obtained.
Although there is an evidence base for the use of both traditional and innovative cessation interventions, evidence is lacking about which may be more effective. Results from this multi-component study may help to accelerate adoption of findings to all NCCCP and NTCPs. One of the NCCCP-funded program mandates is the development, implementation, and dissemination of a comprehensive cancer control (CCC) plan for each program. Findings from this study may be used to help update existing plans to reflect the adoption of study findings and usage of quitlines versus previous used methods as standard practice. The results of this study may also provide valuable information for clinicians and healthcare providers who counsel their patients on tobacco cessation. Since data was received from four state health departments from a diverse population across the U.S. region, information on cessation and promotional strategy interventions that is targeted to specific populations (patient and larger community) has potential in increasing tobacco cessation rates in the United States.
Acknowledgements
None.
Financial Support
This study was funded by the American Recovery and Reinvestment Act through the Office of the Secretary Award #: 200-2008-27958.
Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Use of trade names is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services.
Conflicts of Interest
None.
Ethical Standards
While this research did not involve human subject experimentation, all protocols were reviewed and approved by appropriate Institutional Review Boards.
