Background
Universal health care in Singapore is provided by a combination of government subsidies, compulsory individual healthcare savings accounts (Medisave), risk-pooling via both voluntary private and mandatory government health insurance plans (MediShield Life), out-of-pocket contributions from patients, and a government endowment fund that acts as a safety net for the needy (Medifund). Although Singapore has a successful healthcare system that is well known for its efficiency and quality, it is facing rising healthcare costs driven by a rapidly aging population, an increasing burden of chronic disease, and growing demands from citizens for expanded medical services (Reference Pearce, Lin, Teo, Ng and Khoo1).
To overcome such challenges, the Agency for Care Effectiveness (ACE) was established in August 2015 within the Ministry of Health (MOH) as the national health technology assessment (HTA) agency to initially focus on evaluating drugs for subsidy considerations. HTA is used as a tool to review health technologies and provide evidence of the value that these technologies can deliver to patients and their families, health system stakeholders, and to society more broadly. It is a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its life cycle, with the purpose of informing decision making in order to promote an equitable, efficient, and high-quality health system (Reference O'Rourke, Oortwijn and Schuller2).
In 2017, the ACE expanded its capacity to assessing other health technologies beyond drugs and supporting the MOH Medical Technology Advisory Committee (MTAC) to make evidence-based subsidy recommendations for medical technologies, including diagnostics, medical devices, and services, but excluding models of care, information technology system, and telemedicine (3). To increase transparency in decision making and facilitate implementation, the ACE publishes guidance on the rationale of subsidy recommendations, which include a summary of the comparative safety, effectiveness, and cost-effectiveness evidence along with the subsidy criteria for eligible populations.
Although established structures and processes exist in the Singapore's public healthcare system for implementing drug subsidy decisions, these are lacking for medical technologies, given that their adoption is highly dependent on the organizational feasibility of healthcare institutions. Adopting new medical technologies has a far-reaching impact beyond generating additional costs to the healthcare system, which may include changes in the organization of care and the modification of facilities, manpower resources, training, and credentialing requirements.
Establishing a robust and adaptive implementation framework for medical technology subsidy is important in Singapore. This ensures that funding decisions are translated into sustainable and effective clinical practice changes, leading to improvements in patient health outcomes. We aim to identify actionable steps to implement HTA recommendations for medical technologies in the Singapore setting.
Methods
For this review, the appropriate steps required in Singapore to operationalize HTA implementation beyond the subsidy decision itself were defined as the key themes of HTA implementation.
Search Strategy
We conducted a literature search adopting three complementary searches—electronic databases, a manual search of reference lists, and a review of specific HTA agency Web sites—to identify articles on implementation frameworks and related strategies.
EMBASE, PUBMED, and CRD databases were searched. The search strategy employed three concepts:
(1) Implementation OR Adoption OR Diffusion OR Dissemination OR Spread.
(2) Medical Technology OR Medical Device OR Health Technology OR Health Technology Assessment OR Health Innovation.
(3) 1 AND 2.
The search concepts were deliberately broad to comprehensively capture relevant articles because HTA implementation was poorly reported. The first concept captured the aspects of implementation, whereas the second concept captured the aspects of HTA of medical technology. The third concept was a combination of concepts one and two. The search period was from 2008 to 2020, pragmatically chosen to simultaneously cover more recent developments in implementation science and seminal HTA articles published in 2008, such as by Drummond et al. (Reference Drummond, Schwartz, Jönsson, Luce, Neumann and Siebert4). The electronic search was supplemented by a manual search of the reference lists of articles identified through electronic databases. There was no period restriction imposed on the manual search, as the intent was to identify widely cited landmark articles in implementation science that may have preceded 2008. Articles were restricted to the English language and included primary research, reviews, and grey literature such as conference posters and abstracts. Articles were selected if they described an implementation framework, process model, or theory, including case studies of prior implementation efforts. Our evidence hierarchy prioritized articles that contributed to implementation frameworks with direct relevance to our research question. Articles focused on the implementation of drugs, e-health interventions, digital health services, clinical practice guidelines, or patient safety guidelines were excluded because they did not meet the local definition of “medical technology”, and HTA implementation for medical technologies was more challenging in Singapore (3;Reference Drummond, Schwartz, Jönsson, Luce, Neumann and Siebert4).
We also reviewed the online resources of specific HTA agencies including the Canadian Agency for Drugs and Technology in Health (CADTH), Alberta Health, U.K. National Institute for Health and Care Excellence (NICE), Australian Medical Services Advisory Committee (MSAC), U.S. Agency for Healthcare Research and Quality (AHRQ), and the International Network of Agencies for Health Technology Assessments (INAHTA). The INAHTA Secretariat was contacted in November 2020, seeking responses from member agencies for existing policies, procedures, or frameworks that support the implementation of HTA-informed subsidy decisions for medical technologies.
Data Abstraction and Analysis
Two independent reviewers evaluated full-text articles to minimize bias. Each article was reviewed with the aim of eliciting whether an implementation model was proposed and what themes were proposed in those models. The specific themes from each model were synthesized. A thematic analysis was conducted using the Braun and Clark (Reference Braun and Clarke5) framework as a guide to identify key themes using a coding frame (Supplementary File 1).
Thematic Analysis
An inductive method was applied to the evidence synthesis and a semantic approach was used to code the emerged themes. We used open coding, with numeric codes assigned to describe the content in sentences extracted from the articles. We then collated the assigned codes into broader themes, which were refined through iterative discussions to remove overlaps and contradictions across the different models that were analyzed. The final set of themes, including names and definitions, was shared with all coauthors and was accepted to be coherent and distinctive.
We counted how often themes appeared in each article, and the top ten themes were identified. We assumed that a high occurrence, especially in identified landmark articles, corresponded to high internal validity. Unique counts were assigned to the same theme, which may have occurred more than once in an article. Landmark articles on implementation frameworks were internally defined as frameworks that were widely cited and applied in qualitative or quantitative studies in our evidence base. We were cognizant that, although frequency of occurrence was chosen to present the data, it might not necessarily imply significance in answering our research question. For the purposes of internal validity, a matrix analysis mapped the top ten themes occurring in the implementation frameworks that were widely cited in our evidence base.
Results
The literature searches were initially conducted on 15 October 2019 and were updated on 20 December 2020. Our searches yielded 9,854 hits that comprised 9,828 retrievals from electronic databases and 22 citation reviews and 4 articles from HTA agencies. After removing duplicates and screening titles and abstracts, 121 articles remained. Of these, seventy-seven were excluded. Of the remaining forty-four articles, eighteen articles were retrieved from the databases, twenty-two were retrieved from citation reviews, and four were found on HTA agency Web sites. A flow diagram of our search results is shown in Figure 1.
Figure 1. Flow diagram of search results.
INAHTA member agencies who responded to the survey were NICE UK, CADTH Canada, CDE Taiwan, SBU Sweden, CONITEC Brazil, and NIPH Norway. They do not have formal frameworks in place to support the implementation of subsidy decisions for medical technologies.
Prominent Implementation Frameworks
There were eleven prominent implementation frameworks widely cited in the retrieved literature. The top ten themes that emerged from the evidence synthesis were mapped back to the implementation frameworks and are reported in Table 1.
Table 1. Occurrence of the top ten themes in prominent implementation frameworks
CICI, Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions; CFIR, Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science; PRISM, A practical, robust implementation and sustainability model for integrating research findings into practice; PARiSH, Enabling the implementation of evidence-based practice: a conceptual framework; Re-AIM, Re-AIM resources; ISF, Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation; QIF, The quality implementation framework: a synthesis of critical steps in the implementation process; NPT, Normalization process theory: a framework for developing, evaluating, and implementing complex interventions; R = MC2: A practical implementation science heuristic for organizational readiness; INHATA guidance document: HTA agencies and decision makers; The Alberta framework: Maximizing the impact of health technology assessment.
The most often cited framework was CFIR (Reference Damschroder, Aron, Keith, Kirsh, Alexander and Lowery7) with 4,097 citations, but it only covered seven themes and missed the “Information Dissemination” domain. The most often cited theme was “conducting an organizational needs assessment”. The framework that included eight out of ten themes and covered all domains was the QIF (Reference Meyers, Durlak and Wandersman12), but it did not include “conducting a pre-implementation context analysis of various domains”, and “employing diverse and intelligent strategies to disseminate information, for example, knowledge translation tools”.
Helpful Findings for Developing Singapore's Specific Implementation Framework
The top ten recurrent themes and examples of the coding frame are shown in Table 2. Supplementary File 2 captures the five domains and respective themes and details their descriptions.
Table 2. Top ten themes from the literature review and examples from the coding frame
The top ten themes holistically covered key concepts in the prominent implementation frameworks and were considered the most relevant findings for developing a specific implementation framework for Singapore. These were shared with all coauthors and were further synthesized into five distinct domains. Broad acceptance was obtained that these themes were representative of good practice standards for HTA implementation.
Five-Domain HTA Implementation Framework
A five-domain model for HTA implementation was developed:
(1) Implementation Strategy,
(2) Organizational Support,
(3) Stakeholder Engagement,
(4) Information Dissemination, and
(5) Implementation Outcomes and Evaluation.
Presented in sequential order as shown in Figure 2, this five-domain framework provides operational guidance of the action items across the implementation spectrum from planning and execution to impact monitoring. Domains represented in the figure are relevant to implementing HTA for medical technologies in Singapore. Refer to Supplementary File 2 for complete descriptions of each domain.
Figure 2. The five-domain model HTA implementation framework provides operational guidance in a sequential order for the implementation of HTA-informed subsidy decisions for medical technologies.
Implementation Strategy
A prevalent assumption of an implementation strategy was “clinical serendipity”—a natural and smooth adoption of new innovations or interventions by clinicians (Reference Parston, McQueen, Patel, Keown, Fontana and Al Kuwari24). In contrast, successful implementation required various, if not all, parts of the implementation strategy (e.g., dissemination, training, and support) to be carefully designed (Reference Damschroder, Aron, Keith, Kirsh, Alexander and Lowery7). This included outlining specific tasks and timelines (Reference Damschroder, Aron, Keith, Kirsh, Alexander and Lowery7;Reference Meyers, Durlak and Wandersman12) and ensuring that the implementation strategy was adaptable to address unique requirements in any particular context (Reference Pfadenhauer, Mozygemba, Gerhardus, Hofmann, Booth and Lysdahl25).
Organizational Support
Organizational readiness assessments were referred to explicitly, with specific tools created to assess readiness (Reference Scaccia, Cook, Lamont, Wandersman, Castellow and Katz14) based on organizational factors such as communication channels (Reference Damschroder, Aron, Keith, Kirsh, Alexander and Lowery7), leadership (Reference Greenhalgh, Robert, Macfarlane, Bate and Kyriakidou26), culture and morale (Reference Feldstein and Glasgow8), and resource allocations (Reference Greenhalgh, Robert, Macfarlane, Bate and Kyriakidou26); and implicitly by inferring the need to develop an understanding of the local operating environment or context (Reference Pfadenhauer, Gerhardus, Mozygemba, Lysdahl, Booth and Hofmann6;Reference Damschroder, Aron, Keith, Kirsh, Alexander and Lowery7;Reference Murray, Treweek, Pope, MacFarlane, Ballini and Dowrick13). Different medical technologies require different aspects of organizational readiness to be surveyed, taking context into account.
Stakeholder Engagement
The theme of stakeholder engagement was widely reported. Implementation efforts should be consultative, allowing for ample stakeholder participation. Early engagement of stakeholders helped address and manage any concerns upstream (Reference Drummond, Schwartz, Jönsson, Luce, Neumann and Siebert4) and by doing so, ensured support for the implementation strategy (Reference Meyers, Durlak and Wandersman12;Reference Murray, Treweek, Pope, MacFarlane, Ballini and Dowrick13). Stakeholders representative of a range of different backgrounds catered for adaptability in the implementation strategy (Reference Anton and Jones27;Reference Llewellyn, Procter, Harvey, Maniatopoulos and Boyd28).
Information Dissemination
Dissemination was defined as an active process of knowledge transfer, one that was vertical, planned, formal, and centralized (Reference Glasgow, Vinson, Chambers, Khoury, Kaplan and Hunter29). The more extensive and multipronged the mode of information dissemination was, the greater the success in implementation (Reference Dearing20;Reference Dobbins, Ciliska, Cockerill, Barnsley and DiCenso30). Specific to HTA, the INAHTA's guidance document highlighted that mail-outs and journal presentations might be ineffective as standalone modes of communication, and instead, recommended a mixed approach. This involved face-to-face consultations, in tandem with the distribution of written materials (Reference Hailey, Babidge and Cameron15). Having a localized unit (Reference Meyers, Durlak and Wandersman12;Reference Braithwaite, Marks and Taylor19) within the organization to orchestrate implementation efforts was shown to improve the success rate of implementation. Such a unit introduced structure and clarity of roles, key deliverables, and timelines for seamless information flow (Reference Meyers, Durlak and Wandersman12). It was suggested that these units possessed the necessary technical, communication, financial, and project management skills required to carry out the implementation effort (Reference Parston, McQueen, Patel, Keown, Fontana and Al Kuwari24).
Implementation Outcomes and Evaluation
Evaluation of implementation outcomes was explicitly described as evaluating fidelity, adoption, appropriateness, costs, feasibility, penetration, and sustainability (Reference Proctor, Silmere, Raghavan, Hovmand, Aarons and Bunger31) and was implicitly described as incorporating a component for audit (Reference Kitson, Harvey and McCormack9) or feedback (Reference Greenhalgh, Robert, Macfarlane, Bate and Kyriakidou26) in any implementation strategy. Regardless of the method chosen, ensuring that some form of feedback mechanism was hardwired into the evaluation enhanced the robustness of an implementation strategy. Evaluation outcomes created in partnership (22) with stakeholders ensured that the outcomes were relevant and specific to their context, instilling greater ownership among stakeholders.
Discussion
Based on our literature review, we identified key themes for implementation and conceptualized them into an HTA implementation framework with domains most relevant to Singapore. The framework is a visual representation of the key themes that could be considered in sequence, when implementing HTA recommendations of medical technologies.
In Singapore, the provision of services involving medical technologies is highly influenced by institutional practices and user preferences at healthcare institutions. In recognition of the fact that a “one-size-fits-all”, “top-down” implementation strategy is not adaptable nor practical, we attempted to cater implementation efforts to unique operating environments. As the national HTA agency, the ACE developed an Organizational Readiness Assessment (ORA) tool (Supplementary File 3) to highlight backend processes at the healthcare institutions that might be affected when a medical technology is recommended for subsidy. This self-assessment allows hospital institutes to identify potential implementation barriers early and make changes before subsidy comes into effect.
Clinical stakeholders such as doctors often assumed the role of clinical champions to identify and solve implementation issues unique to their institutions. However, a proper implementation team at the healthcare institutional level, comprising people with the necessary skillsets and authority to drive multifaceted changes, should be established to solve organizational problems. Establishing such teams will complement existing multipronged modes of information dissemination that the ACE employs, including face-to-face consultations, frequent email exchanges, issuing formal circulars, and publishing guidance documents on our Web site. The ACE has yet to explore mobile applications as a mode of information dissemination but acknowledges that it is worth exploring to cater to increasing demand from information-seeking behaviors.
Implementation success in Singapore is currently measured by the timeliness of subsidy implementation. Such a measure is not reflective of the true benefits to patients and the healthcare system. As such, implementation outcomes should be codeveloped with relevant stakeholders to ensure the collection of meaningful data that measures changes in clinical practice and improvements in patient outcomes in a sustainable manner. We acknowledge that the key themes of the five-domain model were considered most relevant to HTA implementation in the Singapore setting. Other HTA agencies may consider its local relevance and applicability. The ten key themes of implementation need to be regularly reviewed to keep pace with the evolving needs of the healthcare landscape and be adapted to address any potential barriers to framework compliance.
Conclusion
Policy decision making in Singapore is increasingly reliant on HTA when assessing the value of medical technologies. A robust and adaptable implementation of HTA-informed subsidy decisions for medical technologies is crucial to optimize patient outcomes commensurate with costs. Although there is a plethora of implementation science literature to guide the implementation of novel or complex health interventions, much of the existing literature is about conceptual developments. The top ten themes were identified to emphasize the important aspects of organizational feasibility, which should be considered when implementing HTA recommendations for medical technologies in the Singapore setting. We will continue reviewing the applicability of key themes when implementing HTA-informed subsidy decisions as we gain more experience in the future.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321000222.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and materials
The dataset(s) supporting the conclusions of this article are included within the article and its Supplementary files.
Acknowledgments
We would like to acknowledge SC for her contribution to the data collection and analysis process during her time of employment in the ACE.
Funding
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Authors Contributions
KN, PA, and CF conceived and designed the study. VS collected, analyzed, and interpreted the data. SC collected, analyzed, and interpreted the data during her time of employment in the ACE. VS and PA were instrumental in the development of the HTA framework with input from KN and CF. VS led the drafting of the manuscript, which was reviewed and edited by PA, CF, and KN. All authors read and approved the final manuscript.
Conflicts of interests
There are no conflicts of interest.
Open access
