Crossref Citations
This article has been cited by the following publications. This list is generated based on data provided by Crossref.
Younas, Ayesha
Khan, Rehan Ahmed
and
Yasmin, Raheela
2021.
Entrustment in physician-patient communication: a modified Delphi study using the EPA approach.
BMC Medical Education,
Vol. 21,
Issue. 1,
Campeau, Anthony
Huiskamp, Maud
Sykes, Nicole
Kriening, Susan
Bourn, Scott
and
VanAarsen, Kritine
2021.
An Inter-Agency Expert Panel's Prioritisation of Clinical Quality Improvement Topics in a Paramedic System.
Australasian Journal of Paramedicine,
Vol. 18,
Issue. ,
p.
1.
van Zuuren, Esther J
Logullo, Patricia
Price, Amy
Fedorowicz, Zbys
Hughes, Ellen L
and
Gattrell, William T
2022.
Existing guidance on reporting of consensus methodology: a systematic review to inform ACCORD guideline development.
BMJ Open,
Vol. 12,
Issue. 9,
p.
e065154.
Souza-Silva, Maira Viana Rego
Domingues, Mara Luiza de Paiva
Chagas, Victor Schulthais
Pereira, Daniella Nunes
de Sá, Laura Caetano
Almeida, Mychelle Stefany Santos
Sales, Thaís Lorenna Souza
Raposo, Magda César
Guimarães, Nathalia Sernizon
Oliveira, João Antônio de Queiroz
Ribeiro, Antonio Luiz Pinho
Cardoso, Clareci Silva
Martins, Maria Auxiliadora Parreiras
Enes, Thais Bueno
Soares, Thiago Barbabela de Castro
Baldoni, André Oliveira
and
Marcolino, Milena Soriano
2022.
Implementation of a text messaging intervention to patients on warfarin therapy in Brazilian primary care units: a quasi-experimental study.
BMC Primary Care,
Vol. 23,
Issue. 1,
Luo, Christine T.
Bailey, Jessica A.
Yarris, Lalena M.
Kornegay, Joshua G.
Regner, Kimberly A.
and
Mayersak, Ryanne J.
2023.
Top emergency medicine faculty development papers since 2000: A Delphi study.
AEM Education and Training,
Vol. 7,
Issue. 2,
Sadati, Leila
Motaharipour, Morteza
Younas, Ayesha
Farajidana, Hoorvash
Hosseini, Fakhrolsadat
and
Abjar, Rana
2023.
Development of Professional Ethics Curriculum in the Operating Room for the Current Era of Surgery: A Mixed Method Study.
Canadian Journal of Bioethics,
Vol. 6,
Issue. 3-4,
p.
57.
What do the RAND corporation, top literature in emergency medicine, and bougie-assisted cricothyroidotomy have in common? The Delphi method.
INTRODUCTION
In this issue of CJEM, we find two education papers that use the Delphi method, or at least a modification of such, to determine the outcome of their study. The Delphi is the most commonly applied consensus group method, or systematic means to measure and develop a consensus.
The modified Delphi study has become a more frequently used investigatory tool in the last several years, especially in medical education. This method can be especially useful when examining questions requiring expertise, and therefore might be applied to a variety of emergency medicine (EM) questions and problems for which uncertain empirical evidences exist.
WHAT IS THE DELPHI METHOD?
The Delphi method was originally developed to harness the wisdom of a group of experts in a field to determine an anticipated outcome. Usually, it requires an anonymous method to iteratively refine the answer to a question. Key elements of a Delphi method include the assembly of a large and diverse group of experts who anonymously provide their opinions, a mechanism for a structured data collection and feedback, and a facilitator who collects the data and reports them back to the experts for reconsideration and recalibration. In its original implementation by the RAND Air Force corporation of America in the 1950s, this was done via paper-based questionnaires and surveys, but today most of these techniques use some form of an electronic method of data collection and reporting.
The Delphi technique was developed by the RAND Corporation to effectively gather experts’ opinions to predict the probability of enemy attacks in wartime.Reference Jones and Hunter1 Since then, the Delphi has been used extensively in multiple fields of study, including medicine, nursing, business, public policy, science and technology, and medical education. In medical education, where decisions must often be made with limited information, the consensus methods, including the Delphi, are used extensively for the purposes of developing or renewing curricula, developing assessment tool items, defining competencies, and more recently to determine entrustable professional activities (EPAs).Reference Humphrey-Murto, Varpio and Wood2 In this issue of the journal, the modified Delphi has been used to derive essential steps for a novel technical skill (bougie-assisted cricothyroidotomy) and for educational resource development (top EM papers to teach junior trainees).Reference Dharamsi, Gray and Hicks3, Reference Bazak, Sherbino, Upadhye and Chan4 These studies provide excellent examples of how the Delphi can be used to collect expert judgements to arrive at a decision that is arguably considered more credible and defensible than a singular opinion.
STRENGTHS AND WEAKNESSES OF THE DELPHI METHOD
Recent studies have demonstrated the variability in how the Delphi method has been applied. Both studies in this issue of CJEM use a modified Delphi, yet there is no clear consensus on the true definition of the “modified Delphi.” This lack of clear definition has led to considerable confusion and substantial variation in the quality of reporting of Delphi studies. For example, a recent review demonstrated that one-third of medical education Delphi studies failed to report that a literature review on the topic of interest had been conducted, and over half failed to report key aspects such as what background information was provided to participants; the response rates for each round; what formal feedback of group rating was shared between rounds; a statement that anonymity was maintained; and a clear definition of consensus.Reference Humphrey-Murto, Varpio and Wood2
One of the drawbacks of the Delphi method is that the underlying assumption that decision-making in groups leads to wider perspectives and, ultimately, better decisions has not been definitely proven. Critics have suggested that consensus methods generate nothing more than collective ignorance or simply reflect a formalized form of peer pressure.Reference Jones and Hunter1 In the event of multiple rounds, attrition in participation can lead to a false consensus, because those with dissenting views drop out.
Despite these pitfalls, consensus methods have many potential benefits when rigorously conducted and reported. Table 1 lists potential strengths and weaknesses of the Delphi.
Table 1. Potential strengths and weaknesses of the Delphi method
CAN WE DO BETTER?
Although modified Delphi methods have their benefits, inconsistent definitions and poor reporting can lead to a lack of scientific credibility. Lack of clarity in the reporting of procedures and methodological choices associated with modified Delphi studies can prevent readers from effectively appraising and interpreting findings. In response to this, many “how to” guidelines have been published.Reference Sinha, Smyth and Williamson6–Reference Diamond, Grant and Feldman8 For a quick reference, we have adapted a list of best practices in Box 1 of this paper.
Box 1. Best practices for conducting a rigorous Delphi study
1) Define the purpose or objective of the study.
2) Outline each step of the process: If modifications were made, provide a rationale for your choices.
3) Describe the selection and preparation of the scientific evidence for the participants.
4) Describe how items were selected for inclusion in the initial questionnaire: Describe the process in sufficient detail.
5) Describe how the participants were selected and their qualifications. Include a description of the facilitator's credentials.
6) Describe the number of rounds planned and/or criteria for terminating the process.
7) Clearly describe how consensus was defined.
8) Report response rates and results after each round.
9) Describe the type of feedback provided after each round.
10) Describe how anonymity was maintained.
11) Address potential methodological issues (e.g., lack of consensus) or limitations in the discussion (e.g., low response rate).
Adapted from Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach 2017;39(1):14–9.
In addition to rigorously designing, conducting, and reporting Delphi studies, there is a need for further research to understand best practices for defining consensus, identifying participants, exploring the nature of consensus building, determining outcomes, and how to select a consensus method.Reference Humphrey-Murto, Varpio and Wood2
Medical educators have identified Delphi methods as a useful way to approach educational problems that require synthesizing expert opinion to make decisions. The work of the Dharamsi and Bazak teams is a helpful example of different applications of this method and, although generally provides rigorous reporting, could be improved upon. For example, some readers may have desired more information regarding the expert participants in the Bazak study,Reference Bazak, Sherbino, Upadhye and Chan4 and the inclusion of otolaryngology or trauma surgeon participants in the Dharamsi article might have provided a more holistic understanding of key steps in this procedure.Reference Dharamsi, Gray and Hicks3
EM researchers and educators who wish to apply this method to consensus decision-making are encouraged to use a systematic approach to study design and implementation, with careful attention to and reporting of published recommendations to ensure methodological rigor. Although word count limitations may provide a barrier to detailed descriptions of the methods, authors can reference method papers in the citations, or submit more detailed methods for consideration for publications as supplemental material. Methodological rigor and transparent reporting are essential to assure readers that the consensus results are applicable to their environment, and to translate expert opinion into practice.
Competing interests
None declared.