Following the coronavirus disease (COVID-19) pandemic, in-person attendance by health care professionals has risen for many conferences. Meeting and event numbers reached nearly 90% of pre-pandemic levels at the end of 2022. Reference Zgraggen1 According to a 2023 survey of 275 health care providers, the most valuable outcome of attending medical congresses is the opportunity to learn and keep up to date with new developments and the current thinking. 2
However, knowledge gained or presented during conferences is often poorly documented. Conference proceedings, even if published, typically include only the content of the papers and abstracts. Important conversations, conclusions, and issues that are discussed are usually not documented. For instance, groups may discuss and even gain consensus on issues without recording this consensus. Even conferences that include consensus generating techniques—such as focus groups—frequently limit these activities to key conference speakers and organizers without involving conference attendees, in general.
The goal of this study was to pilot the use of the Delphi methodology as a way to document issues discussed during a national conference and translate these discussions into actionable information. The Delphi method is a common and well documented method for achieving consensus when consensus is difficult to obtain. Reference Sinead, Felicity and Hugh3 The method is an iterative survey technique, where a group of participants (commonly called experts) is asked to rate statements on a specific scale. Statements are deemed to have reached consensus when experts’ ratings converge toward a common value.
The specific objective of this study was to assess the difference in rate of statement consensus in a group of professionals attending an in-person conference in comparison to the rate of statement consensus among professionals with similar backgrounds who did not attend the conference.
The null hypothesis of no difference between the proportion of statements reaching consensus was tested against the alternative hypothesis of a significant difference.
Methods
Study Design
The overall study design followed the Delphi methodology. Invited speakers from the conference were asked to provide 3 to 5 statements to be used in the first round of the Delphi study. Speakers’ statements were to include content specifically addressed in their own educational sessions. In total, 28 statements were obtained. Examples of statements included:
Patient monitoring devices allowing continuous monitoring of vital signs during mass casualty incidence responses can enhance lifesaving interventions.
Potential biases in data sets used to train artificial intelligence algorithms generate serious ethical concerns.
Prehospital mass casualty incident response training should be delivered to all first responders.
Statements that did not reach consensus were advanced to the next round.
The results of the Delphi ratings were compared between experts who attended a national conference and those who did not.
Study Setting
The in-person conference was the Society of Disaster Medicine and Public Health Preparedness Ten Year Anniversary Conference and Annual Meeting, which took place in Washington DC on December 4–6, 2023. The conference focus was the effects of disasters on vulnerable populations, including climate change and disaster response; conflict medicine and population health; artificial intelligence; decision aids and equipment, and their effect on mass casualty incident response; disaster medicine education, training, and competencies that are culturally and geographically relevant; and mental health. The conference had 15 speakers and 129 attendees.
Participants
All participants of the in-person conference were invited to partake in the Delphi study as experts. These participants formed the “attending” group. All members of the Society of Disaster Medicine and Health Preparedness who did not attend the conference were invited to join the “nonattending” expert group. Participants were contacted through email and enrolled into the study prior to the in-person conference. All participants completed an online informed consent process.
Outcome Measures
The primary outcome measure was the number of statements reaching consensus in the Delphi study. Consensus was defined as a standard deviation of less than or equal to 1.0 on a 7-point rating scale. Reference Franc, Hung, Pirisi and Weinstein4
Data Collection
Statements collected from the invited speakers were entered into the online Delphi method software STAT59 (STAT59 Services Ltd, Edmonton, Alberta, Canada). The experts were contacted by email to enroll in the STAT59 platform to answer the survey rounds. In the first round, experts were asked several demographic questions, including current profession, location of work, primary employment, and years of expertise in disaster medicine. Experts were then asked to rate each of the 75 statements on a 7-point linear scale for importance, where 1 represented “Strongly disagree” and 7 represented “Strongly agree.”
Data Analysis
Data analysis was performed using STAT59 and Python, Version 3.6.9 (Python Software Foundation, Wilmington, Delaware, USA). Reference Van Rossum and Drake5
Sample Size
No formal sample size calculation was performed. All participants attending the conference and all members of the national society were invited.
Results
The final study group included 27 experts who attended the conference and 10 who did not. Most participants (25/37) were from North America, whereas others were from Europe and Central Asia, East Asia and Pacific, Middle East and North Africa, and sub-Saharan Africa. The most indicated professions included “education and training” (71%), “physician” (43%), “research” (40%), “response/field operation” (20%), and “administration and support” (11%). Employment included “university or research center” (49%), “governmental organization” (23%), “private sector” (20%), and “non-governmental organization” (9%). Among the participants, 40% had greater than 15 years of experience in disaster medicine, 31% had 11 to 15 years of experience, 17% had 6 to 10 years, and 11% had 5 years or less.
The first round of the Delphi study contained 6 demographic statements and 28 rating statements. The second round contained 26 statements. The third round contained 11 statements. In total, each expert rated 75 statements.
In total, there were 2634 ratings on the 7-point linear scale. The overall mean rating on the 7-point scale was 5.4, with a standard deviation of 1.4.
Among the participants who attended the conference, the mean overall rating of the 75 statements on the 7-point scale was 5.4 with a standard deviation of 1.13 (Figure 1). Among the attending participants, 26 statements reached consensus. Among the participants who did not attend the conference, the mean overall rating of the 75 statements was 5.3, with a standard deviation of 1.17 (Figure 2). In this group, 27 statements reached consensus.
Figure 1. Attending experts standard deviation from 75 statements.
Figure 2. Nonattending experts standard deviation from 75 statements.
Using the proportion test, no significant difference was found between the proportion of statements reaching consensus among the attending participants (26/75) when compared to the proportion of 27/75 among the nonattending participants (Z-stat = −0.14; P = 0.89).
Which statements reached consensus differed between the attending and nonattending participants. Eleven statements reached consensus in the attending group only. Ten statements reached consensus in the nonattending group only. Sixteen statements reached consensus in both groups.
At the completion of the study and after analyzing the results in aggregate by combining the attending and nonattending groups and calculating the standard deviation for each of the statements, 35 statements reached consensus: 12 in round one, 15 in round two, and 8 in round three. The final list of statements reaching and not reaching consensus will be published separately.
Discussion
This study demonstrated no significant difference in the number of statements reaching consensus in the Delphi study when comparing experts who had attended the national conference versus those who did not. While the difference in standard deviation between the attending and nonattending group was small, Figure 1 and Figure 2 suggest that there was less dispersion in the SD among the attending group. Nonetheless, which statements reached consensus differed considerably between the 2 groups.
At present, there is no published universal best practice for achieving consensus. Waggoner et al. reviewed common methods of consensus gathering in 2016 and included nominal group process, consensus development panels, and the Delphi method. Reference Waggoner, Carline and Durning6 Although they failed to find documented best practices, the manuscript ended with the following statement: “We hope that future studies will be conducted using more stringent standards, especially as it pertains to reporting of design, methods, and results.” Reference Waggoner, Carline and Durning6
This pilot demonstrates that the Delphi method can be used as a more rigorous method to document conclusions from an in-person conference. The Delphi method offers several advantages over more informal methods of conference proceedings. First, the Delphi method can be used with any size of expert group, whereas focus groups and expert panels often become difficult to manage when more than 10–15 participants are involved. Second, the Delphi method can be used to combine the opinions of those present at the conference with those who could not attend, leading to a more inclusive and diverse expert panel. Third, while both focus groups and Delphi studies aim to provide a set of actionable statements, only the Delphi method provides an objective measure of the degree of consensus and also prioritizes these statements. Finally, the Delphi method is also an excellent way to document statements that are considered but do not reach consensus; these statements can be invaluable for anticipating future trends and highlighting areas for further investigation.
While this pilot study demonstrates that the Delphi method can be implemented as a robust approach to documenting conference proceedings and indicates that consensus is similar in both attendees and nonattendees, further rigorous studies of the Delphi method in conference proceedings are indicated.
Strength and Limitations
To our knowledge, this is the first published study to pilot the use of the Delphi method to document conference proceedings and compare attending versus nonattending participants.
Limitations of this study include the small sample size of only 35 experts and application of the method to only a single conference. However, the size of the expert group was large by the standard for consensus which typically suggests panels of 6 to 11 members. Reference Waggoner, Carline and Durning6
Furthermore, while the Delphi method is commonly employed to gain consensus, the statistical analysis of these studies is controversial. In this study, we chose to use the 7-point linear rating scale for agreement, and to define consensus as a standard deviation of less than or equal to 1 as this has been previously shown to be a robust and reliable method to analyze Delphi survey data. Reference Franc, Hung, Pirisi and Weinstein4 A different rating scale or different definition of consensus may have led to different results.
Conclusion
The Delphi method represents a viable and implementable technique to prioritize and document consensus for conference proceedings. The technique allows conference attendees and nonattendees to assess statements developed at the conference. The proportion of statements obtaining consensus after the first round appears similar between attendees and nonattendees. However, which statements reached consensus were markedly different between the 2 groups. Advantages over traditional conference proceedings include the ability to involve larger groups of experts, statistical measurement of the degree of consensus, and prioritization of the results.
Acknowledgments
The authors would like to thank Sandra Franchuk for her editorial assistance.
Author contributions
All authors contributed to the overall study inception and design. Data analysis was performed by JMF. JMF wrote the initial manuscript draft. All authors have reviewed and approved the final manuscript.
Funding statement
No financial support.
Competing interests
JMF is the CEO and founder of STAT59. All other authors declare that they have no competing interests.
Ethical standards
The study was reviewed by the University of South Florida Institutional Review Board and deemed exempt from a formal ethics review (STUDY006415).
Open access
