Strategy one: cultivate digital mentors

Traditionally, learners have cultivated mentors from their local medical school and residency program, their elective rotations, and possibly conferences and other networking opportunities. With the large growth in online education within our specialty of EM, there is vast potential for digital mentorship in the FOAM community. Digital mentors are online accounts that learners can follow that serve several mentorship roles. First, they help steer learners toward new resources and other community members to follow. Second, by observing and interacting with these mentors, learners can benefit from role modelling, such as how to contribute to the online community, access FOAM material sustainably, and integrate this learning into practise.

Mentors may range from local to international in scope, from single author accounts to FOAM sites with teams of editors, and from general EM topics to specialty niches in EM. Each learner can personalize their choices for digital mentors to meet their learning goals. To start, learners can seek out websites and social media accounts representing their local department, residency program, or regional and national medical bodies. Next, learners can follow notable blogs, podcasts, and social media accounts for individual medical educators whom they consider to be mentors. Lastly, learners can identify FOAM resources of interest, whether in general or a specialty area of EM, and follow the social media accounts of these sites and their editors.

While the online EM community crosses multiple platforms and websites, attempting to use a variety of these can be overwhelming. Twitter is arguably the current “virtual town hall” of the FOAM community where novices and mentors interact and share ideas, making it an ideal place to start seeking out digital mentors. A recent network analysis showed how certain participants on Twitter could help to connect users to new ideas or people.Reference Riddell, Brown, Kovic and Jauregui ¹² Such analyses may hold value for those who are seeking mentors. By observing and interacting with these accounts on social media platforms such as Twitter, learners benefit from the mentorship roles mentioned above.

Strategy two: browse the most popular FOAM websites

In virtual communities of practice, there is substantial sharing of information. As such, finding the highest yielding content can be difficult. General Web searches (e.g., Google, Bing, or Yahoo) are optimized for the lay population, but not for practising emergency physicians and trainees. While using custom search engines (e.g., FOAMsearch.net, also known as GoogleFOAM),Reference Raine, Thoma, Chan and Lin ¹³ RSS feeds and Twitter hashtags may help to filter resources for learners as mentioned in previous publicationsReference Thoma, Joshi, Trueger, Chan and Lin ⁵ ; it can still be daunting to decide what to read.

As emergency physicians are well networked,Reference Kovic, Lulic and Brumini ¹⁴ their followership of certain online resources may suggest usefulness. Novices can start by looking at the most popular content among the entire community of FOAM as ranked by the Social Media Index (SMi) that can be found at the following link: https://www.aliem.com/social-media-index/.Reference Thoma, Sanders and Lin ¹¹ The SMi is built upon the premise that FOAM content is directed at the specific community of practising emergency physicians. Broad followership lends some credibility as the websites included on the SMi are of minimal interest to the general public but of significant interest to this community, and followership correlates with other measures of impact and value to the specialty (such as journal impact factor and pageviews of a website).Reference Thoma, Sanders and Lin ¹¹ The SMi is somewhat analogous to an impact factor for journals and h-index for scholars and is the only published tool for such a measure to date that was developed specifically for online resources. This comparative index is derived from three indicators: Alexa rank of the website, Twitter followers of the most prominent editor, and Facebook likes.Reference Thoma, Sanders and Lin ¹¹ The SMi is published on the ALiEM website and updated quarterly to capture the most popular and current FOAM sources (Figure 1).Reference Thoma, Sanders and Lin ¹¹

Figure 1 Social Media Index ranking for November, 2016

The SMi is controversial within the FOAM community. Criticisms of the SMi include: 1) it can be gamed through the purchase of Facebook and Twitter followers; 2) the Alexa Score, which makes up a significant portion of the ratings, is proprietary, US-centric, and opaque; and 3) the constituent elements of SMi measure popularity, not quality per se.Reference Thoma ¹⁵ These criticisms are fairly similar to those of impact metrics for medical journals. However, SMi is one of the few blog metrics that has evidence demonstrating that it is correlated with other measures of impact if applied to medical journals.Reference Thoma, Sanders and Lin ¹¹ While the relationship of the SMi with quality is being studied, the SMi is the most highly cited metric in the field and can at the very least direct learners to impactful, prominent, and generally respected resources within the EM community.

Strategy three: use existing critical appraisal tools developed for FOAM

Since the start of the evidence-based movement in the late twentieth century, critical appraisal of scientific literature has become better integrated into medical school and residency curricula. ¹⁶ Scientific articles may be appraised using tools such as the JAMA Users’ Guide to the Medical Literature Reference Guyatt, Drummond, Meade and Cook ¹⁷ ; however, secondary resources such as lectures and textbooks are seldom held to such standards. Blogs and podcasts follow in the tradition of secondary educational resources but are met with disproportionate skepticism, likely because of their novel and disruptive nature.Reference Thoma, Chan, Benitez and Lin ¹⁸ In addition, by the nature of being open access, the wide availability of the internet, and with the ease of online publication, there are inevitably many FOAM articles and resources that are purely opinion based and lacking in quality evidence. Therefore, as the reach of these resources continues to expand rapidly, new tools and additional training are needed to assist with their critical evaluation, as gestalt ratings have proven unreliable.Reference Krishnan, Thoma, Trueger, Lin and Chan ^{19 ,} Reference Thoma, Sebok-Syer and Krishnan ²⁰

Tools including the DISCERN scoreReference Rees, Ford and Sheard ²¹ and Health On the Net (HON) Code of ConductReference Boyer, Selby and Appel ²² have been developed to appraise online resources for patient use. Along similar lines, a number of recent studies have developed and examined quality indicators for online medical education resources. Three resulting critical appraisal systems have been published in the literature to date. The first is the Quality Checklists for Blogs and Podcasts.Reference Paterson, Colmers, Lin, Thoma and Chan ⁶ The checklists are qualitative sets of criteria that learners can apply to appraise critically specific online content for their personal learning. These checklists were derived from recently published studies on blog and podcast quality indicators (a systematic review,Reference Paterson, Thoma, Milne, Lin and Chan ²³ an expert Delphi with podcasters and bloggers,Reference Thoma, Chan and Paterson ²⁴ and a Delphi with expert educatorsReference Lin, Thoma and Trueger ²⁵ ).

The second is the METRIQ score.Reference Chan, Thoma and Krishnan ⁸ A recent derivation study proposed two shortened scores named METRIQ5 and METRIQ8 that are more efficient for individuals seeking to evaluate online educational resources.Reference Chan, Thoma and Krishnan ⁸ Lastly, the ALiEM Approved Instructional Resources (AIR) score,Reference Lin, Joshi and Grock ²⁶ has also been shown to be reliable and consistent with expert educator opinions of quality, if used by a calibrated team.Reference Chan, Grock, Paddock, Kulasegaram, Yarris and Lin ^{7 ,} Reference Chan, Thoma and Krishnan ⁸ The AIR Score was developed by the ALiEM group of educators to help select high-quality FOAM resources for their self-study curriculum, the ALiEM AIR Series. The AIR Score was derived through consensus by medical educators involved in the curation and production of FOAM. It has been determined to have reasonable reliability with nine raters and to correlate with expert educator gestalt. It has not yet been validated for use among non-educators or learners.Reference Chan, Grock, Paddock, Kulasegaram, Yarris and Lin ^{7 ,} Reference Lin, Joshi and Grock ²⁶ Ongoing studies are presently looking to assess the validity evidence supporting the METRIQ scores and to begin to optimize these quality assessment tools for widespread adoption.

The ubiquity of online education resources in EM educationReference Cadogan, Thoma, Chan and Lin ^{1 -} Reference Thurtle, Banks, Cox, Pain and Furyk ³ makes it important for teachers to consider the role of critical appraisal skills that are required for this unique genre of information. Table 1 summarizes some tools of importance for teachers who wish to address the issue of critical appraisal with their learners. It may be worthwhile to use structured appraisal tools such as METRIQ or ALiEM AIR scoresReference Chan, Thoma and Krishnan ^{8 ,} Reference Lin, Joshi and Grock ²⁶ and Quality Checklists for Blogs and PodcastsReference Paterson, Colmers, Lin, Thoma and Chan ⁶ during residency training. While the evidence regarding these scores is mounting, faculty-guided discussions around how best to appraise and apply knowledge from these secondary resources can help junior learners develop skills at critically considering online resources. In the future, these evaluation instruments may guide producers of FOAM materials to improve their resources and allow institutions to have a more objective measure for the quality of FOAM content.

Table 1 A selection of critical appraisal tools developed for online educational resources

AIR=Approved Instructional Resources; HON=Health On the Net.

Strategy four: participate by contributing and producing

Beyond navigating and evaluating online educational resources, learners can further engage in the FOAM community by participating in discussion, criticism, peer review, and creation of open educational products.

As an alternative to creating a new website or podcast to host content, new developers should consider contributing new content to existing resources (Table 2). Not only will this save content developers the time and effort of creating a new platform, but also many of these sites incorporate a peer-review process to provide feedback and coaching for new contributors. Finally, these resources will offer a considerably larger readership for newly contributed material than most new sites.

Table 2 A selection of EM FOAM outlets that have open submission processes

EM=emergency medicine; FOAM=free open access medical education.

Other practical and valuable ways of contributing to FOAM resources include participating in the peer review of content, reporting errors in blogs or podcasts to authors, commenting and engaging in a debate, and participating in discussions by joining in Tweet chatsReference Choo, Ranney and Chan ²⁷ and online journal clubsReference Roberts, Perera and Lawrentschuk ²⁸ (e.g., #JGMEScholarReference Lin and Sherbino ²⁹ and #ALiEMJCReference Lin, Joshi, Hayes and Chan ³⁰ ).

Next steps

The online education landscape is rapidly evolving. Some academic institutions are starting to support the production of open educational resources that have lead to the continued success of several prominent FOAM resources such as Core EM, which is supported by New York University,Reference Swaminathan and Core ³¹ and Emergency Medicine Cases, which is supported by the University of Toronto Division of Emergency Medicine and the Schwartz/Reisman Emergency Medicine Institute.Reference Chartier and Helman ³² New definitions of scholarship are now created by international consensus, highlighting key changes on how online educational resources might be reconceptualized as scholarly work by academic institutions.Reference Sherbino, Arora and Van Melle ³³ Additionally, some institutions have begun rewarding the development of FOAM products as recognized acts of scholarship that add value toward promotion.Reference Cabrera ³⁴

We encourage both educators and learners to use the strategies in this article to integrate the use of FOAM into their teaching and continuing medical education (CME). Innovative teaching strategies using FOAM, such as the “flipped classroom,” are increasingly described in medical education literature.Reference Sherbino, Chan and Schiff ^{35 ,} Reference Prober and Khan ³⁶ The appearance and popularity of FOAM-related projects such as online journal clubsReference Roberts, Perera and Lawrentschuk ^{28 -} Reference Lin, Joshi, Hayes and Chan ³⁰ suggest both interest and advantage in these novel self-learning formats.

Notably absent from the menu of strategies presented within this article is guidance on how to find and evaluate educational videos. While many resource producers embed videos from Vimeo^TM or YouTube^TM onto their websites, many video-based resources are uploaded directly to video-hosting websites. Research in this area is sorely needed, as contrary to blog posts and podcasts, there are currently no guidelines to assist in the production or critical appraisal of these resources, nor are there strategies to find high-quality channels.

To foster FOAM integration into formal medical education, it will be important to develop a more rigorous approach for content production and evaluation techniques. While we describe advancements in the assessment of production quality, further development of more robust measures of educational quality and tools for learner evaluation is warranted. Those motivated to pioneer in this niche may benefit from incorporation of existing educational techniques such as identifying a target audience and subject matter, developing a formal curriculum (e.g., using Kern’s six-step framework),Reference Kern, Thomas and Hughes ³⁷ and incorporating measures for evaluating learners beyond attitudinal data (e.g., using Kirkpatrick levels of evaluation).Reference Yardley and Dornan ³⁸