Design Phase

We established an interdisciplinary design group that included clinical research educators, computer scientists, organizational psychologists, instructional designers, graphic artists, and end users. The group engaged in an 8-month iterative development process, building from a partnership between UC San Diego Altman Clinical and Translational Research Institute and VirBELA, Inc., a creator of virtual worlds for collaboration and education. The design group’s initial plan was to develop a virtual world that addressed broad team science and leadership competencies, with a series of “mini-games” that simulated multiple steps in the clinical trial process (e.g., staff selection, Gantt charts). These ideas were presented to our targeted end users (i.e., predoctoral and postdoctoral scholars and junior faculty engaged in clinical research training and their instructors) with storyboards, and feedback revealed tensions between devising a simulation experience to teach the elements of clinical trials Versus gameplay that focused more on the interactions and strategies for communication. Feedback indicated that linking communication and collaboration strategies with performance in a stylized clinical trial would best capitalize on the unique strengths of the virtual world. Moreover, the initial design was intended to connect progressive experiences over multiple sessions. Pragmatic considerations regarding feasibility led to the conclusion that the most scalable approach would be a shorter game that could be embedded in multiple courses containing team science content. We then developed a facilitator guide and didactic content connecting game experience with the learning objectives.

The choice of a multisite clinical trial as a setting was selected over other options (e.g., collaborative planning for a novel study) because of the variety of roles in a clinical trial, applicability to broad range of learners, and the inherent necessity to simultaneously consider individual, “site” and trial-wide performance in a multisite trial. Detailed game elements, including the duration, perspective, scoring and performance metrics were specified in a game concept document [Reference Ibrahim and Jaafar9]. Assessment and refinement of the beta-version involved about 15 meetings with 40 end users and administrators, and over 500 hours of software development time.

Game Learning Objectives and Design

The learning objectives of the completed game are to (1) identify interdependencies between professionals in a clinical trial needed to achieve a shared goal, (2) reflect on communication as a critical component of individual role performance to support team-level performance, and (3) learn strategies for using trial accrual data to facilitate team coordination discussions aimed for performance improvement.

The game is played in a virtual world 3D campus programmed with the commercially available Unity 3D engine. Learners first download the application (available for Macintosh or Windows operating systems). They next register an account and create an avatar to represent themselves. After team members view game-play instructions, they enter the game, which can be calibrated to last either 5 or 10 minutes. The object of the game is to recruit, enroll, and complete the protocol for study participants during a timed period. The game is played by teams of 4 learners, with each individual team member taking 1 of the 4 “roles” which represent professionals in a clinical trial: Recruiters, who are responsible for identifying and prescreening potential study participants among a general population; Screeners, who identify eligibility of recruited individuals based on inclusion and exclusion criteria; Coordinators, who manage the day-to-day completion of study-related activities; and Principal Investigators (PIs), who use budget-limited resources to improve processes and manage overall team performance (see Fig. 1). Each team may also include an optional nonparticipant Observer who does not play a role but is present in the virtual world. Each player works semi-independently in a distinct virtual space (with the exception of the PI, who is able to navigate freely throughout the virtual setting). The minimum number of players is 4, and up to 4 teams or “sites” may play the game for a maximum of 16 simultaneous players.

Fig. 1 Screenshots of team roles. From top left: Recruiter, Screener, Coordinator, Principal Investigator. At the top of each screen, time remaining, budget, and enrollment metrics are displayed.

Although players are tied to their individual tasks, they are able and encouraged to communicate with their teammates using Voice over IP. The game is designed with several potential opportunities for “bottlenecks” in managing study flow, as well as several relevant points of information that are not revealed fully to all players, such as the impact of various recruitment strategies on the complexity of the screener’s tasks and the negative impact of participant “wait times” on attrition. Thus, intrateam communication is critical in successfully completing the game. Once the allotted time has run out, team members reconvene in a debrief “room,” in which they view indicators of their performance in the clinical trial, including the number of participants recruited, screened, retained and completed (see Fig. 2). Learners can compare their outcome with other teams, introducing competition to increase engagement. The teams can play multiple times and experiment with different communication and teamwork strategies in order to improve their performance.

Fig. 2 Screenshot of the debrief room. After completion of the game, team performance metrics are displayed and discussed in comparison with other teams.

In-Class Facilitation

We developed the game to be played within a 1.5–2-hour facilitated discussion, with discussion connecting game experience with relevant principles in team science. The following in-class agenda was employed:

1. (1) Group discussion (10 min): learners first reflect on their experiences in effective and ineffective teams, in research teams or, without such experience, in clinical teams.

2. (2) Team science and clinical trials (10 min): trends in increased prevalence of multiauthored manuscripts and multi-PI funding proposals [1], as well as the increasing complexity of research projects, are discussed. Typologies of scientific teams, from unidisciplinary to interdisciplinary and transdisciplinary [1] provide context for trainees to reflect on their current clinical and research teams.

3. (3) Communication and coordination in team science (10 min): using a brief case study in workplace communication [Reference Pentland10], facilitators and trainees discuss the effect of communication in team performance. Common conflicts and challenges in team science [Reference Zucker11], as well as potential mitigation strategies, are discussed.

4. (4) Gameplay (30–40 min): two 5 or 10 minute rounds of the game are played, with time for debrief and strategy-building between rounds. Mid-point debrief discussion elucidates pitfalls in gameplay and communication, strategies for improved performance, and team dynamics.

5. (5) Final debrief (10 min): the facilitator guides trainees through a discussion of the generalization of their game experience with “real-world” teams.

6. (6) Team science resources (10 min): the discussion ends with additional reading and topical resources available for trainees, which include an NIH-sponsored toolkits and field guides [Reference Bennett, Gadlin and Levine-Finley12, 13], resources to assess team readiness and performance [Reference Goldsmith and Morgan14, Reference Olson and Olson15] and conflict mitigation information [Reference Gadlin and Jessar16].

The freely downloadable software program, facilitator guide, and didactic materials are available through http://www.ctri.ucsd.edu/education/Pages/GameBasedLearning.aspx.

Evaluation, Analysis, and Results

Our initial focus of evaluation is acceptability, and we have implemented the game in different groups of learners as a single session embedded in research training curricula at UC San Diego, including: (1) a career development seminar for MD or PhD junior faculty in clinical research, such as KL2 scholars (n=15), (2) a credit-bearing applied lab-based course in a Master’s in Clinical Research with medical students (n=20), and (3) an interdisciplinary T32 training program for junior faculty members with MDs or PhDs (n=5). Evaluations were gathered voluntarily and anonymously via Qualtrics, eliciting Likert-type satisfaction ratings and open-ended questions soliciting perceived benefits and ideas for improvement and future application; our primary outcome was user satisfaction with the class session. Given the focus on acceptability, analyses were descriptive and no data on current or prior team participation or long-term outcomes were collected. These data were collected anonymously as part of routine evaluations and therefore the study was not subject to human research protections. Data analysis was approved by the UC San Diego HRPP, under Exempt Status.

Responses indicate a high level of satisfaction with the session (mean of 8.8 out 10, SD=1.6), perceived relevance to the understanding the topic of team science, as well as usefulness to research careers (Table 1). Qualitative feedback aligned with the learning objectives in providing an applied experience in coordination and communications. Suggestions for improvement included modifications to game instructions and aspects of the user controls. Learners identified potential extensions of the game in the context of stimulating discussion on roles and communication in actual research teams.

Table 1 Scholar evaluation data (n=19)