1 Introduction
Online experiments have become a valuable research tool for psychologists and economists interested in human cooperation (Reference Arechar, Gächter and MollemanArechar et al., 2018; Reference Horton, Rand and ZeckhauserHorton et al., 2011). Online experiments offer potential advantages, such as large sample sizes (Reference Hauser, Hendriks, Rand and NowakHauser et al., 2016) and access to diverse participants (Reference Nishi, Christakis and RandNishi et al., 2017); and increase the feasibility of multi- and cross-national studies (Reference Dorrough and GlöcknerDorrough & Glöckner, 2019; Reference Romano, Balliet, Yamagishi and LiuRomano et al., 2017). When social dilemmas experiments are conducted online, researchers often use asynchronous designs, where participants complete an experiment individually and are matched with partners after data collection has been completed. Asynchronous experiments differ from synchronous experiments in two important ways: First, participants are not matched to a specific partner before decisions are made. Second, participants do not receive immediate feedback on the consequences of their choices. The present research tests whether pre-decision matching and immediate feedback influence cooperation, and whether the effects of these design features are moderated by individual differences in prosocial preferences. The proposed research contributes to our understanding of how common research methods influence psychological processes and behavior in online social dilemmas experiments.
1.1 Conducting Social Dilemmas Experiments Online
Social dilemmas are situations where individuals must make a choice between pursuing self-interest and the collective good (Reference DawesDawes, 1980), and the study of social dilemmas is an important point of intersection for researchers in the behavioral sciences (Reference Van LangeVan Lange et al., 2013). As the use of online samples in psychological research has grown (Reference BirnbaumBirnbaum, 2000; Reference Buhrmester, Kwang and GoslingBuhrmester et al., 2011; Reference Gosling, Vazire, Srivastava and JohnGosling et al., 2004), many social dilemmas researchers have begun to rely on online participant pools such as Amazon’s Mechanical Turk (mTurk) and Prolific Academic.
Importantly, online social dilemmas experiments also produce valid data: Amir et al. (2012) found that online behavior in typical economic games (e.g., the public good game, the trust game, the ultimatum game, and the dictator game) resembles behavior observed in laboratory experiments (e.g., proposers in the ultimatum game reject unfair offers, and reciprocity in the trust game is proportional to the initial level of trust). Arechar et al. (2018) also demonstrated that online participants can be used to study behavior in repeated games: as in laboratory studies, cooperation deteriorated in later rounds of a repeated interaction, but was bolstered by introducing the possibility of peer punishment.
There are also potential limitations to online experiments. Although data quality is generally adequate (Reference Buhrmester, Kwang and GoslingBuhrmester et al., 2011), experimental manipulations may be affected by the non-naivete and prior experience of online participants (Reference Chandler, Mueller and PaolacciChandler et al., 2014). Illustrating this problem, Rand et al. (2014) found that manipulations of time pressure in social dilemmas became progressively less effective over time, in part because more participants developed prior experience with experimental games. Similarly, Chandler et al. (2015) found that effect sizes in a range of psychological tasks became less pronounced among experienced participants. However, the effects of participant sophistication can be attenuated by presenting participants with new variants of existing paradigms (Reference Rand, Peysakhovich, Kraft-Todd, Newman, Wurzbacher, Nowak and GreeneRand et al., 2014).
1.2 Synchronicity in Online Experiments
Although synchronous experiments, where participants have live interactions and receive feedback in real-time, are possible using software programs such as oTree (Reference Chen, Schonger and WickensChen et al., 2016) and classEx (Reference Arechar, Gächter and MollemanArechar et al., 2018), most online experiments are asynchronous. To illustrate this point, we surveyed articles published in the Judgment and Decision Making journal over an eight-year period (2013–2020): There were 40 published studies using online samples to study behavior in social dilemmas, and all of these studies were conducted using asynchronous methods. No studies that we know of have systematically compared behavior in synchronous and asynchronous online experiments. In this section, we review research suggesting that behavior may be affected by two defining features of fully synchronous experiments, pre-decision matching and immediate feedback.
1.2.1 Pre-decision Versus Post-decision Matching
In synchronous experiments, participants are assigned to specific partners before any decisions are made; in asynchronous experiments, partner assignment happens only after data collection is complete. We predicted that pre-decision matching would reduce the perceived social distance between participants. This prediction is motivated by research on charitable giving and the identifiable victim effect: people are more willing to help a single victim compared to a group of statistical victims (Reference Kogut and RitovKogut & Ritov, 2005a, 2005b). Critically, this effect does not depend on the specific characteristics of the individual victim (Reference Kogut and RitovKogut & Ritov, 2005a), and it can even occur when the recipient of help remains unidentified (Reference Lee and FeeleyLee & Feeley, 2016; Reference Small and LoewensteinSmall & Loewenstein, 2003). Arguably, people experience a stronger emotional connection to the recipient of help when the recipient is identified as any specific person (Reference Small and LoewensteinSmall & Loewenstein, 2003; Reference Small, Loewenstein and SlovicSmall et al., 2007). In social dilemmas, we anticipated that assigning participants to interact with specific partners before they make decisions would reduce perceived social distance.
Additionally, other work suggests that pre-decision matching may reduce distance by leading people to interpret their interactions as social exchanges, rather than abstract reasoning problems: Research on the strategy method, which requires participants to make conditional decisions for all possible situations (Reference Brandts and CharnessBrandts & Charness, 2011), suggests that people sometimes become less trusting (Reference Murphy, Rapoport and ParcoMurphy et al., 2006) and less trustworthy (Reference Casari and CasonCasari & Cason, 2009) when social decisions are presented abstractly. Subtle social cues, such as referring to other players as partners versus opponents (Reference McCabe, Rigdon and SmithMcCabe et al., 2003) or describing an interaction as a community game versus Wall Street game (Reference Liberman, Samuels and RossLiberman et al., 2004), can encourage prosocial behavior by making the norm of reciprocity salient. In the same way, pre-decision matching may change how participants construe their decisions, leading them to feel closer and more interconnected with their interaction partners.
We expected that pre-decision matching would reduce social distance, and that reduced social distance, in turn, would result in increased cooperation. People are generally more prosocial towards those they feel close to (Reference Jones and RachlinJones & Rachlin, 2006, 2009), and this increase in cooperation can even occur when proximity is based on arbitrary procedures, such as the minimal group paradigm (Reference Balliet, Wu and De DreuBalliet et al., 2014; Reference Goette, Huffman and MeierGoette et al., 2006). In summary, our first prediction was that pre-decision matching would reduce social distance and, in turn, increase cooperation among strangers.
1.2.2 Immediate Versus Delayed Feedback
In synchronous experiments, participants immediately receive feedback on the outcomes of their decisions; in asynchronous online experiments, there is a delay, sometimes hours or days, before participants learn about interaction partners’ decisions and their final payoffs for the experiment. Delaying feedback adds a temporal dimension to the social dilemma, and previous studies have found that cooperation is more difficult to sustain when potential outcomes are projected into the future (Reference Joireman, Van Lange and Van VugtJoireman et al., 2004; Reference Kortenkamp and MooreKortenkamp & Moore, 2006).Footnote 1
We predicted that immediate feedback would increase cooperation by reducing feelings of aversive uncertainty related to the fear of exploitation. Undesirable outcomes, such as losing money or receiving an electric shock, are perceived as worse when they are projected into the future (Reference LoewensteinLoewenstein, 1987). For risky decisions, delayed feedback affects the subjective perception of the likelihood and impact of negative outcomes, and thus makes people more likely to select relatively safe options (Reference Kogler, Mittone and KirchlerKogler et al., 2016; Reference Muehlbacher, Mittone, Kastlunger and KirchlerMuehlbacher et al., 2012). In social dilemmas, defection may be seen as “safer” than cooperation because choosing defection eliminates the possibility of the worst possible outcome (i.e., the sucker’s payoff). Therefore, our second hypothesis was that immediate feedback would reduce feelings of aversive uncertainty and lead to increased cooperation.
1.2.3 Synchronicity and Social Value Orientation
In addition to considering the group-level effects of synchronous experiments, we also investigated whether the effects of synchronous designs were moderated by individual differences in Social Value Orientation (SVO) (Reference Murphy and AckermannMurphy & Ackermann, 2014; Reference Van LangeVan Lange, 1999). We hypothesized that prosocial individuals (i.e., those with stronger preferences to maximize joint outcomes or outcome equality) would be more likely to be affected by the distinction between synchronous and asynchronous experiments. Compared to fully self-focused individualists, prosocials are more sensitive to situational cues (Reference Bogaert, Boone and DeclerckBogaert et al., 2008) and are more likely to adapt their expectations and behavior based on context (Reference Van LangeVan Lange, 1999). In other words, prosocials cooperate in social dilemmas when it can be justified by the constraints of the situation; individualists, on the other hand, tend to be unconditionally self-interested (Reference Epstein, Peysakhovich and RandEpstein et al., 2016; Reference Yamagishi, Li, Takagishi, Matsumoto and KiyonariYamagishi et al., 2014). Thus, we expected that pre-decision matching and immediate feedback would have stronger (positive) effects on cooperation for individuals with stronger prosocial preferences.
1.3 Overview of Hypotheses
We conducted an experiment comparing behavior in synchronous versus asynchronous dilemmas. We examined how two features of synchronous experiments, pre-decision matching and immediate feedback, influenced cooperation. We predicted that pre-decision matching and immediate feedback would both increase cooperation. More precisely, we expected that pre-decision matching would reduce feelings of social distance and that immediate feedback would reduce feelings of aversive uncertainty. In addition to considering the group-level effects of synchronous experiments, we also tested whether individual differences in SVO moderated their effects on behavior. We hypothesized that pre-decision matching and immediate feedback would have stronger positive effects on the behavior of individuals with prosocial preferences. Our Stage 1 report can be viewed at https://osf.io/7qnej/?view_only=d79a1ba27cd34a01b0d14fa0b8cb03a2.
2 Method
2.1 Participants
Power Analysis
We conducted power analyses using G*Power 3.1 (Reference Faul, Erdfelder, Buchner and LangFaul et al., 2009) to identify the number of participants needed to detect a small effect (ϕ =.1) using a Chi Square test with 3 groups, 80% power, and α = .05: minimum N = 964. We adjusted this estimate based on an expected dropout rate of roughly 10%.Footnote 2 Our total planned sample size was N = 1,200.
We recruited 1238 participants from Prolific Academic. We ended up with slightly more participants than expected because Prolific sometimes classified participants as “timed out” while they were still completing the experiment. The average age was 26.64 years (SD = 9.46); and there were 648 men, 383 women, 12 non-binary participants, and 195 participants who did not report genders.
Recruitment
We conducted twelve experimental sessions with 100 available spaces per session. Sessions were launched on weekdays at 2:00PM CEST. The first three sessions were conducted in November 2020, and the remaining nine sessions were conducted in January 2021. Participants were prevented from completing the study more than once using Prolific’s “previous study” filter. Participants received a show up payment of £1.50 each, and those who finished the experiment also received bonus payments based on their choices (£0.50 to 4.00 per person).
2.2 Materials and Procedure
The experiment was administered using oTree (Reference Chen, Schonger and WickensChen et al., 2016). The experiment did not involve any deception.
Prisoners Dilemma
Participants made decisions in a binary choice prisoners dilemma. Each participant was assigned to a partner and chose Keep or Transfer: If both players choose Keep, then they received 100 points each; If both players chose Transfer, then they received 200 points each; If one player chose Keep and the other chose Transfer, then they received 300 points and 0 points, respectively. After reading the instructions, participants were presented with four comprehension questions (Example: “If both you and the other participant choose “Transfer”, how many points will you receive?”). The majority of participants (75%) answered all four questions correctly; and rates of accuracy did not differ across experimental conditions: asynchronous = 75%; partially synchronous = 75%; synchronous = 73% (χ 2(2) = 0.55, p = .76). Our primary analyses included all participants. Following our pre-registration, we also conducted supplemental analyses using only data from participants who answered all four questions correctly.
The full instructions and experiment materials are reported in the Appendix.
Proposed Mediators
We hypothesized that social distance and aversive uncertainty would mediate the effects of pre-decision matching and immediate feedback (respectively) on cooperation. The items measuring these constructs are reported in Table 1a. We randomized whether participants responded to the proposed mediators immediately before or immediately after they made decisions in the prisoners dilemma, and this randomization occurred at the session level.
Table 1: Summary of hypothesized mediators, post-decision measures, and post-feedback measures
Post-decision Measures
After cooperation decisions were made, participants were presented with a series of questions before they received feedback on the outcome of the interaction. We measured expectations of cooperation in their interaction partners; their confidence in these expectations; and feelings of anticipated satisfaction and regret. These items are reported in Table 1b. After completing these post-decision measures, participants then completed the 6-item slider measure of SVO (Reference Murphy and AckermannMurphy & Ackermann, 2014). In this measure, each participant was asked to make a series of hypothetical allocation decisions, where they decided how many points to share with an anonymous interaction partner.
Post-feedback Measures
After participants learned about their partners’ choices and the outcome of the interaction (including information about their bonus payment), they responded to a series of items before concluding the experiment. To measure participants’ reactions to feedback, we asked participants how satisfied they were with the outcome of the interaction and whether they regretted their choice in the interaction. Participants also played a mini-dictator game, where they decided whether to share additional money with the other participant from the prisoners dilemma. We also measured how much participants enjoyed the experiment, whether they believed that they were interacting with a real person in the interaction, the perceived fairness of the interaction, and basic demographics questions. The complete measures are included in Table 1c.
Experimental Conditions
Participants were assigned to partners based on the time at which they begin the experiment. There were three experimental conditions (summarized in Table 2): asynchronous (post-decision matching with delayed feedback); partially synchronous (pre-decision matching with delayed feedback); and fully synchronous (pre-decision matching with immediate feedback). Experimental conditions were assigned at the level of session, with four sessions for each condition.
Table 2: Summary of experimental conditions and descriptive statistics
Note: “Matching rate” refers to the proportion of participants who were successfully assigned to partners during the first part of the experiment. “Follow up rate” refers to the proportions of participants that completed the second part of the study (which was administered one week after the first part).
Pre- vs post-decision matching
Participants in the pre-decision matching conditions were assigned to partners before they made decisions in the prisoners dilemma. If no partner was immediately available, then participants were redirected to a waiting screen where they were asked to wait for a period of up to five minutes. If no partner could be located during that time frame, then participants had the option to continue waiting for another five minutes or to terminate the experiment, in which case they received the show up payment (but they did not receive any bonus payment). Across sessions, almost all participants (0.98) were successfully matched with partners. The average waiting time was 9.66 seconds (SD = 66.02).
In the post-decision matching condition, participants were assigned to partners after they had already made decisions in the prisoners dilemma.
Immediate vs delayed feedback
In the immediate feedback condition, participants learned about the outcome of the prisoners dilemma as soon as both players made their decisions and answered the post-decision questions. In the delayed feedback conditions, participants were contacted via Prolific messages one week after the initial experiment session and invited to complete the study. A majority of participants (80%) from the delayed feedback conditions completed the second part of the study.
2.3 Deviations from pre-registered protocol
There were five ways in which our study deviated from our pre-registered protocol:
-
1. We recruited participants from Prolific Academic rather than MTurk. We made this switch given growing concerns about the presence of bots and deteriorating data quality on MTurk (Reference Chmielewski and KuckerChmielewski & Kucker, 2020).
-
2. We reduced participant payments from $3 to £1.50. Given the duration of our study (typical completion time of 5–8 minutes), $3 would have been an unusually high rate-of-payment on the Prolific platform (typical studies pay £0.10 to 0.20 per minute).
-
3. Our original plan was to administer multiple sessions per day. We decided to run only one session per day during the peak usage hour on Prolific (https://researcher-help.prolific.co/hc/en-gb/articles/360011657739-When-are-Prolific-participants-most-active-). The purpose of this change was to increase the number of available participants, and to maximize the chance that participants would be successfully matched in real-time.
-
4. We randomized whether mediation questions were presented before (or after) the prisoners dilemma. Originally, we planned to randomize this factor at the level of dyad. Instead, we decided to implement this randomization at the session level.
-
5. After we collected data for sessions 1–3 of the study, a participant informed us about a potential problem. Some participants were able to view their condition assignments in the Internet browser’s URL bar. We paused further data collection until we were able to fix this issue. Note that excluding participants from the first three sessions did not change any of our results.
3 Results
3.1 Primary Analyses
Our primary analyses focused on the effects of pre-decision matching and immediate feedback on cooperation, social distance, and feelings of aversive uncertainty. Descriptive statistics for these variables by condition are shown in Figure 1. Following our analysis plan, we conducted two-tailed tests with α = .05.
Figure 1: Levels of cooperation (A), social distance (B), and aversive uncertainty (C) by experimental condition. Plots B and C use violin plots and box plots to show distributions of responses within each condition. Bold lines indicate median responses, and box widths indicate values that lie within the first and third quartiles.
Pre-decision matching
Our first prediction was that pre-decision matching would increase cooperation via reduced social distance. First, we used a logistic regression to compare the rates of cooperation in the asynchronous and partially synchronous conditions (-.5 = asynchronous; +.5 = partially synchronous). Pre-decision matching had no significant effect on cooperation (b = −0.19, SE = 0.15, p = .20). Then, we compared the levels of social distance between the two conditions. Pre-decision matching did not significantly affect social distance (b = 0.28, SE = 0.16, p =.084). Social distance was, however, negatively associated with cooperation (b = −0.08, SE = 0.03, p = .012).
Immediate feedback
Our second hypothesis was that immediate feedback would reduce feelings of uncertainty and lead to increased cooperation. We compared the rates of cooperation in partially synchronous and fully synchronous conditions (-.5 = partially synchronous; +.5 = fully synchronous). Immediate feedback had no significant effect on cooperation (b = −0.05, SE = 0.15, p = .75). Then, we compared the levels of aversive uncertainty across conditions. Immediate feedback had no significant effect on aversive uncertainty (b = −0.04, SE = 0.20, p = .83). Aversive uncertainty was not significantly associated with cooperation (b = 0.05, SE = 0.03, p = .052).
Social Value Orientation
Our third hypothesis was that the effects of synchronous experiments would be moderated by individual differences in SVO. More specifically, we expected that pre-decision matching and immediate feedback would have stronger effects on cooperation for individuals with stronger prosocial orientations. To test this prediction, we estimated a series of regression models predicting cooperation, social distance, and aversive uncertainty. Each model included the following variables as predictors: pre-decision (vs. post-decision) matching; immediate (vs. delayed) feedback; Social Value Orientation angle (mean centered); and two SVO by experimental condition interaction terms. The results are reported in Table 3. Reassuringly, SVO was positively correlated with cooperation, but we found no support for the predicted SVO-by-condition interactions.
Table 3: The interactive effects of Social Value Orientation, pre-decision matching, and immediate feedback
3.2 Secondary Analyses
Robustness checks
To test the robustness of our primary analyses, we conducted two sets of alternative analyses: First, we estimated models including participants demographics (i.e., age, gender, English proficiency, and income) as covariates. Then, we estimated the models outlined in the previous section using only participants who correctly answered all four comprehension questions. Results were consistent with our primary analyses (See Appendix).
Post-decision measures
We also conducted a series of exploratory analyses of the effects of pre-decision matching and immediate feedback on four post-decision measures: expectations of cooperation, confidence in expectations, anticipated satisfaction, and anticipated regret. The results are reported in Table 4. There were no significant effects of either pre-decision matching or immediate feedback.
Table 4: The effects of pre-decision matching and immediate feedback on post-decision measures
Post-feedback measures
Next, we examined the effects of pre-decision matching and immediate feedback on six outcomes measured after participants learned about the outcome of the prisoners dilemma: behavior in a mini dictator game; experienced satisfaction with the outcome of the prisoners dilemma; experienced regret about choices in the prisoners dilemma; enjoyment of the experiment; the perceived fairness of the experiment; and the perceived realism of the experiment. We also included participant payoff (the amount of points earned in the prisoners dilemma, scaled from 0 points = −1 to 400 points = +1) as a predictor, as well as two payoff by experimental condition interaction terms. The results are reported in Table 5. Pre-decision matching and immediate feedback had little, if any effect, on post feedback outcomes. Out of 24 regression coefficients involving experimental condition, one was significant at p < .05. Not surprisingly, participant payoff had large effects on post-feedback outcomes: Participants who earned more in the prisoners dilemma were more altruistic in the dictator game; were more satisfied with their outcomes and had less regret about their choices; and enjoyed the experiment more and perceived it as more fair.
Table 5: The effects of payoff and experimental condition on post-feedback measures
* p<.001;
** p<.01;
*** p<.05.
Waiting time
To conclude, we investigated whether waiting time (in the two pre-decision matching conditions, N = 787) was correlated with cooperation, perceived closeness, aversive uncertainty, or perceived realism. To account for the non-normality of the waiting time data, we log-transformed it. There were no significant correlations (r’s < .05, p’s > .18).
4 Discussion
Researchers often use asynchronous online experiments to examine the psychological processes underlying cooperation. However, it is unclear how asynchronous interactions differ psychologically from synchronous interactions, where participants interact in real-time and receive immediate feedback. We examined the effects of two salient features of synchronous experiments, pre-decision matching and immediate feedback, on cooperation in the prisoners dilemma. We found that pre-decision matching and immediate feedback had no significant effects on cooperation (or on how participants perceived the interaction). The present results suggest that synchronous design features have little effect on behavior in online experiments measuring cooperation.
4.1 Synchronous Online Experiments
In our study, we tested three hypotheses about the effects of synchronicity on cooperation: Our first hypothesis was that pre-decision matching would increase cooperation via reduced social distance. Pre-decision matching had no effects on cooperation or perceived social distance; however, social distance was associated with decreased cooperation. Previous studies have found consistent evidence that people help and cooperate with socially proximate interaction partners (Reference Jones and RachlinJones & Rachlin, 2006, 2009). Here, we found that merely assigning participants to a specific interaction partner is not sufficient to create feelings of social proximity.
Our second hypothesis was that immediate feedback would increase cooperation by reducing participants’ feelings of aversive uncertainty about the possibility of exploitation. Immediate feedback had no effects on cooperation and aversive uncertainty, and aversive uncertainty was not significantly associated with cooperation. Introducing a (one-week) delay in outcome does not substantially affect behavior in the prisoners dilemma.
Our third hypothesis was that SVO would moderate the effects of pre-decision matching and immediate feedback on behavior. We found that SVO was correlated with cooperation and negatively correlated with social distance. However, SVO did not moderate the effects of synchronicity. This is unsurprising, as participants were generally insensitive to the differences between experimental conditions.
We also examined the effects of synchronicity on perceptions of the realism of the experiment: Interestingly, having participants engage in real-time interactions had no significant effects on the perceived realism of the experiment. Responses to our “perceived realism” question were relatively close to the midpoint, 5 out of 10, across all conditions; and participants did not feel socially close to their interaction partners. In terms of realism, synchronous experiments do not convey much advantage over asynchronous experiments. This may point to a general limitation of experiments using economic games, rather than a specific limitation of asynchronous experiments. Other design features likely have larger effects on the extent to which participants perceive a social dilemma experiment as a real interaction. To increase realism, researchers may need to provide participants with identifying information about their interaction partners (Reference Evans and KruegerEvans & Krueger, 2016), or allow participants to communicate directly during the experiment (Reference Dawes, McTavish and ShakleeDawes et al., 1977).
4.2 Limitations
In our study, we measured behavior using one social dilemma, the dyadic prisoners dilemma. We cannot rule out the possibility that synchronous designs would matter more in other settings. For example, synchronicity could matter more in sequential (rather than simultaneous) interactions, such as the trust game or the ultimatum game. Previous research has argued that contextual cues play an important role in activating the norm of reciprocity in sequential exchanges (Reference McCabe, Rigdon and SmithMcCabe et al., 2003). On the other hand, research suggests that people show high levels of consistency in their behavior across different types of experimental games (Reference Yamagishi, Mifune, Li, Shinada, Hashimoto, Horita, Miura, Inukai, Tanida and KiyonariYamagishi et al., 2013). Researchers interested in pursuing these questions further should be prepared for the possibility that partner matching and feedback effects are small and relatively difficult to detect.
Additionally, our manipulation of delayed feedback focused only on one time interval, immediate feedback versus a one-week delay. We focused on the interval of one-week because this is a typical delay of payment in online experiments. However, we cannot rule out the possibility that longer time delays (one-month or longer) could have effects on aversive uncertainty and cooperation. It is also important to consider whether our choice of payoff stakes affected our results: We used standard payoff stakes for online experiments (£0.50 to 4.00), which were equivalent to the payments participants would receive 5 to 40 minutes of work on Prolific. It is possible, but not likely, that larger payoff stakes would increase participants’ sensitivity to pre-decision matching and delayed feedback (Reference Amir, Rand and GalAmir et al., 2012).
Finally, it is important to note that our results may have been influenced by participant non-naivete (Reference Chandler, Mueller and PaolacciChandler et al., 2014). Previous studies have raised the possibility that experimental manipulations have weaker effects on participants once they have become familiar with a paradigm (Reference Chandler, Mueller and PaolacciChandler et al., 2014; Reference Rand, Peysakhovich, Kraft-Todd, Newman, Wurzbacher, Nowak and GreeneRand et al., 2014). This is a general problem for online experiments conducted on platforms like Prolific or mTurk. Pre-decision matching and immediate feedback may have larger effects on participants who are generally unfamiliar with social dilemmas. Moreover, some participants may have been skeptical about the veracity of our study. Concerns about deception and the contamination of shared subject pools are unavoidable (Reference Hertwig and OrtmannHertwig & Ortmann, 2008). Indeed, a small number of participants sent messages indicating that they did not believe they were actually partnered with other participants. However, we found no evidence that belief in the realism of the study was correlated with behavior in the prisoners dilemma.
4.3 Advice for Social Dilemmas Researchers
At this point, researchers interested in social dilemmas may wonder whether it is worthwhile to conduct synchronous experiments: On the positive side, our study demonstrates that it is feasible to conduct large scale studies involving real-time partner matching and multiple time measurements. Nearly all participants were successfully matched to partners (98%) and participant retention was relatively high across the two waves of the study (80%). At the same time, synchronous experiments also require a substantial time investment (compared to asynchronous experiments conducted using Qualtrics, or similar software).
While we did not find any effects of synchronicity on behavior, allowing for the possibility of synchronous interactions opens up new potential research questions. The study of social dilemmas has largely focused on anonymous, one-shot interactions without communication and minimal social information (Reference Thielmann, Spadaro and BallietThielmann et al., 2020; Reference Van LangeVan Lange et al., 2013). However, interactions with strangers account for a relatively small percentage (~10%) of daily social interactions (Reference Columbus, Molho, Righetti and BallietColumbus et al., 2021). Arguably, this focus has been influenced by the relative ease of running asynchronous experiments. As our study demonstrates, software packages such as oTree (Reference Chen, Schonger and WickensChen et al., 2016) are making it easier for researchers to conduct high-powered studies of cooperation that go beyond zero-acquaintance interactions.
4.4 Conclusion
How does synchronicity influence perception and behavior in online experiments measuring cooperation? We found that pre-decision matching and immediate feedback had no significant effects on behavior in the prisoners dilemma or on how participants perceived the interaction. The present results suggest that synchronous designs and asynchronous designs can produce similar results in studies of online cooperation.
Appendix
Experiment Materials
Prisoners Dilemma Instructions (Screen 1)
In the next part of this study, you will play a game.
You will be randomly paired with another participant. Each of you simultaneously and privately choosesKeeporTransfer. Your payoffs will be determined by your choice and the other participant’s choice:
In each cell, the amount (in points) to the left is the payoff for you and the amount to the right is the payoff for the other participant.
You will receive a bonus payment based on the total number of points you earn. 100 points = $1.00.
Before you continue, please answer the following comprehension questions:
If both you and the other participant choose “Transfer”, how many points will you receive? [0, 100, 200, 300]
If both you and the other participant choose “Keep”, how many points will you receive? [0, 100, 200, 300]
If you choose “Keep” and the other participant choose “Transfer”, how many points will you receive? [0, 100, 200, 300]
If you choose “Transfer” and the other participant choose “Keep”, how many points will you receive? [0, 100, 200, 300]
Time Delay Instructions (Screen 2)
[Pre-decision matching with Immediate feedback]
You will learn about the outcome of the game as soon as you and the other participant make your decisions.
[Pre-decision matching with Delayed feedback]
[Post-decision matching with Delayed feedback]
You will learn about the outcome of the game in one week (when we complete data collection for this experiment.
The date today is X. This means you will be contacted one week from today, on X.
Partner matching screen (Screen 3)
Figure 2
[Participants can not proceed until their partner has also read and progressed through the instruction screen]
Decision Screen (Screen 4)
Additional Analyses
We tested the effects of including covariates and excluding participants who did not pass our comprehension checks on our results. Covariate models also included age, gender, income, and English proficiency as predictors: There were fourteen income levels (ranging from “less than £10,001 per year” to “over £100,000 per year”) and four levels of English proficiency (fluent, advanced, basic, poor); and these two variables were entered into our models using dummy coding. The results across models are reported in Table A1.
Table 6: Analyses including covariates and excluding participants who did not pass comprehension checks
Open access
