We comment on Sijtsma’s (2014) thought-provoking essay on how to minimize questionable research practices (QRPs) in psychology. We agree with Sijtsma that proactive measures to decrease the risk of QRPs will ultimately be more productive than efforts to target individual researchers and their work. In particular, we concur that encouraging researchers to make their data and research materials public is the best institutional antidote against QRPs, although we are concerned that Sijtsma’s proposal to delegate more responsibility to statistical and methodological consultants could inadvertently reinforce the dichotomy between the substantive and statistical aspects of research. We also discuss sources of false-positive findings and replication failures in psychological research, and outline potential remedies for these problems. We conclude that replicability is the best metric of the minimization of QRPs and their adverse effects on psychological research.

When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.

By making building instructions freely accessible to everyone, open-source machine tools (OSMTs) promise to democratize manufacturing by enabling users in marginalized settings to build machines tools by themselves. There is, however, a lack of empirical evidence of the replicability of OSMT designs in low-resource contexts. This article explores OSMT replicability through qualitative and empirical methods to answer the central research question: Are designs that are fully open source also globally replicable? A comparative experiment was carried out by replicating an open-source 3D printer in two different locations: in Germany (resource-rich) and in Oman (resource-poor). The experiment aimed to determine the barriers faced with the replication in each location. It was significantly more challenging to replicate the 3D printer in Oman, primarily due to difficulties in sourcing and manufacturing, necessitating extensive modifications, which demanded greater skills and dexterity from users compared to those in Germany. Qualitative interviews found that limited digital literacy posed a significant barrier for microenterprise owners in replicating OSMT. Finally, design guidelines were proposed to enhance the global replicability of contextualized OSMT designs.

With its promise of nondestructive processing, rapid low-cost sampling, and portability to any field site or museum in the world, portable X-ray fluorescence (pXRF) spectrometry is rapidly becoming a standard piece of equipment for archaeologists. Even though the use of pXRF is becoming standard, the publication of pXRF analytical methods and the resulting data remains widely variable. Despite validation studies that demonstrate the importance of sample preparation, data collection settings, and data processing, there remains no standard for how to report pXRF results. In this article, we address the need for best practices in publishing pXRF analyses. We outline information that should be published alongside interpretive results in any archaeological application of pXRF. By publishing this basic information, archaeologists will increase the transparency and replicability of their analyses on an inter-analyst/inter-analyzer basis and provide clarity for journal editors and peer reviewers on publications and grant proposals for studies that use pXRF. The use of these best practices will result in better science in the burgeoning use of pXRF in archaeology.

Religious belief is a topic of longstanding interest to psychological science, but the psychology of religious disbelief is a relative newcomer. One prominently discussed model is analytic atheism, wherein cognitive reflection, as measured with the Cognitive Reflection Test, overrides religious intuitions and instruction. Consistent with this model, performance-based measures of cognitive reflection predict religious disbelief in WEIRD (Western, Educated, Industrialized, Rich, & Democratic) samples. However, the generality of analytic atheism remains unknown. Drawing on a large global sample (N = 3461) from 13 religiously, demographically, and culturally diverse societies, we find that analytic atheism as usually assessed is in fact quite fickle cross-culturally, appearing robustly only in aggregate analyses and in three individual countries. The results provide additional evidence for culture’s effects on core beliefs.

Bilingualism is hard to define, measure, and study. Sparked by the “replication crisis” in the social sciences, a recent discussion on the advantages of open science is gaining momentum. Here, we join this debate to argue that bilingualism research would greatly benefit from embracing open science. We do so in a unique way, by presenting six fictional stories that illustrate how open science practices – sharing preprints, materials, code, and data; pre-registering studies; and joining large-scale collaborations – can strengthen bilingualism research and further improve its quality.

The diversity of design research studies and their associated methods and reporting style make it difficult for the design research community of practice to leverage its work into further advancing the field. We illustrate how a structured multilevel analysis of diverse studies creates a canonical model that allows for the transfer of insight between studies, enhances their comprehension, and supports improved study designs. The benefits of such an approach will increase if different stakeholders adopt such structured approaches to enrich the design research community of practice.

This article provides an accessible tutorial with concrete guidance for how to start improving research methods and practices in your lab. Following recent calls to improve research methods and practices within and beyond the borders of psychological science, resources have proliferated across book chapters, journal articles, and online media. Many researchers are interested in learning more about cutting-edge methods and practices but are unsure where to begin. In this tutorial, we describe specific tools that help researchers calibrate their confidence in a given set of findings. In Part I, we describe strategies for assessing the likely statistical power of a study, including when and how to conduct different types of power calculations, how to estimate effect sizes, and how to think about power for detecting interactions. In Part II, we provide strategies for assessing the likely type I error rate of a study, including distinguishing clearly between data-independent (“confirmatory”) and data-dependent (“exploratory”) analyses and thinking carefully about different forms and functions of preregistration.

Psychology and neighboring disciplines are currently consumed with a replication crisis. Recent work has shown that replication can have the unintended consequence of perpetuating unwarranted conclusions when repeating an incorrect line of scientific reasoning from one study to another. This tutorial shows how decision researchers can derive logically coherent predictions from their theory by keeping track of the heterogeneity of preference the theory permits, rather than dismissing such heterogeneity as a nuisance. As an illustration, we reanalyze data of Barron and Ursino (2013). By keeping track of the heterogeneity of preferences permitted by Cumulative Prospect Theory, we show how the analysis and conclusions of Barron and Ursino (2013) change. This tutorial is intended as a blue-print for graduate student projects that dig deeply into the merits of prior studies and/or that supplement replication studies with a quality check.

Replicability of findings is an essential prerequisite of research. For both basic and clinical research, however, low replicability of findings has recently been reported. Replicability may be affected by research biases not sufficiently controlled for by the existing research standards. Several biases such as researcher allegiance or selective reporting are well-known for affecting results. For psychotherapy and pharmacotherapy research, specific additional biases may affect outcome (e.g. therapist allegiance, therapist effects or impairments in treatment implementation). For meta-analyses further specific biases are relevant. In psychotherapy and pharmacotherapy research these biases have not yet been systematically discussed in the context of replicability. Using a list of 13 biases as a starting point, we discuss each bias's impact on replicability. We illustrate each bias by selective findings of recent research, showing that (1) several biases are not yet sufficiently controlled for by the presently applied research standards, (2) these biases have a pernicious effect on replicability of findings. For the sake of research credibility, it is critical to avoid these biases in future research. To control for biases and to improve replicability, we propose to systematically implement several measures in psychotherapy and pharmacotherapy research, such as adversarial collaboration (inviting academic rivals to collaborate), reviewing study design prior to knowing the results, triple-blind data analysis (including subjects, investigators and data managers/statisticians), data analysis by other research teams (crowdsourcing), and, last not least, updating reporting standards such as CONSORT or the Template for Intervention Description and Replication (TIDieR).