Social psychology findings have fared poorly in multi-site replication attempts. This article considers and evaluates multiple factors that may contribute to such failures, other than the “crisis” assumption that most of the field’s published research is so badly flawed that it should be dismissed wholesale. Low engagement by participants may reduce replicability of some findings (while not affecting certain others). Incentives differ between original researchers and replicators. If multi-site replications are indeed biased toward failure, this may have a damaging effect on the field’s ability to build correct theories.

We conducted a replication of Shafir (1993) who showed that people are inconsistent in their preferences when faced with choosing versus rejecting decision-making scenarios. The effect was demonstrated using an enrichment paradigm, asking subjects to choose between enriched and impoverished alternatives, with enriched alternatives having more positive and negative features than the impoverished alternative. Using eight different decision scenarios, Shafir found support for a compatibility principle: subjects chose and rejected enriched alternatives in choose and reject decision scenarios (d = 0.32 [0.23,0.40]), respectively, and indicated greater preference for the enriched alternative in the choice task than in the rejection task (d = 0.38 [0.29,0.46]). In a preregistered very close replication of the original study (N = 1026), we found no consistent support for the hypotheses across the eight problems: two had similar effects, two had opposite effects, and four showed no effects (overall d = −0.01 [−0.06,0.03]). Seeking alternative explanations, we tested an extension, and found support for the accentuation hypothesis.

Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences.

Science is based on facts—facts that are systematically gathered by a community of enquirers through detailed observation and experiment. In the twentieth century, however, philosophers of science claimed that the facts that scientists “gather” in this way are shaped by the theories scientists accept, and this seemed to threaten the authority of science. Call this the old worries about science. By contrast, what seemed not to threaten that authority were other factors that shaped the facts that scientists gather—for example, the mere questions scientists pursue. Call this the old nonworries about science. What I suggest is that the old nonworries are turning out to be far more worrisome than the old worries, and I use recent goings-on such as the “Death of Evidence” protests in Canada, the “replication crisis,” and the ongoing feminist critiques of science to illustrate my case. All this raises interesting new questions for philosophers of science to tackle.

False positive findings in science are inevitable, but are they particularly common in psychology and psychiatry? The evidence that we review suggests that while not restricted to our field, the problem is acute. We describe the concept of researcher ‘degrees-of-freedom’ to explain how many false-positive findings arise, and how the various strategies of registration, pre-specification, and reporting standards that are being adopted both reduce and make these visible. We review possible benefits and harms of proposed statistical solutions, from tougher requirements for significance, to Bayesian and machine learning approaches to analysis. Finally we consider the organisation and methods for replication and systematic review in psychology and psychiatry.