Replication is already mainstream in areas of psychology that use small-N designs. Replication failures often result from weak theory, weak measurement, and weak control over error variance. These are hallmarks of phenomenon-based research with sparse data. Small-N designs, which focus on understanding processes, treat the individual rather than the experiment as the unit of replication and largely circumvent these problems.

Direct replications are not always affordable or feasible, and for some phenomena they are impossible. In such situations, methods of blinded data analysis can help minimize p-hacking and confirmation bias, increasing our confidence in a study's results.

In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensure reproducibility.

Data collection in psychology increasingly relies on “open populations” of participants recruited online, which presents both opportunities and challenges for replication. Reduced costs and the possibility to access the same populations allows for more informative replications. However, researchers should ensure the directness of their replications by dealing with the threats of participant nonnaiveté and selection effects.

Scientists should, above all else, value the truth. To do this effectively, scientists should separate their identities from the data they produce. It will be easier to make replications mainstream if scientists are rewarded based on their stance toward the truth – such as when a scientist reacts positively to a failure to replicate – as opposed to a particular finding.

Although Zwaan et al. argue that original researchers should provide a replication recipe that provides great specificity about the operational details of one's study, I argue that it may be as important to provide a recipe that allows replicators to conduct a study that matches the original in as many conceptual details as possible (i.e., an exact conceptual replication).

Psychology is in the middle of a replicability revolution. High-profile replication studies have produced a large number of replication failures. The main reason why replication studies in psychology often fail is that original studies were selected for significance. If all studies were reported, original studies would fail to produce significant results as often as replication studies. Replications would be less contentious if original results were not selected for significance.

Whether or not a replication attempt counts as “direct” often cannot be determined definitively after the fact as a result of flexibility in how procedural differences are interpreted. Specifying constraints on generality in original articles can eliminate ambiguity in advance, thereby leading to a more cumulative science.

Replication failures were among the triggers of a reform movement which, in a very short time, has been enormously useful in raising standards and improving methods. As a result, the massive multilab multi-experiment replication projects have served their purpose and will die out. We describe other types of replications – both friendly and adversarial – that should continue to be beneficial.

Scientific knowledge is supposed to be verifiable. Replications promote verifiability in several ways. Most straightforwardly, replications can verify empirical claims. Replication research also promotes dissemination of information needed for other aspects of verification; creates meta-scientific knowledge about what results to treat as credible even in the absence of replications; and reinforces a broader norm of scientists checking each other's work.

We advocate that replications should be an integral part of the scientific discourse and provide insights about the conditions under which an effect occurs. By themselves, mere nonreplications are not informative about the “truth” of an effect. As a consequence, the mechanistic continuation of multilab replications should be replaced by diagnostic studies providing insights about the underlying causes and mechanisms.

We discuss the authors' conceptualization of replication, in particular the false dichotomy of direct versus conceptual replication intrinsic to it, and suggest a broader one that better generalizes to other domains of psychological research. We also discuss their approach to the evaluation of replication results and suggest moving beyond their dichotomous statistical paradigms and employing hierarchical/meta-analytic statistical models.

The widespread replication of research findings in independent laboratories prior to publication is suggested as a complement to traditional replication approaches. The pre-publication independent replication approach further addresses three key concerns from replication skeptics by systematically taking context into account, reducing reputational costs for original authors and replicators, and increasing the theoretical value of failed replications.

We contest the “building a wall” analogy of scientific progress. We argue that this analogy unfairly privileges original research (which is perceived as laying bricks and, therefore, constructive) over replication research (which is perceived as testing and removing bricks and, therefore, destructive). We propose an alternative analogy for scientific progress: solving a jigsaw puzzle.

Replications can make theoretical contributions, but are unlikely to do so if their findings are open to multiple interpretations (especially violations of psychometric invariance). Thus, just as studies demonstrating novel effects are often expected to empirically evaluate competing explanations, replications should be held to similar standards. Unfortunately, this is rarely done, thereby undermining the value of replication research.

Before replication becomes mainstream, the potential for generating theoretical knowledge better be clear. Replicating statistically significant nonrandom data shows that an original study made a discovery; replicating a specified theoretical effect shows that an original study corroborated a theory. Yet only in the latter case is replication a necessary, sound, and worthwhile strategy.

The commentaries on our target article are insightful and constructive. There were some critical notes, but many commentaries agreed with, or even amplified our message. The first section of our response addresses comments pertaining to specific parts of the target article. The second section provides a response to the commentaries' suggestions to make replication mainstream. The final section contains concluding remarks.

The domain of “folk-economics” consists in explicit beliefs about the economy held by laypeople, untrained in economics, about such topics as, for example, the causes of the wealth of nations, the benefits or drawbacks of markets and international trade, the effects of regulation, the origins of inequality, the connection between work and wages, the economic consequences of immigration, or the possible causes of unemployment. These beliefs are crucial in forming people's political beliefs and in shaping their reception of different policies. Yet, they often conflict with elementary principles of economic theory and are often described as the consequences of ignorance, irrationality, or specific biases. As we will argue, these past perspectives fail to predict the particular contents of popular folk-economic beliefs and, as a result, there is no systematic study of the cognitive factors involved in their emergence and cultural success. Here we propose that the cultural success of particular beliefs about the economy is predictable if we consider the influence of specialized, largely automatic inference systems that evolved as adaptations to ancestral human small-scale sociality. These systems, for which there is independent evidence, include free-rider detection, fairness-based partner choice, ownership intuitions, coalitional psychology, and more. Information about modern mass-market conditions activates these specific inference systems, resulting in particular intuitions, for example, that impersonal transactions are dangerous or that international trade is a zero-sum game. These intuitions in turn make specific policy proposals more likely than others to become intuitively compelling, and, as a consequence, exert a crucial influence on political choices.

Folk-economic beliefs may be regarded as “evidential fictions” that exploit the natural tendency of human cognition to organize itself in narrative form. Narrative counter-arguments are likely more effective than logical debunking. The challenge is to convey sound economic reasoning in narratively conspicuous forms – an opportunity for economics to rethink its role and agency in public discourse, in the spirit of its old classics.

Boyer & Petersen (B&P) lay out a compelling theory for folk-economic beliefs, focusing on beliefs about markets. However, societies also allocate resources through mechanisms involving power and group decision-making (e.g., voting), through the political economy. We encourage future work to keep folk political economic beliefs in mind, and sketch an example involving pollution and climate change mitigation policy.