In this chapter we cover alternative approaches to multiverse analysis, such as specification curve analysis, sample splitting, and model selection.

Are female hurricanes more deadly? In this chapter we demonstrate multiverse analysis using analytical inputs from many scholars in a high-profile empirical debate. In results from more than 10,000 model specifications, only 12 percent of estimates are statistically significant and 99 percent are smaller in magnitude than what the original authors reported. Multiverse analysis shows that some published findings are extremely weak and nonrobust.

Having developed the multiverse framework in detail, let’s take it for empirical road tests. Are banks biased in mortgage lending? Do job training programs lead to higher wages? How much do the answers to these questions depend on modeling assumptions? Sometimes “significant” results are very stable and robust across models, while other results are mostly null, supported in one in a hundred credible models. In this chapter we demonstrate how to conduct and interpret a basic multiverse analysis, and we cover basic multiverse commands in Stata.

In this chapter we explain how to build a plausible space using the all-combinations algorithm, and how to evaluate its results. When the true model is unknown, what models are plausible? What set of models would a task force of rival scholars embrace? This provides a central foundation for a broader multiverse that develops in later chapters. We also show how to incorporate concepts of necessary controls and estimated consistency.