No such thing as unconditional data

A presupposition behind integrative experiment design – and related proposals like multiverse analysis, which attempts to amend the limitations of a single analysis by rendering explicitly a design space to sample from (Steegen, Tuerlinckx, Gelman, & Vanpaemel, Reference Steegen, Tuerlinckx, Gelman and Vanpaemel2016) – is that by zooming out from a narrow focus (on just a few variables, or a single analysis path) and sampling results from a larger space, they will produce unbiased evaluations of a claim. In integrative modeling, tests of surrogate models take the form of prediction problems in a supervised learning paradigm, with the added implied constraint that they must “accurately explain the data researchers have already observed.”

But the theories that arise from integrative experiment design will be conditional on more than just the features used to train them. How to interpret the “tests” of surrogate models is an important degree of freedom, for example. Measures like sample complexity can supply requirements to resolve prediction accuracy within a chosen error bound, but not what bound should constitute sufficient predictive performance, or how it should differ across domains. There is a chicken-and-egg problem in attempting to separate the experimental findings from the definition of the learning problem and sampling approach.

If integrative experiment design also incorporates explanatory methods, and the explanations take the form of causal mechanisms proposed to operate in different regions of the design space, then this explanatory layer may very well make it easier for experimenters to draw on domain knowledge, helping retain predictive accuracy when moving out-of-distribution relative to a “pure prediction” approach. But this is difficult to conclude without defining what makes a surrogate model interpretable.

Goldilocks and the crud factor

Both multiverse analysis and integrative experiment design can seem to presuppose that our prior knowledge can take us just far enough to produce results more complex than current results sections, but not so complicated that we get overwhelmed. The “new kinds of theories” associated with integrative experimentation are meant to “capture the complexity of human behaviors while retaining the interpretability of simpler theories.” This may be possible, but we should be careful not to assume that we can always zoom out until we find the dimensionality that is considerably greater than the dimensionality of the problem implied by the status quo single experiment, but not so great as to be noncomprehensible to a human interpreter.

If we take seriously Meehl's notion of the crud factor, we might easily list hundreds of potential influences, for example, on group performance, from interpersonal attractions among group members to their religious orientations to recent current events. Even if the prior literature boils some of these down to encompassing unidimensional summaries (e.g., religious homogeneity) there will be many ways to measure each, and many ways to analyze the results which might have their own consequences. How do we guarantee that the number of choices that matter yield interpretable explanations? To take seriously the promises of approaches like integrative experimentation, we must contextualize them within a theory of interpretability.

Multiverse and integrative experiment design provide solutions that are more relative than precise: Sampling from the larger space better captures our fundamental ontological uncertainty about the true data-generating model than not defining and sampling from the larger space, but cannot eliminate it. By prioritizing data over theory, both approaches gesture toward completeness, but cannot provide guarantees. Under philosophical scrutiny, their clearest value seems to be rhetorical. Consequently the completeness that such methods seem to promise can be misleading.

These points should not discourage adoption of integrative experimentation, which is likely to improve learning from experiments by addressing many important criticisms raised with the status quo. However, as confident but often informal proposals scientific reforms abound, it is always worth deep consideration of what problems are addressed, and what promises, if any, can be made (Devezer, Navarro, Vandekerckhove, & Buzbas, Reference Devezer, Navarro, Vandekerckhove and Ozge Buzbas2021). Integrative experiment design is one way of improving learning from experiments, which can complement but cannot replace the need to clarify what we learn from any experiment – single or integrative – in the first place. To reform science we will also need to “zoom in” by formalizing our expectations within a theoretical framework and foregrounding the conditionality of our inferences.