Discovery space

We astronomers mostly work in the ‘discovery space’, the region where effects are statistically significant at less than three-sigma or near boundaries in data or parameter space. Working in the discovery space is a normal astronomical activity; few published results are initially found at large confidence. This can lead to anomalous results, e.g., positive definite quantities (such as mass, fractions, star formation rates, dispersions, etc.) are sometimes found to be negative or, more generally, quantities are sometimes found at unphysical values (completeness larger than 100%, V/Vmax > 1 or fractions larger than 1, for example). Working in the discovery space is typical of frontier-line research because almost every significant result reaches this status after having appeared first in the discovery space, and because a good determination of known effects or trends usually triggers searches for finer, harder to detect, effects, mostly falling once more in the discovery space.

Many of us are very confident that commonly used statistical tools work properly in the situations in which we use them. Unfortunately, in the discovery space, and sometimes outside it, we should not take this for granted, as shown below with a few examples. We cannot avoid working in this grey region, because to move our results into the statistically significant area we often need a larger or better sample. In order to obtain this, we first need to convince the community (and the telescope time allocation committees) that an effect is probably there, by working in the discovery space.