The problem

No matter how often billiard balls have moved when struck in the past, the next billiard ball may not move when struck. For philosophers, this ‘theoretical’ possibility of being wrong raises a problem about how to justify our theories and models of the world and their predictions. This is the problem of induction. In practice, nobody denies that the next billiard ball will move when struck, so many scientists see no practical problem. But in recent times, scientists have been presented with competing methods for comparing hypotheses or models (classical hypothesis testing, BIC, AIC, cross-validation, and so on) which do not yield the same predictions. Here there is a problem.

Model selection involves a trade-off between simplicity and fit for reasons that are now fairly well understood (see Forster and Sober, 1994, for an elementary exposition). However, there are many ways of making this trade-off, and this chapter will analyse the conditions under which one method will perform better than another. The main conclusions of the analysis are that (1) there is no method that is better than all the others under all conditions, even when some reasonable background assumptions are made, and (2) for any methods A and B, there are circumstances in which A is better than B, and there are other circumstances in which B will do better than A. Every method is fraught with some risk even in well-behaved situations in which nature is ‘uniform’. Scientists will do well to understand the risks.