A central theme of this book is how people learn from samples. In this chapter, we look at this issue from two viewpoints. First, we question the manner in which we – as scientists – sample the environments of the experimental participants who engage in the judgment and decision-making tasks that we study. To what extent are these samples representative of the natural decision-making ecology that our participants face in their everyday lives? Second, by actually sampling people's activities in their natural ecologies, we seek to characterize how they experience these environments. In particular, we investigate how one feature of environments (the presence or absence of feedback) affects inferences (the confidence people express in their decisions).

The first question is, of course, not new. For many years, psychologists have been concerned about how to generalize behavior from experimental evidence (see, e.g., Brunswik, 1956; Chapanis, 1961, 1967; Cronbach, 1975; Ebbesen & Konečni, 1980; Hammond, 1978; Hogarth, 1986; Lipshitz et al., 2001). I do not propose to add to this debate. Instead, the contribution is to demonstrate how the use of readily available technology can greatly facilitate random sampling of decision behavior outside the psychological laboratory. Indeed, as I shall argue in the following, obtaining appropriate samples of human decision behavior is not as difficult as might be imagined by researchers trained within experimental paradigms.