3.1 Simon-related articles

We analyzed the 377 articles published in the Journal of Behavioral Decision Making and Judgment and Decision Making from 2006 to 2010. At first glance, most of these articles relate to bounded rationality in the general terms that Simon described it: “…theories that incorporate constraints on the information processing capacities of the actor may be called theories of bounded rationality” (1972, p. 162). We used stricter criteria for judging an article to be Simon-related or not. First, we looked at citations and found 44 articles that cite Simon. Second, we judged 47 additional articles as relating directly to one or both of Simon’s goals even without citing him.

For examples of descriptive articles consider two special issues in Judgment and Decision Making edited by Reference Newell and BroederNewell and Broeder (2008) and Reference Marewski, Pohl and VitouchMarewski, Pohl and Vitouch (2010). The two issues have the same structure: Models from Gigerenzer, Todd and the ABC research group’s (1999) adaptive toolbox, such as take-the-best and the recognition heuristic, are compared in terms of their descriptive adequacy to more mainstream psychological models such as evidence accumulation models and exemplar models.

As an example of a prescriptive article consider Reference Serwe and FringsSerwe and Frings (2006). In order to predict the winner of a match at the Wimbledon tennis tournament, they proposed the following: “Pick the player who is recognized by more laypeople”. This fits with Simon’s approach of basing prescription on people’s heuristics given that people, under some conditions, use the recognition heuristic (Hilbig, 2010, among others, argues that these conditions are narrow). For more examples see the special issue in honor of Paul Meehl by Reference Morera and DawesMorera and Dawes (2006) in the Journal of Behavioral Decision Making. The articles in this issue study the accuracy of linear models with regression or unit weights which have also been proposed as descriptive models.

The 91 articles are cited in the reference list and are also provided in the two tables below where Table 1 summarizes the results for the Journal of Behavioral Decision Making (45 articles) and Table 2 for Judgment and Decision Making (46 articles).

Table 1: The 45 articles published in the Journal of Behavioral Decision Making from 2006 to 2010 that cite Simon or that we judged as relating to his prescriptive or descriptive goal. For each article, we judged whether it relates to Simon’s prescriptive goal (1) or not (0) and whether it is integrative (1*) (see text for definitions). The same coding was used for our judgments of relation to Simon’s descriptive goal. We also judged whether articles targeted the topics of formal modeling, experimental methods of process tracing, the distinction between intuition and deliberation and the distinction between maximizing and satisficing (1 for yes and 0 for no). The main results are: (a) 6 out of 9 prescriptive articles are judged to be integrative, (b) 17 out of 40 descriptive articles are judged to be integrative and (c) the most popular topic is by far formal modeling (21 articles).

Table 2: The 46 articles published in Judgment and Decision Making from 2006 to 2010 that cite Simon or that we judged as relating to his prescriptive or descriptive goal. For the coding used for our judgments see the caption of Table 1. The main results are: (a) 12 out of 18 prescriptive articles are judged to be integrative, (b) 20 out of 31 descriptive articles are judged to be integrative and (c) the most popular topic is by far formal modeling (28 articles).

For each article, we judged whether it relates to Simon’s prescriptive and descriptive goals (marked by 1) or not (marked by 0). Simon pursued his descriptive and prescriptive goals by strongly relying on theory and our main interest was to judge whether Simon-related JDM articles are theoretically integrative (marked by 1*). The integration of theories has been argued to be “psychology’s most important task” (Reference GigerenzerGigerenzer, 2010, p. 733; for an example of what he means see Reference Luan, Schooler and GigerenzerLuan, Schooler and Gigerenzer, 2011) and there is a history of complaints that it has not been achieved (Reference WatkinsWatkins, 1984; Reference MischelMischel, 2009). These authors have made the point for descriptive theories and we make it for prescriptive theories as well. We are interested in the integration of theories only, not in the integration of other parts of research such as experimental methods for testing theories or analyzing data. To ensure that our judgments were consistent, we specified a number of criteria for integrative articles:

For a descriptive article to be judged as integrative, we requested that it contributes to building a theory that reconciles different conceptualizations of cognition such as neural networks, evidence accumulation models and heuristics. As an example consider Reference Newell and BroederNewell and Broeder (2008) who used a number of concepts from cognitive psychology to place a number of formal JDM models in the same theoretical framework (Figure 1, p. 200). In our definition of integrative descriptive articles we use a somewhat strict meaning of reconciling models. For instance, we did not judge as integrative those studies that compared the descriptive adequacy of competing cognitive models but did not create a super model that incorporated concepts from the various models. Of course, an article was not judged as integrative if it studied just one descriptive model.

For a prescriptive article to be judged as integrative, we requested that it contributes to building a method that combines elements of other methods such as heuristics and optimization models or incorporates different yardsticks of rationality such as Hammond’s (2007) coherence and correspondence. For examples of integration see the Judgment and Decision Making special issue edited by Dunwoody (2009), as in Tape (2009) who argues that “using the best of both approaches [coherence and correspondence] could…lead to striking advances in medical science” (p. 134). As with descriptive articles we use a somewhat strict meaning of combining methods. For instance, we did not judge as integrative those studies that compared the performance of competing methods but did not create a hybrid method that included ideas from the various methods. Of course, an article was not judged as integrative if it studied just one method for making decisions.

In Section 3.2. we discuss whether the 91 articles in Tables 1 and 2 were found to be integrative. Before that we organize the content of the articles. The vast majority of the articles target one or more of the following topics: formal models of decision making, experimental methods of process tracing, the distinction between intuition and deliberation and the distinction between maximizing and satisficing. Formal modeling is by far the most popular topic, targeted by 49 articles (9 articles target process tracing and the intuition/deliberation distinction and 4 target the maximizing/satisficing distinction).

Simon himself worked on all of these topics. For example, he was skilled in mathematics and computing and wrote extensively on formal economic models of decision making such as utility theory. Simon also worked on experimental methods such as protocol analysis (Reference Bereby-Meyer and GrosskopfEricsson & Simon, 1993). And he was preoccupied with the relative merits of a theoretically optimal solution versus a practically good enough solution as can be seen in The Sciences of the Artificial, especially Chapter 5. Finally, Simon touched on the now popular distinction between intuition and deliberation, stating, for example: “we cannot, of course, rule out the possibility that the unconscious is a better decision-maker than the conscious” (1955, p. 104).Footnote ²

3.2 Are the Simon-related articles integrative?

We emphasize that our judgments are subjective and should not be taken at face value but rather be used in order to stimulate further thinking. As seen in Tables 1 and 2, the results are: (i) The number of descriptive articles is much higher than the number of prescriptive articles (71 vs. 27) and (ii) the proportion of integrative articles is higher for prescriptive than for descriptive articles but not much higher than half of the articles (67% and 52% respectively).Footnote ³

It seems that a number of articles focuses on producing novel effects, not on developing theory. This is consistent with complaints about a general lack of theory in social (Reference FiedlerFiedler, 1991) and cognitive psychology (Reference GigerenzerGigerenzer, 1998). According to our definitions, such articles cannot be integrative. We next provide more subtle examples of the lack of theoretical integration.Footnote ⁴

Reference Serwe and FringsSerwe and Frings (2006) came up with the prescriptive rule “Pick the player who is recognized by more laypeople” for predicting the outcomes of tennis matches and showed that it matched the accuracy of official rankings. But they did not go a step further and investigate how the rule could be combined with other methods. Yet this step is necessary because we know that recognition-based rules do not always perform well (Reference Andersson and RakowAndersson & Rakow, 2007). For example, it would be worthwhile to evaluate the performance of a super method that uses recognition-based rules in the first rounds of a tennis tournament and switches to the official rankings in latter rounds. In general, no method performs well for all decision problems in the real world and it is important to think about how to construct hybrids that capitalize on diverse methods such as heuristics and optimization (for a family of such hybrids see Katsikopoulos, 2011, Figure 2, p. 25).

As an example of lack of integration in descriptive articles, consider the numerous debates on the descriptive adequacy of the adaptive toolbox and its alternatives. A sign of segregation in these debates is that researchers have sometimes been divided into “advocates” and “critics” or “proponents” and “critics” of various models (as in Marewski et al., 2010). Such divisions are not helpful. In fact they are harmful in the sense that importance is placed on who said what instead of what has been said. An extreme version of the problem is focusing on who said what when (more recently, Reference Marewski, Pohl and VitouchMarewski, Pohl & Vitouch, 2011, p. 371 have also discussed this problem). For example, one critique of the current theory of the recognition heuristic is that it is inconsistent with its previous versions. Leaving the validity of this critique aside, we cannot see why researchers should not try to improve their understanding of a topic in the course of time. As Herbert Simon is anecdotally quoted to have said, articles are often progress reports, not the final word. An integrative approach to the descriptive theory of JDM should focus on the current shared understanding of a topic and on improving it in the future, not on paying a lot of attention to who was right or wrong in the past.

The last example refers not to a particular article but to an exchange in the Journal of Behavioral Decision Making. After Reference Gloeckner, Betsch and SchindlerGloeckner, Betsch, and Schindler (2010) argued that it is not precisely specified how people select a heuristic from the adaptive toolbox, Marewski (2010) responded that it is Gloeckner et al.’s (2010) own parallel constraint satisfaction (PCS) model that cannot solve this selection problem, to which Reference Gloeckner, Betsch and SchindlerGloeckner and Betsch (2010) replied that PCS does not need to solve the selection problem. In this discussion the researchers’ different conceptualizations of cognition are not reconciled. There is little progress on what kind of concepts in general would allow models to solve the selection problem, only arguments why their model can do so or the others’ model cannot.

In the next section, we speculate on the reasons for the lack of integration.

3.3 Why Is there a lack of integration and what can we do about it

We first consider descriptive Simon-related articles. We begin with an observation by Gigerenzer (2010). He pointed out that in psychology “We teach our students how to test theories, not the art of theory construction in the first place” (2010, p. 733, emphasis added). The same can be said for JDM. It seems to us that the level of effort in analyzing data is not matched by the level of effort expended in thinking about the primitive concepts of theories. Let us explain this with some examples.

Reference Gaissmaier, Schooler and MataReisen, Hoffrage and Mast (2008) and Gloeckner (2009) developed methods for uncovering people’s decision models that jointly use measures such as choices, reaction times, eye movements, verbal protocols and information search in a Mouselab environment.

These are serious efforts to solve a difficult problem. Contrast it with work on modeling, say, the recognition heuristic. It is clear that such models need to be grounded on memory research (Reference Dougherty, Franco-Watkins and ThomasDougherty, Franco-Watkins & Thomas, 2008; Reference Erdfelder, Küpper-Tetzel and MatternErdfelder, Küpper-Tetzel & Mattern, 2011). The discussion on how to do this revolves around questions such as how to model recognition in a way that is not all-or-none (as it was done originally).

Such questions are important. But it is not what we mean by thinking hard about primitive concepts. What we have in mind are questions such as “When should JDM models use variables that are observable (e.g., yes/no judgments of recognition) or latent (e.g., strength of memory trace)?” There are no obvious answers as the following disagreement illustrates. In Luce’s view (1997, p. 83), “latent, not directly observable structures … afford too many options, leading to badly underdetermined models with many free parameters”. On the other hand, Reference Newell, Lagnado and ShanksNewell, Lagnado and Shanks (2007) are happy with the move in the Rescorla-and-Wagner learning model, from predicting “response probabilities” to “internal association strengths or weights,” because “it is a person’s belief about an association between an action and an outcome that we want to understand, rather than the superficial manifestation of that belief” (p. 155). Both views have merit but are in contrast and have to be reconciled. In our observation, little effort has been made towards such integration.

Based on the above, we conclude that in JDM we do not always think hard about the concepts on which our descriptive models are based. But without such thinking, theory integration is bound to be incomplete.

Next we consider prescriptive Simon-related articles. As with descriptive articles, the lack of integration likely has to do with research tradition: A single view of prescriptive theory, the one based on Bayesian inference and expected utility theory, has dominated (Reference Katsikopoulos and FasoloKatsikopoulos & Fasolo, 2006). This can to a large extent be attributed to Ward Edwards, arguably the father of JDM research (Reference HeukelomHeukelom, 2010).

For example, in a panel discussion on what constitutes a good decision at the 1983 Subjective Probability and Utility in Decision Making conference in Groningen, Edwards announced that “no principle other than maximizing SEU [subjective expected utility] deserves a moment of consideration” (Reference VlekVlek, 1984, p. 7). This conviction is apparent throughout Edwards’ career and writings (Reference Weiss and WeissWeiss & Weiss, 2009; Reference KatsikopoulosKatsikopoulos, 2010). Findings that question the dominance of the prescriptive theory of Bayesian inference and expected utility theory, such as those of Reference Diab, Gillespie and HighhousePayne, Bettman and Johnson (1993) and Gigerenzer et al. (1999) are often dismissed (Reference HogarthHogarth, 2011) and have only recently begun to be viewed as serious challenges.

What can we do to promote theoretical integration in JDM research? As a starting point, we can use Gigerenzer’s (2010) suggestions of encouraging submissions of integrative articles in leading journals and teaching theory construction in graduate school. In our view, a precondition for constructing good theories is precise conceptual thinking. For instance, a textbook that cultivates this kind of thinking and illustrates it with examples from JDM research is Cognitive Modeling by Reference Busemeyer and DiederichBusemeyer and Diederich (2010). Furthermore, the pursuit of integration with disciplines outside psychology can be fostered by recalling Simon’s commitment to particular research problems in JDM, not blindly to a single discipline that is part of JDM. He found a nice way of saying that it is necessary to consider how to integrate one’s point of view with that of other disciplines: “If you see any of these disciplines dominating you, you join the opposition and fight it for a while” (quoted by Reference Augier and MarchAugier & March, 2002, p. 2).