In this introduction to the special issue on methodology, we provide background on its original motivation and a systematic overview of the contributions. The latter are discussed with correspondence to the phase of the scientific process they (most strongly) refer to: Theory construction, design, data analysis, and cumulative development of scientific knowledge. Several contributions propose novel measurement techniques and paradigms that will allow for new insights and can thus avail researchers in JDM and beyond. Another set of contributions centers around how models can best be tested and/or compared. Especially when viewed in combination, the papers on this topic spell out vital necessities for model comparisons and provide approaches that solve noteworthy problems prior work has been faced with.

According to Karl Popper, we can tell good theories from poor ones by assessing their empirical content (empirischer Gehalt), which basically reflects how much information they convey concerning the world. “The empirical content of a statement increases with its degree of falsifiability: the more a statement forbids, the more it says about the world of experience.” Two criteria to evaluate the empirical content of a theory are their level of universality (Allgemeinheit) and their degree of precision (Bestimmtheit). The former specifies how many situations it can be applied to. The latter refers to the specificity in prediction, that is, how many subclasses of realizations it allows. We conduct an analysis of the empirical content of theories in Judgment and Decision Making (JDM) and identify the challenges in theory formulation for different classes of models. Elaborating on classic Popperian ideas, we suggest some guidelines for publication of theoretical work.

How many judgment and decision making (JDM) researchers have not claimed to be building on Herbert Simon’s work? We identify two of Simon’s goals for JDM research: He sought to understand people’s decision processes—the descriptive goal—and studied whether the same processes lead to good decisions—the prescriptive goal. To investigate how recent JDM research relates to these goals, we analyzed the articles published in the Journal of Behavioral Decision Making and in Judgment and Decision Making from 2006 to 2010. Out of 377 articles, 91 cite Simon or we judged them as directly relating to his goals. We asked whether these articles are integrative, in the following sense: For a descriptive article we asked if it contributes to building a theory that reconciles different conceptualizations of cognition such as neural networks and heuristics. For a prescriptive article we asked if it contributes to building a method that combines ideas of other methods such as heuristics and optimization models. Based on our subjective judgments we found that the proportion of integrative articles was 67% of the prescriptive and 52% of the descriptive articles. We offer suggestions for achieving more integration of JDM theories. The article concludes with the thesis that although JDM researchers work under Simon’s spell, no one really knows what that spell is.

The aim of this article is to evaluate the contribution of process tracing data to the development and testing of models of judgment and decision making (JDM). We draw on our experience of editing the “Handbook of process tracing methods for decision research” recently published in the SJDM series. After a brief introduction we first describe classic process tracing methods (thinking aloud, Mouselab, eye-tracking). Then we present a series of examples of how each of these techniques has made important contributions to the development and testing of process models of JDM. We discuss the issue of large data volumes resulting from process tracing and remedies for handling those. Finally, we argue for the importance of formulating process hypotheses and opt for a multi-method approach that focuses on the cross-validation of findings.

Currently, a disparity exists between the process-level models decision researchers use to describe and predict decision behavior and the methods implemented and metrics collected to test these models. The current work seeks to remedy this disparity by combining the advantages of work in decision research (mouse-tracing paradigms with contingent information display) and cognitive psychology (eye-tracking paradigms from reading and scene perception). In particular, we introduce a new decision moving-window paradigm that presents stimulus information contingent on eye fixations. We provide data from the first application of this method to risky decision making, and show how it compares to basic eye-tracking and mouse-tracing methods. We also enumerate the practical, theoretical, and analytic advantages this method offers above and beyond both mouse-tracing with occlusion and basic eye tracking of information without occlusion. We include the use of new metrics that offer more precision than those typically calculated on mouse-tracing data as well as those not possible or feasible within the mouse-tracing paradigm.

The history of judgment and decision making is defined by a trend toward increasingly nuanced explanations of the decision making process. Recently, process models have become incredibly sophisticated, yet the tools available to directly test these models have not kept pace. These increasingly complex process models require increasingly complex process data by which they can be adequately tested. We propose a new class of data collection that will facilitate evaluation of sophisticated process models. Tracking mouse paths during a continuous response provides an implicit measure of the growth of preference that produces a choice—rather than the current practice of recording just the button press that indicates that choice itself. Recent research in cognitive science (Spivey & Dale, 2006) has shown that cognitive processing can be revealed in these dynamic motor responses. Unlike current process methodologies, these response dynamics studies can demonstrate continuous competition between choice options and even online preference reversals. Here, in order to demonstrate the mechanics and utility of the methodology, we present an example response dynamics experiment utilizing a common multi-alternative decision task.

The two most influential models in delay discounting research have been the exponential (E) and hyperbolic (H) models. We develop a new methodology to design binary choice questions such that exponential and hyperbolic discount rates can be purposefully manipulated to make their rate parameters orthogonal (Pearson’s R = 0), negatively correlated (R = –1), positively correlated (R = +1), or to hold one rate constant while allowing the other to vary. Then we extend the method to similarly contrast different versions of the hyperboloid model. The arithmetic discounting model (A), which is based on differences between present and future rewards rather than their ratios, may easily be made orthogonal to any other pair of models. Our procedure makes it possible to design choice stimuli that precisely vary the relationship between different discount rates. However, the additional control over the correlation between different discount rate parameters may require the researcher to either restrict the range that those rate parameters can take, or to expand the range of times the participant must wait for future rewards.

Narrow self-interest is often used as a simplifying assumption when studying people making decisions in social contexts. Nonetheless, people exhibit a wide range of different motivations when choosing unilaterally among interdependent outcomes. Measuring the magnitude of the concern people have for others, sometimes called Social Value Orientation (SVO), has been an interest of many social scientists for decades and several different measurement methods have been developed so far. Here we introduce a new measure of SVO that has several advantages over existent methods. A detailed description of the new measurement method is presented, along with norming data that provide evidence of its solid psychometric properties. We conclude with a brief discussion of the research streams that would benefit from a more sensitive and higher resolution measure of SVO, and extend an invitation to others to use this new measure which is freely available.

One major statistical and methodological challenge in Judgment and Decision Making research is the reliable identification of individual decision strategies by selection of diagnostic tasks, that is, tasks for which predictions of the strategies differ sufficiently. The more strategies are considered, and the larger the number of dependent measures simultaneously taken into account in strategy classification (e.g., choices, decision time, confidence ratings; Glöckner, 2009), the more complex the selection of the most diagnostic tasks becomes. We suggest the Euclidian Diagnostic Task Selection (EDTS) method as a standardized solution for the problem. According to EDTS, experimental tasks are selected that maximize the average difference between strategy predictions for any multidimensional prediction space. In a comprehensive model recovery simulation, we evaluate and quantify the influence of diagnostic task selection on identification rates in strategy classification. Strategy classification with EDTS shows superior performance in comparison to less diagnostic task selection algorithms such as representative sampling. The advantage of EDTS is particularly large if only few dependent measures are considered. We also provide an easy-to-use function in the free software package R that allows generating predictions for the most commonly considered strategies for a specified set of tasks and evaluating the diagnosticity of those tasks via EDTS; thus, to apply EDTS, no prior programming knowledge is necessary.

We present a classification methodology that jointly assigns to a decision maker a best-fitting decision strategy for a set of choice data as well as a best-fitting stochastic specification of that decision strategy. Our methodology utilizes normalized maximum likelihood as a model selection criterion to compare multiple, possibly non-nested, stochastic specifications of candidate strategies. In addition to single strategy with “error” stochastic specifications, we consider mixture specifications, i.e., strategies comprised of a probability distribution over multiple strategies. In this way, our approach generalizes the classification framework of Bröder and Schiffer (2003a). We apply our methodology to an existing dataset and find that some decision makers are best fit by a single strategy with varying levels of error, while others are best described as using a mixture specification over multiple strategies.

Taking a falsificationist perspective, the present paper identifies two major shortcomings of existing approaches to comparative model evaluations in general and strategy classifications in particular. These are (1) failure to consider systematic error and (2) neglect of global model fit. Using adherence measures to evaluate competing models implicitly makes the unrealistic assumption that the error associated with the model predictions is entirely random. By means of simple schematic examples, we show that failure to discriminate between systematic and random error seriously undermines this approach to model evaluation. Second, approaches that treat random versus systematic error appropriately usually rely on relative model fit to infer which model or strategy most likely generated the data. However, the model comparatively yielding the best fit may still be invalid. We demonstrate that taking for granted the vital requirement that a model by itself should adequately describe the data can easily lead to flawed conclusions. Thus, prior to considering the relative discrepancy of competing models, it is necessary to assess their absolute fit and thus, again, attempt falsification. Finally, the scientific value of model fit is discussed from a broader perspective.

Often research in judgment and decision making requires comparison of multiple competing models. Researchers invoke global measures such as the rate of correct predictions or the sum of squared (or absolute) deviations of the various models as part of this evaluation process. Reliance on such measures hides the (often very high) level of agreement between the predictions of the various models and does not highlight properly the relative performance of the competing models in those critical cases where they make distinct predictions. To address this important problem we propose the use of pair-wise comparisons of models to produce more informative and targeted comparisons of their performance, and we illustrate this procedure with data from two recently published papers. We use Multidimensional Scaling of these comparisons to map the competing models. We also demonstrate how intransitive cycles of pair-wise model performance can signal that certain models perform better for a given subset of decision problems.

Hierarchical Bayesian methods offer a principled and comprehensive way to relate psychological models to data. Here we use them to model the patterns of information search, stopping and deciding in a simulated binary comparison judgment task. The simulation involves 20 subjects making 100 forced choice comparisons about the relative magnitudes of two objects (which of two German cities has more inhabitants). Two worked-examples show how hierarchical models can be developed to account for and explain the diversity of both search and stopping rules seen across the simulated individuals. We discuss how the results provide insight into current debates in the literature on heuristic decision making and argue that they demonstrate the power and flexibility of hierarchical Bayesian methods in modeling human decision-making.

Judgment and decision making research overwhelmingly uses null hypothesis significance testing as the basis for statistical inference. This article examines an alternative, Bayesian approach which emphasizes the choice between two competing hypotheses and quantifies the balance of evidence provided by the data—one consequence of which is that experimental results may be taken to strongly favour the null hypothesis. We apply a recently-developed “Bayesian t-test” to existing studies of the anchoring effect in judgment, and examine how the change in approach affects both the tone of hypothesis testing and the substantive conclusions that one draws. We compare the Bayesian approach with Fisherian and Neyman-Pearson testing, examining its relationship to conventional p-values, the influence of effect size, and the importance of prior beliefs about the likely state of nature. The results give a sense of how Bayesian hypothesis testing might be applied to judgment and decision making research, and of both the advantages and challenges that a shift to this approach would entail.

The calibration of probability or confidence judgments concerns the association between the judgments and some estimate of the correct probabilities of events. Researchers rely on estimates using relative frequencies computed by aggregating data over observations. We show that this approach creates conceptual problems, and may result in the confounding of explanatory variables or unstable estimates. To circumvent these problems we propose using probability estimates obtained from statistical models—specifically mixed models for binary data—in the analysis of calibration. We illustrate this methodology by re-analyzing data from a published study and comparing the results from this approach to those based on relative frequencies. The model-based estimates avoid problems with confounding variables and provided more precise estimates, resulting in better inferences.

It is a long known problem that the preferential publication of statistically significant results (publication bias) may lead to incorrect estimates of the true effects being investigated. Even though other research areas (e.g., medicine, biology) are aware of the problem, and have identified strong publication biases, researchers in judgment and decision making (JDM) largely ignore it. We reanalyzed two current meta-analyses in this area. Both showed evidence of publication biases that may have led to a substantial overestimation of the true effects they investigated. A review of additional JDM meta-analyses shows that most meta-analyses conducted no or insufficient analyses of publication bias. However, given our results and the rareness of non-significant effects in the literature, we suspect that biases occur quite often. These findings suggest that (a) conclusions based on meta-analyses without reported tests of publication bias should be interpreted with caution and (b) publication policies and standard research practices should be revised to overcome the problem.