Interval estimates are commonly used to descriptively communicate the degree of uncertainty in numerical values. Conventionally, low alpha levels (e.g., .05) ensure a high probability of capturing the target value between interval endpoints. Here, we test whether alpha levels and individual differences in numeracy influence distributional inferences. In the reported experiment, participants received prediction intervals for fictitious towns’ annual rainfall totals (assuming approximately normal distributions). Then, participants estimated probabilities that future totals would be captured within varying margins about the mean, indicating the approximate shapes of their inferred probability distributions. Results showed that low alpha levels (vs. moderate levels; e.g., .25) more frequently led to inferences of over-dispersed approximately normal distributions or approximately uniform distributions, reducing estimate accuracy. Highly numerate participants made more accurate estimates overall, but were more prone to inferring approximately uniform distributions. These findings have important implications for presenting interval estimates to various audiences.

The Ratio-Bias phenomenon, observed by psychologist Seymour Epstein and colleagues, is a systematic manifestation of irrationality. When offered a choice between two lotteries, individuals consistently choose the lottery with the greater number of potential successes, even when it offers a smaller probability of success. In the current study, we conduct experiments to confirm this phenomenon and test for the existence of Bias as distinct from general irrationality. Moreover, we examine the effect of introducing a monetary incentive of varying size (depending on the treatment) on the extent of irrational choices within this framework. We confirm the existence of the Bias. Moreover, the existence of an incentive significantly reduces the extent of irrationality exhibited, and that this effect is roughly linear in response to changes in the size of the incentive within the magnitudes investigated.

The ratio-bias (RB) phenomenon is considered to provide systematic evidence of irrationality. When judging the probability of a low-probability event, many people judge it as less likely when it is expressed as a ratio of small numbers (e.g., 1-in-10) than when it is expressed as a ratio of large numbers (e.g., 10-in-100). Four experiments show that the phenomenon is increased by the experimental paradigm, which misleads subjects regarding the aim of the task by inducing equal-ratio neglect. One factor is constant across the texts of the Experiment 1–3: a particular sentence that induces subjects to neglect the equal ratio and invites them to express feelings about the outcome of the target event rather than giving a rational answer. This intent is strengthened by the formulation of the question (Experiment 1), which explicitly asks the subject to express the feeling connected to the lotteries and the absence of a third option (Experiment 1, 4), the right one, which expresses the “indifference” between the two options. In Experiment 4, the task lacks only the third option, and, simply by adding the option that allows subjects to express the correct answer, the RB disappears.

Although the Analysis of Competing Hypotheses method (ACH) is a structured analytic technique promoted in several intelligence communities for improving the quality of probabilistic hypothesis testing, it has received little empirical testing. Whereas previous evaluations have used numerical evidence assumed to be perfectly accurate, in the present experiment we tested the effectiveness of ACH using a judgment task that presented participants with uncertain evidence varying in source reliability and information credibility. Participants (N = 227) assigned probabilities to two alternative hypotheses across six cases that systematically varied case features. Across multiple tests of coherence, the ACH group showed no advantage over a no-technique control group. Both groups showed evidence of subadditivity, unreliability, and overly conservative non-Bayesian judgments. The ACH group also showed pseudo-diagnostic weighting of evidence. The findings do not support the claim that ACH is effective at improving probabilistic judgment.

Previous studies have found that the proportions of people who endorsed utilitarian decisions varied across different variants of the trolley dilemma. In this paper, we explored whether moral choices were associated with beliefs about outcome probabilities in different moral dilemmas. Results of two experiments showed that participants’ perceptions of outcome probabilities were different between two dilemmas that were similar to the classical switch case and footbridge case. Participants’ judgments of the outcome probabilities were associated with their moral choices. The results suggested that participants might not accept task instructions and thus did not perceive the outcomes in the dilemmas as certain. We argued that researchers who endorse descriptive tasks in moral reasoning research should be cautious about the findings and should take participants’ beliefs in the outcomes into account.

The gambler’s fallacy (Tune, 1964) refers to the belief that a streak is more likely to end than chance would dictate. In three studies, participants exhibited a retrospective gambler’s fallacy (RGF) in which an event that seems rare appears to come from a longer sequence than an event that seems more common. Study 1 demonstrates this bias for streaks, while Study 2 does so with single rare events and shows that the appearance of rarity is more important than actual rarity. Study 3 extends these findings from abstract gambling domains into real world domains to demonstrate the generalizability of the effects. The RGF follows from the law of small numbers (Tversky & Kahneman, 1971) and has many applications, from perceptions of the social world to philosophical debates about the existence of multiple universes.

Brier score decompositions, including those attributed to Murphy and to Yates, provide popular metrics for estimating forecast performance attributes like calibration and discrimination. However, the decompositions are generally limited to situations where forecasters make successive forecast judgments against the same class of substantive event (e.g., rain vs. no rain). They do not readily translate to common situations where: forecasts are weighted unequally; forecasts can be made against a range of heterogeneous topics and events over varying time horizons; forecasts can be updated over time until an event occurs or an event deadline is reached; or outcome alternatives can vary in number and nature (e.g., ordered vs. unordered outcomes) across forecast questions. In this paper, we propose extensions of the Murphy and Yates decompositions to address these features. The extensions involve new analytic expressions for the decompositions of weighted Brier scores, along with proposed resampling methods. We use data from a recent forecasting tournament to illustrate the methods.

Humans are evidently able to learn contingencies from the co-occurrence of cuesand outcomes. But how do humans judge contingencies when observations of cue andoutcome are learned on different occasions? The pseudocontingency frameworkproposes that humans rely on base-rate correlations across contexts, that is,whether outcome base rates increase or decrease with cue base rates. Here, weelaborate on an alternative mechanism for pseudocontingencies that exploits baserate information within contexts. In two experiments, cue and outcome base ratesvaried across four contexts, but the correlation by base rates was kept constantat zero. In some contexts, cue and outcome base rates were aligned (e.g., cueand outcome base rates were both high). In other contexts, cue and outcome baserates were misaligned (e.g., cue base rate was high, but outcome base rate waslow). Judged contingencies were more positive for contexts in which cue andoutcome base rates were aligned than in contexts in which cue and outcome baserates were misaligned. Our findings indicate that people use the alignment ofbase rates to infer contingencies conditional on the context. As such, they lendsupport to the pseudocontingency framework, which predicts that decision makersrely on base rates to approximate contingencies. However, they challengeprevious conceptions of pseudocontingencies as a uniform inference fromcorrelated base rates. Instead, they suggest that people possess a repertoire ofmultiple contingency inferences that differ with regard to informationalrequirements and areas of applicability.

Five experiments addressed a controversy in the probability judgment literature that centers on the efficacy of framing probabilities as frequencies. The natural frequency view predicts that frequency formats attenuate errors, while the nested-sets view predicts that highlighting the set-subset structure of the problem reduces error, regardless of problem format. This study tested these predictions using a conjunction task. Previous studies reporting that frequency formats reduced conjunction errors confounded reference class with problem format. After controlling this confound, the present study’s findings show that conjunction errors can be reduced using either a probability or a frequency format, that frequency effects depend upon the presence of a reference class, and that frequency formats do not promote better statistical reasoning than probability formats.

Despite vigorous research on risk communication, little is known about the social forces that drive these choices. Erev, Wallsten, & Neal (1991) showed that forecasters learn to select verbal or numerical probability estimates as a function of which mode yields on average the larger group payoffs. We extend the result by investigating the effect of group size on the speed with which forecasters converge on the better communication mode. On the basis of social facilitation theory we hypothesized that small groups induce less arousal and anxiety among their members than do large groups when performing new tasks, and therefore that forecasters in small groups will learn the better communication mode more quickly. This result obtained in Experiment 1, which compared groups of size 3 to groups of size 5 or 6. To test whether social loafing rather than social facilitation was mediating the effects, Experiment 2 compared social to personal feedback holding group size constant at 3 members. Learning was faster in the personal feedback condition, suggesting that social facilitation rather than loafing underlay the results.

This paper summarizes an empirical comparison of the accuracy of forecasts included in analysis reports developed by professional intelligence analysts to comparable forecasts in a prediction market that has broad participation from across an intelligence community. To compare forecast accuracy, 99 event forecasts were extracted from qualitative descriptions found in 41 analysis reports and posted on the prediction market. Quantitative probabilities were imputed from the qualitative forecasts by asking seasoned professional analysts, who did not participate in the prediction market, to read the reports and to infer a quantitative probability based on what was written. These readers were also asked to provide their personal probabilities before and after reading the reports. There were two statistically significant results of particular interest. First, the primary result is that the prediction market forecasts were more accurate than the analysis reports. On average prediction market probabilities were 0.114 closer to ground truth than the analysis report probabilities. Second, in cases where analysts (readers) updated their personal probabilities in a direction opposite to what the reports implied, analysts tended to update their probabilities in the correct direction. This occurred even though, on average, reading the reports did not make readers more accurate.

A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments.

The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW”) to improve probability judgments in a four-year series of geopolitical forecasting tournaments sponsored by the U.S. intelligence community. Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition. Cognitive ability and practice also made largely independent contributions to predictive accuracy. Given the brevity of the training tutorials and the heterogeneity of the problems posed, the observed effects are likely to be lower-bound estimates of what could be achieved by more intensive interventions. Future work should isolate which prongs of the multipronged CHAMPS KNOW training were most effective in improving judgment on which categories of problems.

The group-diffusion effect is the tendency for people to judge themselves to be less likely to experience a negative outcome as the total number of people exposed to the threat increases — even when the probability of the outcome is explicitly presented (Yamaguchi, 1998). In Experiment 1 we replicated this effect for two health threat scenarios using a variant of Yamaguchi’s original experimental paradigm. In Experiment 2, we showed that people also judge themselves to be less likely to be selected in a lottery as the number of people playing the lottery increases. In Experiment 3, we showed that explicitly presenting the number of people expected to be selected eliminates the group-diffusion effect, and in Experiment 4 we showed that presenting the number expected to be affected by a health threat without presenting the total number exposed to the threat produces a reverse effect. We propose, therefore, that the group-diffusion effect is related to the ratio bias. Both effects occur when people make risk or likelihood judgments based on information presented as a ratio. The difference is that the group-diffusion effect occurs when the denominator of the relevant ratio is more salient than the numerator, while the ratio bias occurs when the numerator is more salient than the denominator.

The representativeness heuristic suggests that similarity judgments provide a basis for judgments of likelihood. We use Tversky’s (1977) contrast model of similarity to design tests of this underlying mechanism. If similarity is used to judge likelihood, factors that are known to affect similarity should also affect judgments of likelihood. In two experiments, we manipulated two such factors described in the contrast model of similarity: the nature of the task and context effects. In a between-subject design, respondents assessed either similarity of fictive citizens of 15th century Florence, or the likelihood that they belonged to the same family. The factors that affected similarity also affected the likelihood judgments. These results support the assumption that similarity is an important contributor to judgments of likelihood.

The phenomenon of base-rate neglect has elicited much debate. One arena of debate concerns how people make judgments under conditions of uncertainty. Another more controversial arena concerns human rationality. In this target article, we attempt to unpack the perspectives in the literature on both kinds of issues and evaluate their ability to explain existing data and their conceptual coherence. From this evaluation we conclude that the best account of the data should be framed in terms of a dual-process model of judgment, which attributes base-rate neglect to associative judgment strategies that fail to adequately represent the set structure of the problem. Base-rate neglect is reduced when problems are presented in a format that affords accurate representation in terms of nested sets of individuals.