The Cognitive Reflection Test (CRT) allegedly measures the tendency to override the prepotent incorrect answers to some special problems, and to engage in further reflection. A growing literature suggests that the CRT is a powerful predictor of performance in a wide range of tasks. This research has mostly glossed over the fact that the CRT is composed of math problems. The purpose of this paper is to investigate whether numerical CRT items do indeed call upon more than is required by standard math problems, and whether the latter predict performance in other tasks as well as the CRT. In Study 1 we selected from a bank of standard math problems items that, like CRT items, have a fast lure, as well as others which do not. A 1-factor model was the best supported measurement model for the underlying abilities required by all three item types. Moreover, the quality of all these items – CRT and math problems alike – as predictors of performance on a set of choice and reasoning tasks did not depend on whether or not they had a fast lure, but rather only on their quality as math items. In other words, CRT items seem not to be a “special” category of math problems, although they are quite excellent ones. Study 2 replicated these results with a different population and a different set of math problems.

Research on judgment and decision making has suggested that the System 2 process of slow thinking can help people to improve their decision making by reducing well-established statistical decision biases (including base rate neglect, probability matching, and the conjunction fallacy). In a large pre-registered study with 1,706 participants and 23,292 unique observations, we compare the effects of individual differences and behavioral interventions to test the relative benefits of slow thinking on performance in canonical judgment and decision-making problems, compared to a control condition, a fast thinking condition, an incentive condition, and a condition that combines fast and slow thinking. We also draw on the rule-based reasoning literature to examine the benefits of having access to a simple form of the rule needed to solve a specific focal problem. Overall, we find equivocal evidence of a small benefit from slow thinking, evidence for a small benefit to accuracy incentives, and clear evidence of a larger cost from fast thinking. The difference in performance between fast-thinking and slow-thinking interventions is comparable to a one-scale point difference on the 4-point Cognitive Reflection Test (CRT). Inferential rules contribute unique explanatory power and interact with individual differences to support the idea that System 2 benefits from a combination of slower processes and knowledge appropriate to the problem.