We estimate the effects of repeated exposure to the Cognitive Reflection Test (CRT) by examining 14,053 MTurk subjects who took the test up to 25 times. In contrast with inferences drawn from self-reported prior exposure to the CRT, we find that prior exposure usually fails to improve scores. On average, respondents get only 0.024 additional items correct per exposure, and this small increase is driven entirely by the minority of subjects who continue to spend time reflecting on the items. Moreover, later scores retain the predictive validity of earlier scores, even when they differ, because initial success and later improvement appear to measure the same thing.

The present study used the picture perception paradigm to examine the extent to which three well-documented psychophysiological measures demonstrate consistency across time in response to emotional stimuli. The three measures were the eye-blink startle response and the activation in two facial muscle regions (zygomatic and corrugator). Twenty-seven young women were assessed on two occasions, 2 weeks apart. Whereas activation in the corrugator and zygomatic muscle regions demonstrated the predicted patterns at both assessments (with some attenuation in the zygomatic muscle regions), the startle response had limited consistency across the two assessments. The startle response revealed the predicted linear pattern of valence modulation during the first assessment. During the second assessment, startle magnitude response was a quadratic function of valence ratings and a linear function of arousal ratings. The unexpected pattern of startle response during the second session appeared to be related to the content of the pleasant slides, with action slides generating quadratic valence modulation and erotic slides continuing to exhibit the expected linear valence modulation.