Campitelli G, & Gerrans P (2014). Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach. Memory & Cognition, 42, 434–447. doi: 10.3758/s13421-013-0367-9 [ PubMed] [ CrossRef] [ Google Scholar] Johnson ED, Tubau E, & De Neys W (2016). The Doubting System 1: Evidence for automatic substitution sensitivity. Acta Psychologica, 164, 56–64. doi: 10.1016/j.actpsy.2015.12.008 [ PubMed] [ CrossRef] [ Google Scholar] Gangemi A, Bourgeois-Gironde S, & Mancini F (2015). Feelings of error in reasoning—in search of a phenomenon. Thinking & Reasoning, 21, 383–396. doi: 10.1080/13546783.2014.980755 [ CrossRef] [ Google Scholar] B. T., & Stanovich KE (2013). Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8, 223–241. doi: 10.1177/1745691612460685 [ PubMed] [ CrossRef] [ Google Scholar] Koriat A, Lichtenstein S, & Fischhoff B (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6, 107–118. doi: 10.1037/0278-7393.6.2.107 [ CrossRef] [ Google Scholar]

Klayman J, Soll JB, González-Vallejo C, & Barlas S (1999). Overconfidence: It depends on how, what, and whom you ask. Organizational Behavior and Human Decision Processes, 79, 216–247. doi: 10.1006/obhd.1999.2847 [ PubMed] [ CrossRef] [ Google Scholar] Keren G (1988). On the ability of monitoring non-veridical perceptions and uncertain knowledge: Some calibration studies. Acta Psychologica, 67, 95–119. doi: 10.1016/0001-6918(88)90007-8 [ PubMed] [ CrossRef] [ Google Scholar] De Neys W, Cromheeke S, & Osman M (2011). Biased but in doubt: Conflict and decision confidence. PloS one, 6( 1), e15954. [ PMC free article] [ PubMed] [ Google Scholar] Newell BR, & Shanks DR (2014). Unconscious influences on decision making: A critical review. Behavioral and Brain Sciences, 37, 1–19. doi: 10.1017/S0140525X12003214 [ PubMed] [ CrossRef] [ Google Scholar]

Thousands of students from MIT, Harvard, and Princeton had been put through the quiz, and you’d think that anyone in these prestigious universities would be able to solve this problem with an unerring ease. Not so fast. It turned out that more than 50% responded with the knee-jerk—incorrect—answer. The two systems that led to the choices.

For the recall response, a mixed effects logistic regression (with subject as the random variable) was conducted due to the dichotomous dependent variable (with or without “more than”), considering only those participants who wrote down an answer that could be coded as with or without the “more than” phrase (e.g., “don’t know” responses were excluded). Incorrect reasoners usually recalled the standard problem, but not the control, as containing “more than” (see Table 1), with this effect of condition significant, b = 3.02, odds ratio ( OR) = 20.44, χ 2 = 21.69, p< .001, 95% confidence interval ( CI) [5.74, 72.75]. Finally, latency responses to both the standard and control versions were collected. Correlations between response time and response confidence were expected, in line with previous work (e.g., Johnson et al., 2016; Kelley & Lindsay, 1993; Thompson, Prowse Turner, & Pennycook, 2011; Thompson et al., 2013).

WE HAVE TACOS

Bourgeois-Gironde S, & Vanderhenst J-B (2009). How to open the door to System 2: Debiasing the Bat and Ball problem. In Watanabe S, Bloisdell AP, Huber L, & Young A (Eds.), Rational animals, irrational humans (pp. 235–252). Tokyo: Keio University Press. [ Google Scholar] For recognition, all incorrect reasoners’ responses were included in the analyses because their answers came in the form of a multiple-choice forced response. A mixed effects logistic regression (with subject as the random variable) was again conducted due to the dichotomous dependent variable (with or without “more than”). Once again, incorrect reasoners usually recognized the standard problem, but not the control, as containing “more than” (see Table 1), with this effect of condition significant, b = 3.00, odds ratio ( OR) = 20.07, χ 2 = 50.34, p< .001, 95% confidence interval ( CI) [8.76, 45.94]. Hoover JD, & Healy AF (2017). Algebraic reasoning and bat-and-ball problem variants: Solving isomorphic algebra first facilitates problem solving later. Psychonomic Bulletin & Review, 24, 1922–1928. doi: 10.3758/s13423-017-1241-8 [ PubMed] [ CrossRef] [ Google Scholar]

Lichtenstein S, Fischhoff B, & Phillips LD (1982). Calibration of probabilities: The state of the art to 1980. In Kahneman D, Slovic P, & Tversky A (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 306–334). Cambridge, UK: Cambridge University Press. [ Google Scholar] Israeli-American psychologist Daniel Kahneman examines what he calls the machinery of the mind — two distinct systems in our brain that dictate how we think and make decisions — in his book, Thinking Fast, and Slow.The body of work on conflict detection in decision-making provides the theoretical framework for explaining the findings outlined above (e.g., Aczel, Szollosi, & Bago, 2016; De Neys, 2012; Pennycook, Fugelsang, & Koehler, 2012). In this line of theorizing, reasoning errors arise from an inability to inhibit prepotent intuitive responses that, importantly, cannot be explained solely due to miserly cognition because incorrect reasoners demonstrate that they are unsure their reasoning was accurate. Indeed, there is a rich body of research in support of error sensitivity in particular and logical intuitions more generally (e.g., Bago & De Neys, 2017; De Neys, 2012, 2014; De Neys & Bonnefon, 2013; De Neys & Glumicic, 2008; De Neys et al., 2013; Gangemi, Bourgeois-Gironde, & Mancini, 2015; Mata, Schubert, & Ferreira, 2014; but see Singmann, Klauer, & Kellen, 2014, for some caveats regarding this literature). Furthermore, sensitivity to error processing has been demonstrated though converging evidence in latency response investigations ( De Neys & Glumicic, 2008; Frey, Johnson, & De Neys, 2017; Johnson, Tubau, & De Neys, 2016) neuroimaging ( De Neys, Vartanian, & Goel, 2008), and alternative measurements of confidence ( De Neys, Cromheeke, & Osman, 2011). However, sensitivity effects were not obtained by indexing reasoner’s mouse-movements ( Travers, Rolison, & Feeney, 2016) nor their eye-movements ( Mata, Ferreira, Voss, & Kollei, 2017; see the subsequent debate concerning the studies by Mata et al. and Frey et al. discussed by Mata & Ferreira, 2018). On a more general level, unconscious processes have not been found to have much (if any) explanatory power (for a review, see Newell & Shanks, 2014). For the second and third samples, nine simple math problems were included between the experimental questions and the memory questions to serve as interference to limit recall and recognition based on working memory (see the Supplemental Materials). Neither math problems nor any other activity occurred between the De Neys et al. (2013) experimental and the memory questions for the first sample. Specifically, we started by testing 126 MTurk participants and then examined their data. We did not have a precise stopping rule for the sample size, but we decided from the outset to pause data collection after examining the data from an initial sample of MTurk participants. We noted poor memory performance by these initial participants. On the basis of our observations, we tested two additional samples of participants (one from MTurk and one from UCB), each approximately the same size as the initial sample (128 participants), and gave them the math problems to create interference. Fischhoff B, Slovic P, & Lichtenstein S (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3, 552–564. doi: 10.1037/0096-1523.3.4.552 [ CrossRef] [ Google Scholar]

Keren G (1997). On the calibration of probability judgments: Some critical comments and alternative perspectives. Journal of Behavioral Decision Making, 10, 269–278. doi: 10.1002/(SICI)1099-0771(199709)10:3<269::AID-BDM281>3.0.CO;2-L [ CrossRef] [ Google Scholar] Sinayev A, & Peters E (2015). Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 6, 532. doi: 10.3389/fpsyg.2015.00532 [ PMC free article] [ PubMed] [ CrossRef] [ Google Scholar] De Neys W (2012). Bias and conflict: A case for logical intuitions. Perspectives on Psychological Science, 7, 28–38. doi: 10.1177/1745691611429354 [ PubMed] [ CrossRef] [ Google Scholar] Faul F, Erdfelder E, Lang A-G, & Buchner A (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. doi: 10.3758/BF03193146 [ PubMed] [ CrossRef] [ Google Scholar] Lichtenstein S, & Fischhoff B (1980). Training for calibration. Organizational Behavior and Human Performance, 26, 149–171. doi: 10.1016/0030-5073(80)90052-5 [ CrossRef] [ Google Scholar]Thompson VA, Prowse Turner JA, & Pennycook G (2011). Intuition, reason, and metacognition. Cognitive Psychology, 63, 107–140. doi: 10.1016/j.cogpsych.2011.06.001 [ PubMed] [ CrossRef] [ Google Scholar] As for the response confidence scores, the opinion judgments are reported here as proportions rather than percentages. There was a positive relationship between standard question confidence and standard question opinion judgments for incorrect reasoners, r(225) = .480, p< .001, such that incorrect reasoners who were less confident in their response were also less likely to think other reasoners could answer the standard question correctly. This strong relationship lends support to the notion that opinion judgments and response confidence scores are reflecting similar cognitive processes.