The Cognitive Reflection Test (CRT) is among the most common and well‐known instruments for measuring the propensity to engage reflective processing, in the context of the dual‐process theory of high‐level cognition. There is robust evidence that men perform better than women on this test—but we should be wary to conclude that men are more likely to engage in reflective processing than women. We consider several possible loci for the gender difference in CRT performance, and use mathematical modeling to show, across two studies, that the gender difference in CRT performance is more likely due to women making more mathematical mistakes (partially explained by their greater mathematics anxiety) than due to women being less likely to engage reflective processing. As a result, we argue that we need to use gender‐equivalent variants of the CRT, both to improve the quality of our instruments and to fulfill our social responsibility as scientists.