Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test

The cognitive reflection test (CRT) became popular for its impressive power to predict how well people reason and make decisions. Despite the popularity of the CRT, a major issue complicates its interpretation: The numerical nature of the CRT confounds reflection ability with mathematical ability. We have addressed this issue by developing the verbal CRT (CRT‐V), a novel 10‐item measure of cognitive reflection (https://osf.io/xehbv/), using nonmathematical problems with good statistical and psychometric properties and with low familiarity. First, we selected suitable items with relatively low familiarity and optimal difficulty as identified in two different populations (Studies 1 and 2) and with high content validity as judged by an expert panel (Study 3). Second, we demonstrated good criterion and construct validity for the test in different populations with a wide range of variables (Studies 4–6, 8) and a good internal consistency (Studies 4–8) and test–retest reliability (Study 7). The CRT‐V was less associated with math anxiety, objective and subjective numeracy than the original CRT, and it was test equivalent across gender, age groups and administration setting. In contrast with the original CRT (Hedge's g = 0.29, 95% confidence interval [CI] [0.17, 0.40]), the CRT‐V showed no gender differences (Hedge's g = −0.06, 95% CI [−0.18, 0.06]). The CRT‐V can complement existing, numerical, tests of cognitive reflection.

This study investigates the cognitive processes underlying chess expertise by examining planning, cognitive reflection, inter-temporal choice, and risky choice in chess players. The study involves 25 chess players and 25 non-chess players, comparing their performance on the Tower of London (TOL) task, Cognitive Reflection Test (CRT), inter-temporal choice (ITC), and risky choice tasks. Results indicate that chess players outperform non-chess players in TOL and CRT, showing superior planning and cognitive reflection abilities. Chess players also prefer future rewards over immediate ones in ITC, suggesting a higher propensity for future more rewarding options. In risky choice tasks, chess players made more decisions based on expected value than non-chess players, but the evidence in favour of differences between groups is very weak. Despite this study not being able to establish causality, the findings highlight the cognitive advantages associated with chess expertise and suggest potential areas for further research on the transfer of cognitive skills from chess to other domains and differences in general abilities between experts and novices.

IntroductionCognitive reflection is the ability and disposition to reflect on one's own thinking, allowing a person to identify and correct judgments grounded in intuition rather than logic. Cognitive reflection strongly predicts school-aged children's understanding of counterintuitive science concepts. Here, we asked whether children's cognitive reflection similarly predicts a domain-general scientific skill: the interpretation of covariation data.MethodFive- to 12-year-olds (N = 74) completed a children's Cognitive Reflection Test (CRT-D) and measures of executive functioning. They also interpreted covariation data presented in 2 x 2 contingency tables.Results and discussionCRT-D performance predicted children's overall accuracy and the strategies they used to evaluate the contingency tables, even after adjusting for their age, set-shifting ability, inhibitory control, and working memory. Thus, the relationship between cognitive reflection and statistical reasoning emerges early in development. These findings suggest cognitive reflection is broadly involved in children's scientific thinking, supporting domain-general data-interpretation skills in addition to domain-specific conceptual knowledge.

Abstract. The present paper features the adaptation of an existing Big Five questionnaire with a rating scale (RS) response format into a measure using a multidimensional forced choice (MFC) response format. Rating scale response formats have been criticized for their proneness to intentional and unintentional response distortions. Multidimensional forced choice response formats were suggested as a solution to mitigate several types of response sets and response styles by design. The Big Five Inventory of Personality in Occupational Situations (B5PS) is a situation-based questionnaire designed for personnel selection and development purposes which would benefit from fake-proof response formats. MFC response formats require special effort during test construction and calibration which will be laid out here. Changing the response format has severe consequences on item design and scoring. An inherent issue with MFC formats derives from their inability to yield interpersonal comparative results from standard (sum) scoring. This issue can be solved with item response theory (IRT)-based calibration during test construction. The Thurstonian IRT approach (TIRT) was developed by Brown and Maydeu-Olivares (2011) , and aspects of MFC item design and TIRT calibrations are explored in this paper. Evidence on structural and construct validity are presented alongside recommendations on the test development processes. The results support the feasibility of the concept of MFC test construction with TIRT calibration in a contextualized and situation-based item format.

Lack of Deliberation Drives Honesty Among Men But Not Women

The dictator game (DG) is one of the simplest and most commonly used experimental games for examining economic and altruistic behaviors. Altruism became crucial in research on decisions in experimental and behavioral economy. This study extends existing research on the relationship between cognitive performance, measured by the Cognitive Reflection Test (CRT) and the DG, taking into account such factors as gender and declared religiosity. The research assesses whether gender and declared religiosity affect altruistic behavior measured in the DG’s payoffs. 511 participants (master’s students in economics) were asked to respond to six types of DGs and the CRT7 test. Correlation analysis, descriptive statistics, student t-tests, the Mann-Whitney test and Tobit regression analysis were conducted. Cognitive reflection was positively correlated with rational (selfish) behavior in the DG. Those dictators who scored high on the CRT (reflective dictators) kept more money for themselves than those who achieved lower scores on the CRT (altruistic, impulsive dictators). The results confirmed a distinct, inequity aversion attitude among altruistic, impulsive dictators and a selfish attitude among reflective dictators. The dictator’s payoff was significantly related to the gender and declared religiosity of the participants. Women were more concerned about equal distribution of income than men (on average they shared 30% more than men) and religious agents shared 20%–30% more than non-believers.

Laboratory practical exams are a component of knowledge-based assessment for first-year medical students in our Patient-Centered Learning (PCL) curriculum and have historically been constructed in a selected response (multiple choice) format. This study was conducted in order to address influences that question stimulus and/or response formats may have on assessment of student learning in a teaching laboratory setting. A neuroscience laboratory practical exam, previously included as an actual part of PCL assessment, was offered as a pre-exam, self-assessment exercise to two classes of students who were completing the Year 1 neuroscience curriculum. Randomized groups of students received the exercise in either selected-response or constructed-response (fill-in-the-blank) formats and student performance results were compared. Students performed significantly better on the exercise constructed in selected-response format. Comparisons of individual item difficulty and discrimination indices indicated that while asking a student to reflect on multiple options and come to an informed decision about the answer they choose is valid and reliable, “cueing” based on recognition and recall rather than depth of knowledge and understanding can explain student performance differences, raising legitimate issues for continuous improvement in curriculum and assessment. Supported by UNDSMHS.

Repeated testing during learning often improves later memory, which is often referred to as the testing effect. To clarify its boundary conditions, we examined whether the testing effect was selectively affected by covert (retrieved but not articulated) or overt (retrieved and articulated) response format. In Experiments 1 and 2, we compared immediate (5 min) and delayed (1 week) cued recall for paired associates following study-only, covert, and overt conditions, including two types of overt articulation (typing and writing). A clear testing effect was observed in both experiments, but with no selective effects of response format. In Experiments 3 and 4, we compared covert and overt retrieval under blocked and random list orders. The effect sizes were small in both experiments, but there was a significant effect of response format, with overt retrieval showing better final recall performance than covert retrieval. There were no significant effects of blocked vs. random list orders with respect to the testing effect produced. Taken together, these findings suggest that, under specific circumstances, overt retrieval may lead to a greater testing effect than that of covert retrieval, but because of small effect sizes, it appears that the testing effect is mainly the result of retrieval processes and that articulation has fairly little to add to its magnitude in a paired-associates learning paradigm.

The Cognitive Reflection Test (CRT) is a measure of analytical reasoning that cues an intuitive but incorrect response that must be rejected for successful performance to be attained. The CRT yields two types of errors: Intuitive errors, which are attributed to Type 1 processes; and non-intuitive errors, which result from poor numeracy skills or deficient reasoning. Past research shows that participants who commit the highest numbers of errors on the CRT overestimate their performance the most, whereas those with the lowest error-rates tend to slightly underestimate. This is an example of the Dunning-Kruger Effect (DKE). The present study examined how intuitive vs. non-intuitive errors contribute to overestimation in the CRT at different levels of performance. Female undergraduate students completed a seven-item CRT test and subsequently estimated their raw score. They also filled out the Faith in Intuition (FI) questionnaire, which is a dispositional measure of intuitive thinking. Data was separated into quartiles based on level of performance on the CRT. The results demonstrated the DKE. Additionally, intuitive and non-intuitive errors predicted miscalibration among low, but not high performers. However, intuitive errors were a stronger predictor of miscalibration. Finally, FI was positively correlated with CRT self-estimates and miscalibration, indicating that participants who perceived themselves to be more intuitive were worse at estimating their score. These results taken together suggest that participants who perform poorly in the CRT and also those who score higher in intuitive thinking disposition are more susceptible to the influences of heuristic-based cues, such as answer fluency, when judging their performance.

This article reports on the updating and validation of a questionnaire for vocabulary learning strategies. An English as a second language (ESL) version of the Vocabulary Learning Questionnaire (VLQ), first published in Gu and Johnson (1996), was created based on its latest version. Two response formats were piloted, that is, a paper version with a 7-point Likert scale and an online version with a 100-point slider bar. A series of validation procedures resulted in a 62-item instrument which was then administered online among 682 English language learners at the tertiary level in China. The paper presents evidence of content validity, construct validity, internal consistency reliability, and predictive validity. It also provides suggestions for interpreting and using the questionnaire for both research and instructional purposes.

Students with disabilities are frequently granted accommodations for high-stakes standardized tests to provide them an opportunity to demonstrate their academic knowledge without interference from their disability. One type of possible accommodation, test response format, concerns whether students respond in multiple-choice or constructed-response format. An experimental study was conducted to assess the performance differences of third-grade students with mathematics difficulty on a test of mathematics problem solving as a function of response format. Students responding in the multiple-choice format had a significant advantage over students answering in the constructed-response format.

Several studies have demonstrated that altering the format (question wording or presentation format) can alter responses obtained from normative questions. This study extends that research with two experiments. In a study of McKenzie River boaters in Oregon ( n = 225), respondents received either a semiopen (fill in the blank) or closed (scale with anchored points) format of questions about acceptable waiting time at boat launches, percent of time spent in sight of other boaters, and number of groups met per day. In the second study, at Okefenokee National Wildlife Refuge, 774 respondents received either the semiopen format or one of two closed formats (anchored with 30 or 60), asking for the acceptable number of motorboats and canoes. In both cases, norm prevalence was significantly higher in the closed format (46 to 78%) than in the semiopen format (27 to 60%), and the average increase from one format to the other was 23%. This suggests either that respondents are more likely to guess at a number when given a closed response frame or that the closed format provides important context that elicits valid responses. Examination of the mean values obtained under the different response formats suggest that both processes may be operating, but the latter is more likely. It appears that closed formats, like photographs, offer a cognitively easier and more meaningful task.

The Dual strategy model of reasoning, which distinguishes between Counterexample and Statistical strategies, predicts performance on a variety of forms of reasoning and judgment. At least part of the distinction between reasoning strategies has been shown to be attentional, suggesting that strategy use should be a strong predictor of the ability to reason logically under severe time constraint. Recent results have provided evidence that strategy use is a better predictor than IQ of the ability to make logically valid inferences with belief-biased syllogisms under severe time constraint. The following study extended this result to include other measures that correlate with reasoning ability, IQ, Cognitive Reflection Test and Acceptance of Open-minded Thinking. Results show that when given a very short time (5s), strategy use along with measures of CRT are strong predictors of logical reasoning, while IQ is not.

The validity of individual difference variables for predicting important life phenomena, such as job performance, has been studied for over a century. However, the relationships between some of them have scarcely been investigated. This paper presents a study on the relationship between cognitive reflection (CR), satisfaction with life (SWL), and emotional balance (EB) with job performance. The participants were 245 managers (140 men and 105 women) working in companies in various economic sectors. The results showed that CR was not significantly related to job performance but that SWL and EB were valid predictors. Moreover, CR correlated significantly with SWL and EB. Multiple regression analysis showed that neither CR nor EB showed incremental validity over SWL in predicting job performance. These findings suggest that CR may not be an important variable in the workplace, at least regarding job performance, and that it is not a fully cognitive construct because some degree of its variance may be shared with motivational and personality traits.

A widely used measure of individual propensity to utilize analytic processing is the Cognitive Reflection Test (CRT), a set of math problems with intuitively compelling but incorrect answers. Here, we ask whether scores on this measure are temporally stable. We aggregate data from 11 studies run on Amazon Mechanical Turk in which the Cognitive Reflection Test (CRT) was administered and identifyN= 3,302 unique individuals who completed the CRT two or more times. We find a strong correlation between an individual’s first and last CRT performance,r= .806. This remains true even when constraining to data points separated by over 2 years,r= .755. Furthermore, we find that CRT scores from one timepoint correlated negatively with belief in God and social conservatism from the other timepoint (and to a similar extent as scores gathered at the same timepoint). These results show that CRT scores are stable over time, and – given the stable relationship between CRT and religious belief and ideology – provide some evidence for the stability of analytic cognitive style more generally.