Test Taking Strategies: Implications for Test Validation

This article examines the relationship between test administration tools and item types. It firstly examines the difference in the test administration tools such as paper-based test, computer-based test, and web-based test, in particular, dealing with speaking test. The difference is relevant to item or task types. Based on the considerations of Bachman and Palmer (1996) and Chapelle and Douglas (2006) on test development, and Weir's (2005) definition of context validity, the application of computer technologies to test administration needs to be considered an important construct affecting test takers' performance. To investigate the item type difference among current speaking tests and its relationship with computer technology in administrating the tests, twelve test takers, five proctors, and two computer lab administrator participated in the phone interview and e-mail survey. Six popular speaking tests (TSE, PELT, IELTS, MATE, iBT TOEFL, ESPT) were selected to be analyzed. The data were analyzed in terms of item types, computer technology use, and test-takers perceptions. The implication is suggested for students, teachers, and language testers.

The main aim of the study was to investigate the effect of question-generation strategy on Iranian EFL third grade high school students’ ability in reading comprehension passages via multiple-choice question .In this research, the total number of one hundred and twenty male and female students participated. For homogeneity of the students, a proficiency test including a multiple—choice reading comprehension test (Hill, 1980) was ascertained in the twenty selected questions. Eight multiple—choice reading comprehension passages (Hill, 1980) used based on the target of pre-test and post-test in the forty selected questions. In this research, the students were divided into two experimental groups; each group consisting of 30 homogeneous students (i.e., EMG as Experimental Male Group and EFG as Experimental Female Group) and two control groups (i.e., CMG as Control Male Group and CFG as Control Female Group). In the treatment time, the two experimental groups of the students were instructed for 10 weeks. Outcome of the study showed that question-generation strategy had significantly impact on the Iranian EFL third grade high school learners’ in reading comprehension passages.

ABSTRACTThe current study explores the nature and relationships among test takers’ performance appraisals, appraisal calibration, and reported cognitive and metacognitive strategy use in a language test situation. Performance appraisals are executive processes of strategic competence for judging test performance (e.g., evaluating the correctness or appropriateness of responses to given test tasks). Appraisal calibration denotes a perfect match between appraisal confidence and actual test performance. 294 English as a foreign language (EFL) students took an English test, which was designed to measure four language areas (listening, grammar, vocabulary, and reading). The students reported their level of appraisal confidence immediately after answering each test question. At the end of the test, they were asked to report their overall appraisal confidence and perceived cognitive and metacognitive strategy use in the test. First, it was found that test takers were not well calibrated in all test sections. Second, test takers’ appraisal confidence could predict just above one third of the test performance variance. Third, test takers tended to be underconfident in easy questions but overconfident in difficult questions. Fourth, appraisal calibration was not strongly related to reported metacognitive strategy use. The implications of the study in light of strategic competence research and recommendations for further research are discussed.

ABSTRACTThis study was directed to answering the question, “Can the items in a test which is slightly speeded be rearranged in such a way as to discourage cooperation among test takers in a group setting, without affecting the item and score properties of the test?” The literature on the rearrangement problem consists of studies in which the test was either highly speeded or completely unspeeded, or in which rearrangements were extreme. Only moderate rearrangements were used in the present study.Two types of items were used. One group contained 40 items typical of those in the Verbal section of the CEEB Scholastic Aptitude Test (SAT) and one contained 25 items typical of the Mathematics section of the SAT. Four different arrangements of each type of test were administered to spaced samples of a college applicant group in March 1965. Five thousand cases were drawn from each of the eight groups for analysis of total score effects, and independent samples of 495 cases were drawn for item analysis.Results indicated no significant alteration of the item statistics or of the tests' correlations with other tests, but analysis of the regression of the experimental Verbal tests on a longer Verbal test revealed significant differences in intercepts; i.e., some of the Verbal arrangements were slightly more difficult than others. No such difference was found for the Mathematics tests. A possible explanation for these results is that in some of the Verbal arrangements relatively easy items occurred last and were not reached by some students. Thus dropout may have affected test scores differentially.It was concluded that for tests of this type and degree of speededness, an a priori assumption of equivalence among different arrangements might well be false. It was recommended, therefore, that if different arrangements are used in operational administrations of the SAT, separate equating statistics be computed for each arrangement.

ABSTRACTThe controversy over the effectiveness of coaching for standardized admission tests has been fueled by studies that have sought dichotomous answers to questions that are more appropriately posed as ones of degree. Instead of asking categorically “Does coaching work?“, researchers would seem better advised to ask “How much time and effort, devoted to what kinds of coaching experiences, produce how much improvement in test performance?”To explore some relational answers, information on test preparation activities was collected from a large representative sample of candidates who took the GRE Aptitude Test in June 1980. About 3 percent of these candidates indicated that they had attended formal coaching programs for one or more sections of the test and provided information on the length, cost, and offerer of the courses.After adjusting for differences in the background characteristics of coached and uncoached students, effects on test scores were related to the length, the cost, and the type of programs offered. The effects on GRE verbal ability scores were not significantly related to the amount of coaching that examinees received. Quantitative coaching effects appeared to increase only slightly with time, but the relationship was not statistically significant. Effects on analytical ability scores, on the other hand, were related significantly to the length of coaching programs, through improved performance on two analytical item types, analysis of explanations and logical diagrams. These item types were shown previously to be susceptible to improvement through formal instructional intervention and have since been deleted from the test.Test performance was related to the kinds of coaching programs examinees attended only for the quantitative section of the test. With respect to the possibility that some kinds of test takers might profit more than others from coaching, exploratory analyses suggested that examinees intending to pursue higher‐level degrees may have benefited more than lower‐level degree seekers. This finding is consistent with the conjecture that highly motivated test takers may achieve greater effects from coaching than less motivated examinees.Overall, the data suggest that, when compared with the two highly susceptible item types that have been removed from the GRE Aptitude Test, the test item types in the current version of the test (now called the GRE General Test) appear to show relatively little susceptibility to formal coaching experiences of the kinds considered here.

In pursuit of the precise mechanism by which working memory (WM) functions in second language (L2) reading comprehension (RC), the current study examined whether strategy use or task difficulty influences the role of WM in L2 RC.Eighty Korean adult EFL learners participated, and their WM capacity, RC performance, and strategy use were measured.The variable of task difficulty was operationalized by the relative difficulty of texts on which RC items were based.The results showed that difference in strategy use brought about significant difference in RC performance when the readers with insufficient WM capacity had to comprehend difficult texts, whereas no such difference was found with easier texts or among L2 readers with more WM capacity.These findings not only illustrate intricate interrelationships among WM and other variables in L2 RC, but also provide a noteworthy implication for L2 reading instruction that L2 learners with deficient WM capacity could benefit from active strategy use when faced with a difficult L2 reading task.The compensatory role of strategy and the necessity of remedial strategy training for the learners with limited WM is further implicated for L2 reading classrooms.

The paper caused an investigation into the causes of Kuka M/A Primary pupils’ inability to read comprehension passages and to sustain their interest in reading comprehension using eclectic approach. Literature was provided to support the study. Action research design was used in the study. A sample size of fifty (50) pupils was drawn from the entire school population for data gathering purposes. The instruments employed for the study were tests and observation while data were analyzed using frequency tables. The study revealed that lack of participatory activity in the classrooms, lack of pre-reading skills, absenteeism, and others were the causes of the pupils’ inability to read. The study further revealed that using an eclectic approach improves pupils’ performance in class. It became known that most of the pupils were now interested in reading comprehension passages very well because the lessons adopted the interactive activity-based approach. A number of recommendations were made to assist stakeholders of the school to make good use of this current study to improve upon their performance.

ABSTRACTThis study examined the relationship of differential item functioning (DIF) to item difficulty on the Scholastic Aptitude Test (SAT). The data comprise verbal and mathematical item statistics from nine recent administrations of the SAT. In general, item difficulty is related to DIF. The nature of that relationship appears to be independent of the choice of DIF index (either the Mantel‐Haenszel or the standardization approach) as well as of test form. However, the relationship was dependent on the particular group comparison and on both the test sections and the item type being analyzed.The relationship was strong for each of the racial and ethnic group contrasts—in which black, Hispanic, and Asian American examinees were compared in turn with white examinees—but was weak for the female and male examinee contrast. The relationship also appeared stronger on the verbal sections than on the mathematical sections. The relationship is such that more difficult items tended to exhibit positive DIF (DIF favored the focal group over the white reference group). On the verbal sections, only the reading comprehension item type (with the smallest observed range in item difficulty) failed to exhibit a strong relationship.Another index, the standardized difference in percentage omit (DIFPOM), correlated very highly (negatively) with DIF. Differential omission refers to a relative difference in omit rates between groups matched in ability. In fact, DIFPOM was consistently a better predictor of DIF in most models than was item difficulty. The relationship between DIF and DIFPOM held up across all four comparisons, including gender. It was also present in the mathematical sections with nearly the same magnitude exhibited in the verbal sections.Although DIF and DIFPOM are mathematically dependent measures, it was proposed that DIFPOM may be partly responsible for the relationship between DIF and item difficulty. To what extent DIF is a consequence of differential omission and to what extent differential omission is a manifestation of DIF is problematic. Nonetheless, the presence of differential omission on a test has the potential to influence DIF indices and therefore should be an important concern especially for formula‐scored tests, where omission occurs often on difficult items.Among other findings is that Hispanic and black focal groups tended to omit differentially less than did the white reference groups. For Asian American examinees, the reverse holds. For females and males, the direction depends on the test sections. In general, groups that omitted differentially less experienced a relative advantage (high‐positive DIF values) on the more difficult items, as measured by the DIF indices studied here (which treat omits as wrong in their calculation).

ABSTRACTThis study investigated the effect of the interaction between test takers' background knowledge and language proficiency on their performance on the TOEFL iBT® reading section. Test takers with the target content background knowledge (the focal groups) and those without (the reference groups) were identified for each of the 5 selected passages based on their self‐identified academic and cultural backgrounds. The test takers were further classified into high and low proficiency groups based on their TOEFL iBT scores. Differential functioning was investigated at the item, item bundle, and passage levels. The results suggested that background knowledge interacted with language proficiency on certain items, which could be attributed to idiosyncratic passage and item characteristics (i.e., characteristics that were specific to a particular passage or item). Only 1 of the 5 passages investigated showed intermediate differential bundle functioning, favoring the focal group for both the high and low proficiency groups. There was no differential functioning at the passage level. This research sheds new light on our understanding of the effects of background knowledge and its interaction with language proficiency in the context of second language reading comprehension. It also has significant practical implications for test developers in advancing fair assessments.

A brief curriculum of special preparation for the analytical section of the GRE Aptitude Test was developed and administered to a sample of self-selected GRE candidates at a large eastern university. Students' reactions to the materials and procedures used were obtained. The effects of the program on analytical scores were determined by comparing the scores subsequently obtained by the candidates who undertook special preparation with the scores of all the other candidates who took the test on the same date at the university. Analyses revealed an effect on analytical scores that was both practically and statistically significant. The effect stemmed from improved performance on two of the three analytical item types included in the 1977-1981 analytical section. Hypotheses concerning the components of the special preparation that were responsible for the improvement are discussed. A Study of the Effects of Special Preparation on GRE Analytical Scores and Item Types The effects of special preparation on test scores are of interest to test specialists in several respects. First, a test's susceptibility to score changes that result from various interventions may reflect directly on the validity with which the intended construct is measured. For example, tests designed to reflect student achievement are generally more sensitive to certain interventions, even those that are short-term in nature, than are aptitude tests, which are intended to measure abilities or skills that are developed over a longer period of time. For aptitude tests, sensitivity to short-term special preparation of the kind studied here may indicate that a substantial proportion of the variation in test scores is due to differences in examinees' facility with the methods of assessment or familiarity with the format of items, in contrast to differences in the presumably more stable trait that is being measured. Second, if special preparation increases the test score variance due strictly to differential facility in taking the test, then the predictive power of the test may be impaired since it is unlikely that the same method-related variance would underlie any important criterion. And finally, because all candidates may not have equal access to special preparation, a potential for inequity exists, especially for admissions tests, since any benefits may accrue disproportionately to those examinees who have the time and the financial means to obtain special preparation. The Restructured GRE Aptitude Test The analytical section of the GRE Aptitude Test was introduced in 1977 as a result of a major restructuring of the test (Miller & Wild, 1979). The analytical portion of the test is intended to measure analytical reasoning abilities that, like the verbal and quantitative skills measured by the test, are assumed to develop over a relatively long period of time. The item types originally used in the analytical section of the test are described briefly as follows: 1. Analysis of Explanations: A fact situation paragraph and a surprising result, given the situation, are presented. Approximately 10 numbered statements follow. Each statement is to be evaluated separately from the other statements as to whether it: (A) contradicts something in the fact situation, the result, or both taken together; (B) presents a possible adequate explanation of the result;

In the past ten years, there has a research interest in morphological awareness, which refers to an individual’s ability to decode the morphemic structure of words and further analyze them. The current study is an attempt to investigate levels of awareness of EFL Saudi university students and also to discover any potential relationship between their morphological awareness and successful reading comprehension. To this end, the researcher administered a modified version of Morphological Awareness Test (McBride- Chang et al. 2008) to 35 undergraduate students at the Department of Foreign Languages at Faculty of Arts and Humanities of Al Baha University in the academic year 2016/2017. The test included the analytic and synthetic aspects of word formation rules. Instruments of the study also included an adopted version of Reading Comprehension Test for Smart Choice Learners (Oxford, 2007). Results of the study indicated that the average score of the Morpheme Identification section (the analytic aspect of morphological awareness) was the highest among the students (M= 27.11, SD= 6.20) in comparison to the synthetic aspect of morphological awareness (M= 14.66, SD= 11.91). The students noticeably scored better in the Morpheme Identification Test (88.57%) than they did in the Morphological Structure Test (57.6%). The overall mean score of the Morphological Awareness Test was 41.77 out of 68 with a considerable dispersion among the results (SD= 14.63) with overall percentage (65.71%) which indicated that the students had intermediate awareness of word formation rules. In addition, EFL students scored better in the Inflectional affixes in both the analysis section (63%, SD= 5.32) and the synthesis section (50%, SD=9.85) than they did with the Derivational affixes (59.15%, SD= 7.38 in the analysis section, 46.33%, SD=13.72 in the synthesis section). Furthermore, there is high positive correlation between total students' scores on analytical aspect section of the morphological awareness test and reading comprehension test (0.871) (0.009). There is positive correlation between students' scores on synthetic aspect section of the morphological test and reading comprehension (0.841) (0.005), but it is weaker than that of the first section of the test. The total scores of students on morphological test positively correlate with their total scores on reading comprehension test (0.869) (0.005). The present study ended up with a set of pedagogical implications and recommendations to include training on rules of word formation in EFL curriculum so as to increase their morphological awareness and to develop their successful reading comprehension endeavors. Future research themes were also recommended such as the role of morphological awareness in tracing semantic irregularities to enhance learners' understanding of different English words and to explore the relationship between morphological awareness and other language learning skills including listening and speaking. Article visualizations:

Negative worded (NW) items used in psychological instruments have been studied with the bifactor model to investigate whether the NW items form a secondary factor due to negative wording orthogonal to the measured latent construct, a validation procedure which checks whether NW items form a source of construct irrelevant variance (CIV) and hence constitute a validity threat. In the context of educational testing, however, no such validation attempts have been made. In this study, we studied the psychometric impact of NW items in an English proficiency reading comprehension test using a modeling approach similar to the bifactor model, namely the three-parameter logistic cross-classified testlet response theory (3PL CCTRT) model, to account for both guessing and possible local item dependence due to passage effect in the data set. The findings indicate that modeling the NW items with a separate factor leads to noticeable improvement in model fit, and the factor variance is marginal but nonzero. However, item and ability parameter estimates are highly similar between the 3PL CCTRT model and other models that do not model the NW items. It is concluded that the NW items introduce CIV into the data, but its magnitude is too small to change item and person ability parameter estimates to an extent of practical significance.

Reading comprehension skills are critical predictors of students’ academic performance. This study, based on PIRLS 2021 paper-based test data, investigates whether multiple-choice reading comprehension items function differently according to students’ socioeconomic status (SES) in the samples of Türkiye, Ireland, and South Africa. A total of 21 multiple-choice items drawn from the following passages were analyzed: Where is the Honey?, Pemba Sherpa, and Marie Curie. The analyses of Differential Item Functioning (DIF) were conducted by using Mantel–Haenszel (MH) test, Logistic Regression (LR), and Lord’s Chi-Square method. The findings revealed that the highest number of items exhibiting Differential Item Functioning (DIF) was observed in Turkey, while the lowest was found in South Africa. Additionally, Logistic Regression was identified as the method that detected the greatest number of DIF items. Observed differences in item functioning by SES varied significantly across countries, passages, and DIF detection methods. Notably, the number of DIF items identified in the Turkish sample was relatively high. Furthermore, Logistic Regression (LR) demonstrated higher sensitivity in detecting DIF compared to the other methods employed. The results underscore the importance of evaluating measurement fairness not only from a statistical standpoint but also within contextual, cultural, and cognitive dimensions.

Individual differences in performance on comprehension and knowledge tests with and without passages and questions

A brief curriculum of special preparation for the analytical section of the GRE Aptitude Test was developed and administered to a sample of self-selected GRE candidates at a large eastern university. Students' reactions to the materials and procedures used were obtained. The effects of the program on analytical scores were determined by comparing the scores subsequently obtained by the candidates who undertook special preparation with the scores of all the other candidates who took the test on the same date at the university. Analyses revealed an effect on analytical scores that was both practically and statistically significant. The effect stemmed from improved performance on two of the three analytical item types included in the 1977-1981 analytical section. Hypotheses concerning the components of the special preparation that were responsible for the improvement are discussed. A Study of the Effects of Special Preparation on GRE Analytical Scores and Item Types The effects of special preparation on test scores are of interest to test specialists in several respects. First, a test's susceptibility to score changes that result from various interventions may reflect directly on the validity with which the intended construct is measured. For example, tests designed to reflect student achievement are generally more sensitive to certain interventions, even those that are short-term in nature, than are aptitude tests, which are intended to measure abilities or skills that are developed over a longer period of time. For aptitude tests, sensitivity to short-term special preparation of the kind studied here may indicate that a substantial proportion of the variation in test scores is due to differences in examinees' facility with the methods of assessment or familiarity with the format of items, in contrast to differences in the presumably more stable trait that is being measured. Second, if special preparation increases the test score variance due strictly to differential facility in taking the test, then the predictive power of the test may be impaired since it is unlikely that the same method-related variance would underlie any important criterion. And finally, because all candidates may not have equal access to special preparation, a potential for inequity exists, especially for admissions tests, since any benefits may accrue disproportionately to those examinees who have the time and the financial means to obtain special preparation. The Restructured GRE Aptitude Test The analytical section of the GRE Aptitude Test was introduced in 1977 as a result of a major restructuring of the test (Miller & Wild, 1979). The analytical portion of the test is intended to measure analytical reasoning abilities that, like the verbal and quantitative skills measured by the test, are assumed to develop over a relatively long period of time. The item types originally used in the analytical section of the test are described briefly as follows: 1. Analysis of Explanations: A fact situation paragraph and a surprising result, given the situation, are presented. Approximately 10 numbered statements follow. Each statement is to be evaluated separately from the other statements as to whether it: (A) contradicts something in the fact situation, the result, or both taken together; (B) presents a possible adequate explanation of the result;