ALTERNATIVE METHODS OF EQUATING THE GRE GENERAL TEST

From the standpoint of test validation, veterinary medicine provides both a unique context in which to study the validity of Graduate Record Examinations (GRE) test scores and a singular opportunity to address the shortcomings typical of many GRE validity studies. This article documents a study of the validity of the GRE General Test for predicting 1st-year grade averages in a comprehensive sample of veterinary medical colleges. For each of 16 veterinary medical colleges, statistical corrections were applied to correct for the effects of range restriction in the predictors and unreliability of the criterion. When fully corrected for both range restriction and unreliability, the resulting validity coefficients were, on average,.53 for the combination of all 3 GRE General Test scores,.59 for undergraduate grade point average, and.71 for GRE scores and undergraduate grade point average together.

This study examined the relationship between 403 counseling graduate students' scores on the Counselor Preparation Comprehensive Examination (CPCE; Center for Credentialing and Education, n.d.) and 3 admissions requirements used as predictor variables: undergraduate grade point average (UGPA), Graduate Record Examinations (GRE) General Test Verbal Reasoning (GRE‐V) score, and GRE General Test Quantitative Reasoning (GRE‐Q) score. Multiple regression analyses revealed that all predictor variables accounted for somewhat limited, yet significant variations in the CPCE‐Total scores (R2 = .21). Results indicated that UGPAs, GRE‐V scores, and GRE‐Q scores are valid criteria for determining counseling graduate student success on the CPCE.

We investigated the functioning of a new computer-delivered response type for potential use in graduate admissions assessment. This response type, which is open-ended and automatically scorable, presents problems calling for the examinee to draw a graph modeling a given situation. Problem situations can be like the single-best-answer items currently found on the Graduate Record Examinations (GRE) General Test (ETS, 1998) or they can be more loosely defined, allowing for multiple-correct responses. Two graphical modeling (GM) tests differing from one another in the manipulation of specific item features were randomly spiraled among study participants. Results showed that GM scores were very reliable and moderately related to the General Test's quantitative section, suggesting that GM might help broaden the GRE quantitative construct. In exploratory difficulty analyses, 1 of 3 manipulated item features, problem structure, had a dependable effect. No significant gender differences independent of those associated with the GRE quantitative section were detected. Finally, more participants preferred regular multiple-choice graphical reasoning questions to GM items but thought GM was the fairer indicator of their ability to undertake graduate study.

In this study data were examined from several national testing programs to determine whether the change from paper‐based administration to computer‐based tests (CBTs) influences group differences in performance. Performances by gender, racial, and ethnic groups on the Graduate Record Examination General Test, Graduate Management Admissions Test, SAT I: Reasoning Test, and Praxis: Professional Assessment for Beginning Teachers, were analyzed to determine whether the shift in testing format from paper‐and‐pencil tests to CBTs posed a disadvantage to any of these subgroups, beyond that already identified for paper‐based tests. Although all differences were quite small, some consistent patterns were found for some racial‐ethnic and gender groups. African‐American examinees and, to a lesser degree, Hispanic examinees appear to benefit from the CBT format. On some tests, female examinees' performance was relatively lower on the CBT version.

Factor structure of the graduate record examinations general test in handicapped and nonhandicapped groups.

This study investigated (a) the ability to minimize or eliminate stereotype threat by reducing the difficulty of items administered via a computer‐adaptive version of the Graduate Record Examinations General Test; and (b) the generalizability of these findings for Black students as well as women, and for verbal as well as quantitative sections of the test. Standard and easier versions of the test, and measures of stereotype threat and possible symptoms or sequelae of stereotype threat were administered to students bound for graduate school or already there. Reducing test difficulty did not have any differential effects on test performance or on explicit indexes of stereotype threat for White and Black students, and for men and women, that were statistically and practically significant. However, such effects did occur for some symptoms or sequelae, mainly for White students and women.

Graduate record examination General Test scores by intended graduate major field, 1988–1991

This article explores how traditional scores obtained from different forced-choice (FC) formats relate to their true scores and item response theory (IRT) estimates. Three FC formats are considered from a block of items, and respondents are asked to (a) pick the item that describes them most (PICK), (b) choose the two items that describe them the most and the least (MOLE), or (c) rank all the items in the order of their descriptiveness of the respondents (RANK). The multi-unidimensional pairwise-preference (MUPP) model, which is extended to more than two items per block and different FC formats, is applied to obtain the responses to each item block. Traditional and IRT (i.e., expected a posteriori) scores are computed from each data set and compared. The aim is to clarify the conditions under which simpler traditional scoring procedures for FC formats may be used in place of the more appropriate IRT estimates for the purpose of inter-individual comparisons. Six independent variables are considered: response format, number of items per block, correlation between the dimensions, item discrimination level, and sign-heterogeneity and variability of item difficulty parameters. Results show that the RANK response format outperforms the other formats for both the IRT estimates and traditional scores, although it is only slightly better than the MOLE format. The highest correlations between true and traditional scores are found when the test has a large number of blocks, dimensions assessed are independent, items have high discrimination and highly dispersed location parameters, and the test contains blocks formed by positive and negative items.

ABSTRACTThis paper examines item response theory (IRT) scale transformations and IRT scale linking methods used in the Non‐Equivalent Groups with Anchor Test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean‐mean, mean‐var linking, concurrent calibration, Stocking and Lord and Haebara characteristic curves approaches, and fixed‐item parameters scale linkage. The main idea is to view any linking procedure as a restriction on the item parameter space. Then a rewriting of the log‐likelihood function together with an appropriately implemented maximization procedure of the log‐likelihood function under linear (or nonlinear restrictions) will accomplish the linking. The proposed method is general enough to cover both the dichotomous item response models (the one parameter logistic (1PL) model, 2PL, and 3PL) and the polytomous unidimensional IRT models like the generalized partial credit model.

Validity of the Graduate Record Examination: Estimated or tacitly known?

ObjectiveTo validate the Workplace COVID-19 Knowledge and Stigma Scale (WoCKSS) using item response theory (IRT), exploratory factor analysis (EFA) and confirmatory factor analysis (CFA).DesignCross-sectional psychometric validation.SettingManufacturing companies registered in Malaysia.ParticipantsA total of 137 factory workers participated in the exploratory phase and 300 in the confirmatory phase. Inclusion criteria were Malaysian nationality and ability to read Malay.MethodsThe knowledge domain was examined using the two-parameter logistic IRT model in two stages: an exploratory IRT analysis in phase 1 to screen items and a confirmatory IRT analysis in phase 2 to evaluate the final item set. The stigma domain was analysed using EFA followed by CFA. Reliability was assessed using Cronbach’s alpha and McDonald’s omega (ω). The development process and content and face validity results were previously published.Results14 knowledge items were retained after exploratory IRT and formed the final knowledge scale evaluated in confirmatory IRT. For these final items, discrimination parameters ranged from 0.77 to 3.17 and difficulty values from −4.47 to 0.23, with unidimensionality supported (p=0.644). EFA supported a three-factor stigma structure (stereotype, fear, prejudice), and CFA confirmed excellent model fit (χ2=8.91, df=11, p=0.630; root mean square error of approximation=0.00; Comparative Fit Index=1.00; Tucker-Lewis Index=1.00; standardised root mean square residual=0.021). Composite reliability by McDonald’s omega ranged from 0.691 to 0.893.ConclusionWoCKSS is a reliable and valid instrument for assessing workplace COVID-19 knowledge and stigma in industrial sectors in Malaysia.

Using data from a large‐scale exam, in this study we compared various designs for equating constructed‐response (CR) tests to determine which design was most effective in producing equivalent scores across the two tests to be equated. In the context of classical equating methods, four linking designs were examined: (a) an anchor set containing common CR items, (b) an anchor set incorporating common CR items rescored, (c) an external multiple‐choice (MC) anchor test, and (d) an equivalent groups design incorporating rescored CR items (no anchor test). The use of CR items without rescoring resulted in much larger bias than the other designs. The use of an external MC anchor resulted in the next largest bias. The use of a rescored CR anchor and the equivalent groups design led to similar levels of equating error.

In educational testing, it is a common practice to produce test forms under a nonequivalent groups with anchor test (NEAT) design. In this design, the two test forms share a certain number of common items, while the populations who take the test forms might not be equivalent to each other. Test linking is conducted to establish the equivalency of ability scales from separate item response theory (IRT) calibrations of two test forms. Among existing linking methods, two approaches are related to the study in this paper: IRT model-based linking (Loyd & Hoover, 1980; Marco, 1977; Stocking & Lord, 1983) and equipercentile linking (Kolen & Brennan, 1995; von Davier, Holland, & Thayer, 2004b).

This monte carlo study assessed the accuracy of simultaneous estimation of item and person parameters in item response theory. Item responses were simulated using the two- and three-parameter logistic models. Samples of 200, 500, 1,000, and 2,000 simulated examinees and tests of 15, 30, and 60 items were generated. Item and person parameters were then estimated using the appropriate model. The root mean squared error between recovered and actual item characteristic curves served as the principal measure of estimation accuracy for items. The accuracy of estimates of ability was assessed by both correlation and root mean squared error. The results indicate that minimum sample sizes and tests lengths depend upon the response model and the purposes of an investigation. With item responses generated by the two-parameter model, tests of 30 items and samples of 500 appear adequate for some purposes. Estimates of ability and item parameters were less accurate in small sample sizes when item responses were generated by the three-parameter logistic model. Here samples of 1,000 examinees with tests of 60 items seem to be required for highly accurate estimation. Tradeoffs between sample size and test length are apparent, however.

A previous study of the initial, preoperational version of the Graduate Record Examinations (GRE) analytical ability measure (Powers & Swinton, 1984) revealed practically and statistically significant effects of test familiarization on analytical test scores. (Two susceptible item types were subsequently removed from the test.) Data from this study were reanalyzed for evidence of differential effects for subgroups of examinees classified by age, ethnicity, degree aspiration, English language dominance, and performance on other sections of the GRE General Test. The results suggested little, if any, difference among subgroups of examinees with respect to their response to the particular kind of test preparation considered in the study. Within the limits of the data, no particular subgroup appeared to benefit significantly more or significantly less than any other subgroup.