Constructing Better Second Language Assessments Based on Differential Item Functioning Analysis

Differential item functioning (DIF) analysis has been widely used in large-scale assessments and/or commercial and industry produced assessments. Given the fact that DIF analysis has been a routine procedure in item analysis and test validity/fairness analysis, several pieces of DIF software and programs have appeared such as EASY-DIF (Gonzalez, Padilla, Hidalgo, Gomez-Benito, & Benitez, 2011), EZ-DIF (Waller, 1998), DIFAS (Penfield, 2005), and MHDIF (Fidalgo, 1994). To cater for the needs of populous SAS users, two SAS macros (DIF_MC and DIF_Poly) have been developed.

The purpose of this study is to carry out differential item functioning (DIF) analysis for content areas of a reading comprehension subtest using four area indices within Item Response Theory (IRT) framework. The differences in the magnitudes of the area indices were compared based on the subject areas. The DIF analysis was carried out across gender groups only. The item level data of the English reading comprehension subtest were gathered from English Language Achievement exam done in School of Foreign Languages, Ege University, Turkey, in 2013. A sample of 2,117 examinees (1,011 males and 1,116 females) was randomly selected. For the DIF analysis, (a) an IRT model for the item characteristic curves was specified; (b) model-data-fit was investigated for the selected IRT model;(c) Item characteristic curves were separately computed for each group on a common scale; finally, (d) indices indicating the degree of DIF on each item were computed. The results of the study indicated that both un-weighted and weighted area indices showed non-uniformity in DIF in the item characteristic curves in reading comprehension subtest in most cases. A significant correlation was observed between un-weighted and weighted area indices. Key words: Item characteristic curve, item bias, differential item functioning (DIF).

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel method of differential item functioning (DIF) analysis in computerized adaptive tests (CATs). Each simulated examinee received 25 items from a 75-item pool. A three-parameter logistic item response theory (IRT) model was assumed, and examinees were matched on expected true scores based on their CAT responses and estimated item parameters. The CAT-based DIF statistics were found to be highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. Average DIF statistics and average standard errors also were examined for items with various characteristics. Finally, a study was conducted of the accuracy with which the modified Mantel-Haenszel procedure could identify CAT items with substantial DIF using a classification system now implemented by some testing programs. These additional analyses provided further evidence that the CAT-based DIF procedures performed well. More generally, the results supported the use of IRT-based matching variables in DIF analysis. Index terms: adaptive testing, computerized adaptive testing, differential item functioning, item bias, item response theory.

This study provides a demonstration of differential item functioning (DIF) analysis. It made use of test scores of 200 junior high school students on a Chemistry Achievement Test, a measure tested for its psychometric properties. One hundred students came from a public school, while the other 100 were private school examinees; one hundred students were males and the other 100 were females; and 95 students were of low ability and 105 students were of high ability based on their English II grades. Four contingency table approaches, the Chi-Square, Distractor Response Analysis, Logistic Regression and the Mantel-Haenszel Statistic, were applied in the DIF analysis to identify test items indicating bias between examinees matched on school type, gender, and English ability. Thereafter, the results for the four approaches were compared. The findings revealed the presence of items indicating school type-, gender-, and English ability-based DIF. There was a high degree of correspondence between the Logistic Regression and the Mantel-Haenszel Statistic in identifying potentially biased test items.

Differential item functioning (DIF) analyses are used to determine whether certain groups respond differently to a particular item of a test or questionnaire; however, these do not explain the reasons for observed response differences. Many studies have used external reviews of items, sometimes using blinded reviewers, to help interpret these results. The authors conducted a literature review of this topic to describe the current usage of external reviews alongside DIF analyses. It concentrated on studies of health-related quality of life instruments, but studies in other fields were also considered. Relatively few examples of blinded item reviews were identified, and these were mostly from educational studies. A case study using blinded bilingual reviewers alongside translation DIF analyses of a health-related quality of life instrument is described. Future researchers should consider conducting external item reviews alongside DIF analyses.

The purpose of this study is to examine the distractors of items that exhibit differential item functioning (DIF) across gender to explain the possible sources of DIF in the context of large-scale tests. To this end, two non-linear logistic regression (NLR) models-based DIF methods (three parameters, 3PL-NLR and four-parameter, 4PL-NLR) were first used to detect DIF items, and the Mantel-Haenszel Delta (MH-Delta) DIF method was used to calculate the DIF effect size for each DIF item. Then, the multinomial log-linear regression (MLR) model and 2-PL nested logit model (2PL-NLM) were applied to items exhibiting DIF with moderate and large DIF effect sizes. The ultimate goals are (a) to examine behaviors of distractors across gender and (b) to investigate if distractors have any impact on DIF effects. DIF results of the Art Section of the General Aptitude Test (GAT-ART) based on both 3PL-NLR and 4PL-NLR methods indicate that only 10 DIF items had moderate to large DIF effects sizes. According to MLR differential distractor functioning (DDF) results, all items exhibited DDF across gender except for one item. An interesting finding of this study is that DIF items related to the verbal analogy and context analysis were in favor of female students, while all DIF items related to the reading comprehension subdomain were in favor of male students, which may signal the existence of content specific DIF or true ability difference across gender. DDF results show that distractors have a significant effect on DIF results. Therefore, DDF analysis is suggested along with DIF analysis since it signals the possible causes of DIF.

Reading comprehension skills are critical predictors of students’ academic performance. This study, based on PIRLS 2021 paper-based test data, investigates whether multiple-choice reading comprehension items function differently according to students’ socioeconomic status (SES) in the samples of Türkiye, Ireland, and South Africa. A total of 21 multiple-choice items drawn from the following passages were analyzed: Where is the Honey?, Pemba Sherpa, and Marie Curie. The analyses of Differential Item Functioning (DIF) were conducted by using Mantel–Haenszel (MH) test, Logistic Regression (LR), and Lord’s Chi-Square method. The findings revealed that the highest number of items exhibiting Differential Item Functioning (DIF) was observed in Turkey, while the lowest was found in South Africa. Additionally, Logistic Regression was identified as the method that detected the greatest number of DIF items. Observed differences in item functioning by SES varied significantly across countries, passages, and DIF detection methods. Notably, the number of DIF items identified in the Turkish sample was relatively high. Furthermore, Logistic Regression (LR) demonstrated higher sensitivity in detecting DIF compared to the other methods employed. The results underscore the importance of evaluating measurement fairness not only from a statistical standpoint but also within contextual, cultural, and cognitive dimensions.

A study of Quality Assessment of Science Instructional Management in Thailand: An analysis of Differential Item Functioning and Test Functioning in Mixed format Tests

In cross-national comparisons based on questionnaires, accurate translations are necessary to obtain valid results. Differential item functioning (DIF) analysis can be used to test whether translations of items in multi-item scales are equivalent to the original. In data from 10,815 respondents representing 10 European languages we tested for DIF in the nine translations of the EORTC QLQ-C30 emotional function scale when compared to the original English version. We tested for DIF using two different methods in parallel, a contingency table method and logistic regression. The DIF results obtained with the two methods were similar. We found indications of DIF in seven of the nine translations. At least two of the DIF findings seem to reflect linguistic problems in the translation. 'Imperfect' translations can affect conclusions drawn from cross-national comparisons. Given that translations can never be identical to the original we discuss how findings of DIF can be interpreted and discuss the difference between linguistic DIF and DIF caused by confounding, cross-cultural differences, or DIF in other items in the scale. We conclude that testing for DIF is a useful way to validate questionnaire translations.

When a test is translated from a source language into a target language, the 2 tests are generally not psychometrically equivalent. Differential item functioning (DIF) analysis can facilitate an understanding of the differences between the tests in the 2 languages and can help translators to revise translated items to produce non-DIF (or lower DIF) items. Based on the reasons found for DIF in Allalouf, Hambleton, and Sireci (1999; differences in word difficulty, item format, content, and cultural relevance), a panel of translators and researchers revised the target-language DIF items detected in that study. The revised items were then readministered. Results showed that the revisions succeeded in reducing DIF considerably. For cross-validation purposes, the original translated items were readministered and the study results were generally validated. An attempt was made to determine which sources of DIF and which item types could be revised most effectively. The revision process is expensive, so considerations of cost-effectiveness should be taken into account. Revisions to reduce DIF in translated items may follow the steps used in this study.

Ensuring Validity of NCLEX® With Differential Item Functioning Analysis

Challenges in Measuring Applied Cognition: Measurement Properties and Equivalence of the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT) Applied Cognition Item Bank

To determine whether a Portuguese language version of the Child Perceptions Questionnaire for 11-14-year-olds (CPQ(11-14)) showed differential item functioning (DIF) when compared with the original English language version. CPQ(11-14) data from a school-based Brazilian study (n = 138) was compared with CPQ(11-14) data collected as part of a school-based study conducted in New Zealand (n = 322). In order to detect DIF, ordinal logistic regression analysis was performed with each CPQ(11-14) item as the dependent variable. The independent variables were language group (English versus Portuguese), the CPQ(11-14) sub-scale score of which the item was a part, and an interaction term for language*sub-scale score. Nonuniform DIF was deemed to be present if the interaction term was significant. Moderate to large uniform DIF was deemed to be present if after removing the interaction term the beta coefficient (log odds ratio) for language group was significant and numerically greater than 0.64. Analyses were also undertaken to detect pseudo-DIF. Nonuniform DIF was found in five items and moderate to large uniform DIF in an additional four items. Analyses using 'purified' sub-scale scores indicated that little of the DIF detected was pseudo-DIF. A comparison of the language groups using DIF affected and DIF-free overall and subscale CPQ(11-14) scores revealed that the DIF detected had only a marginal effect on the differences between language groups in scores. Oral health-related quality of life questionnaires, particularly those that have been translated, need to be assessed for DIF and its likely impact on group comparisons.

Differential item functioning (DIF) has been informally conceptualized as multidimensionality. Recently, more formal descriptions of DIF as multidimensionality have become available in the item response theory literature. This approach assumes that DIF is not a difference in the item parameters of two groups; rather, it is a shift in the distribution of ability along a secondary trait that influences the probability of a correct item response. That is, one group is relatively more able on an ability such as test-wiseness. The parameters of the secondary distribution are confounded with item parameters by unidimensional DIF detection models, and this manifests as differences between estimated item parameters. However, DIF is con founded with impact in multidimensional tests, which may be a serious limitation of unidimen sional detection methods in some situations. In the multidimensional approach, DIF is considered to be a function of the educational histories of the examinees. Thus, a better tool for understanding DIF may be provided through structural modeling with external variables that describe background and schooling experience.

Studies on the differential item functioning (DIF) are usually considered in the context of manifest groups. Recently, with the increase in the number of analyses conducted with mixture models, investigating the situations that cause differences between groups has come to the forefront. In addition, it is considered important to examine the DIF with mixture models in which levels are also handled. In this study, it is aimed to compare the results of the multilevel mixture item response theory (MMIRT) model and the mixture item response theory (MIRT) model and the results of the DIF analyses based on the manifest groups. The research sample consists of students who answered the second booklet in the electronic Trends in International Mathematics and Science Study (eTIMSS) 2019 and coded their gender. The answers given to 15 items were analyzed with the Mantel Haenszel (MH) method for the gender variable according to the manifest groups, and with the selection of the most appropriate models by varying the number of groups and the number of levels according to the MIRT model and the MMIRT model. DIF analyses of the obtained latent groups were also performed with the MH method. In the light of the findings, the number of items displaying DIF in both the MIRT model and the MMIRT model is higher than the manifest groups. While only one item displayed DIF in the analysis according to gender, 14 items displayed DIF according to the MIRT model and seven items displayed DIF according to the MMIRT model. There is not a complete overlap in the number of DIF items and DIF effect sizes found as a result of the MIRT model and MMIRT model analyses. For this reason, a level analysis should be conducted before the analyses and if there is multi-levelness, the analyses should be conducted by taking this situation into consideration.