Interpretation of differential item functioning analyses using external review

The purpose of this study is to carry out differential item functioning (DIF) analysis for content areas of a reading comprehension subtest using four area indices within Item Response Theory (IRT) framework. The differences in the magnitudes of the area indices were compared based on the subject areas. The DIF analysis was carried out across gender groups only. The item level data of the English reading comprehension subtest were gathered from English Language Achievement exam done in School of Foreign Languages, Ege University, Turkey, in 2013. A sample of 2,117 examinees (1,011 males and 1,116 females) was randomly selected. For the DIF analysis, (a) an IRT model for the item characteristic curves was specified; (b) model-data-fit was investigated for the selected IRT model;(c) Item characteristic curves were separately computed for each group on a common scale; finally, (d) indices indicating the degree of DIF on each item were computed. The results of the study indicated that both un-weighted and weighted area indices showed non-uniformity in DIF in the item characteristic curves in reading comprehension subtest in most cases. A significant correlation was observed between un-weighted and weighted area indices. Key words: Item characteristic curve, item bias, differential item functioning (DIF).

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel method of differential item functioning (DIF) analysis in computerized adaptive tests (CATs). Each simulated examinee received 25 items from a 75-item pool. A three-parameter logistic item response theory (IRT) model was assumed, and examinees were matched on expected true scores based on their CAT responses and estimated item parameters. The CAT-based DIF statistics were found to be highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. Average DIF statistics and average standard errors also were examined for items with various characteristics. Finally, a study was conducted of the accuracy with which the modified Mantel-Haenszel procedure could identify CAT items with substantial DIF using a classification system now implemented by some testing programs. These additional analyses provided further evidence that the CAT-based DIF procedures performed well. More generally, the results supported the use of IRT-based matching variables in DIF analysis. Index terms: adaptive testing, computerized adaptive testing, differential item functioning, item bias, item response theory.

Differential item functioning (DIF) analysis can be used to great advantage in second language (L2) assessments. This study examined the differences in performance on L2 test items between groups from different first language backgrounds and suggested ways of improving L2 assessments. The study examined DIF on L2 (Hebrew) test items for two language groups: Arabic speakers and Russian speakers. Many items (42%) functioned differentially, posing a potential threat to validity. Most problematic were the sentence completion items, which are more loaded on vocabulary and grammar then the other item types. To validate our study results, two special test forms—non-DIF and large DIF—were constructed, based on the results of the study. These test forms were administered during the years 2004–2005. The findings validated the results of the main study. To reduce the number of DIF items in operational tests, we suggest performing DIF analysis on pilot items and changing the ratios of item types included in the test, while preserving the defined test construct.

The European Organisation for Research and Treatment of Cancer (EORTC) QLQ-C30 is a widely used health-related quality of life instrument. The main aim of this study is to investigate whether there are international differences in response to the questionnaire that can be explained by cultural factors. Analyses involved a database of 106 separate studies including data from over 28,000 respondents. Differential item functioning (DIF) analyses using logistic regression were conducted for each item of the EORTC QLQ-C30 with respect to cultural/geographic group. Results were qualitatively compared with previously reported DIF analyses by translation to explore whether the source of the DIF was more likely to be linguistic or cultural in nature. Although most response patterns were similar, there were a number of international differences in how the questionnaire was answered. The largest variations were found in the results for Eastern Europe and East Asia. Results for the UK, the US and Australia tended to be similar. Many of the European results followed patterns that were more clearly explained when grouped by translation than when grouped by geographical region. Our results suggest that, in general, the EORTC QLQ-C30 is suitable for use in a wide variety of countries and settings. Some response variations that have the potential to affect the results of international studies were identified, but it was not always clear whether the source of the variation was primarily linguistic or cultural.

Differential item functioning (DIF) analysis has been widely used in large-scale assessments and/or commercial and industry produced assessments. Given the fact that DIF analysis has been a routine procedure in item analysis and test validity/fairness analysis, several pieces of DIF software and programs have appeared such as EASY-DIF (Gonzalez, Padilla, Hidalgo, Gomez-Benito, & Benitez, 2011), EZ-DIF (Waller, 1998), DIFAS (Penfield, 2005), and MHDIF (Fidalgo, 1994). To cater for the needs of populous SAS users, two SAS macros (DIF_MC and DIF_Poly) have been developed.

The purpose of this study is to examine the distractors of items that exhibit differential item functioning (DIF) across gender to explain the possible sources of DIF in the context of large-scale tests. To this end, two non-linear logistic regression (NLR) models-based DIF methods (three parameters, 3PL-NLR and four-parameter, 4PL-NLR) were first used to detect DIF items, and the Mantel-Haenszel Delta (MH-Delta) DIF method was used to calculate the DIF effect size for each DIF item. Then, the multinomial log-linear regression (MLR) model and 2-PL nested logit model (2PL-NLM) were applied to items exhibiting DIF with moderate and large DIF effect sizes. The ultimate goals are (a) to examine behaviors of distractors across gender and (b) to investigate if distractors have any impact on DIF effects. DIF results of the Art Section of the General Aptitude Test (GAT-ART) based on both 3PL-NLR and 4PL-NLR methods indicate that only 10 DIF items had moderate to large DIF effects sizes. According to MLR differential distractor functioning (DDF) results, all items exhibited DDF across gender except for one item. An interesting finding of this study is that DIF items related to the verbal analogy and context analysis were in favor of female students, while all DIF items related to the reading comprehension subdomain were in favor of male students, which may signal the existence of content specific DIF or true ability difference across gender. DDF results show that distractors have a significant effect on DIF results. Therefore, DDF analysis is suggested along with DIF analysis since it signals the possible causes of DIF.

This study provides a demonstration of differential item functioning (DIF) analysis. It made use of test scores of 200 junior high school students on a Chemistry Achievement Test, a measure tested for its psychometric properties. One hundred students came from a public school, while the other 100 were private school examinees; one hundred students were males and the other 100 were females; and 95 students were of low ability and 105 students were of high ability based on their English II grades. Four contingency table approaches, the Chi-Square, Distractor Response Analysis, Logistic Regression and the Mantel-Haenszel Statistic, were applied in the DIF analysis to identify test items indicating bias between examinees matched on school type, gender, and English ability. Thereafter, the results for the four approaches were compared. The findings revealed the presence of items indicating school type-, gender-, and English ability-based DIF. There was a high degree of correspondence between the Logistic Regression and the Mantel-Haenszel Statistic in identifying potentially biased test items.

In cross-national comparisons based on questionnaires, accurate translations are necessary to obtain valid results. Differential item functioning (DIF) analysis can be used to test whether translations of items in multi-item scales are equivalent to the original. In data from 10,815 respondents representing 10 European languages we tested for DIF in the nine translations of the EORTC QLQ-C30 emotional function scale when compared to the original English version. We tested for DIF using two different methods in parallel, a contingency table method and logistic regression. The DIF results obtained with the two methods were similar. We found indications of DIF in seven of the nine translations. At least two of the DIF findings seem to reflect linguistic problems in the translation. 'Imperfect' translations can affect conclusions drawn from cross-national comparisons. Given that translations can never be identical to the original we discuss how findings of DIF can be interpreted and discuss the difference between linguistic DIF and DIF caused by confounding, cross-cultural differences, or DIF in other items in the scale. We conclude that testing for DIF is a useful way to validate questionnaire translations.

Differential item functioning (DIF) has been informally conceptualized as multidimensionality. Recently, more formal descriptions of DIF as multidimensionality have become available in the item response theory literature. This approach assumes that DIF is not a difference in the item parameters of two groups; rather, it is a shift in the distribution of ability along a secondary trait that influences the probability of a correct item response. That is, one group is relatively more able on an ability such as test-wiseness. The parameters of the secondary distribution are confounded with item parameters by unidimensional DIF detection models, and this manifests as differences between estimated item parameters. However, DIF is con founded with impact in multidimensional tests, which may be a serious limitation of unidimen sional detection methods in some situations. In the multidimensional approach, DIF is considered to be a function of the educational histories of the examinees. Thus, a better tool for understanding DIF may be provided through structural modeling with external variables that describe background and schooling experience.

Ensuring Validity of NCLEX® With Differential Item Functioning Analysis

Challenges in Measuring Applied Cognition: Measurement Properties and Equivalence of the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT) Applied Cognition Item Bank

When a test is translated from a source language into a target language, the 2 tests are generally not psychometrically equivalent. Differential item functioning (DIF) analysis can facilitate an understanding of the differences between the tests in the 2 languages and can help translators to revise translated items to produce non-DIF (or lower DIF) items. Based on the reasons found for DIF in Allalouf, Hambleton, and Sireci (1999; differences in word difficulty, item format, content, and cultural relevance), a panel of translators and researchers revised the target-language DIF items detected in that study. The revised items were then readministered. Results showed that the revisions succeeded in reducing DIF considerably. For cross-validation purposes, the original translated items were readministered and the study results were generally validated. An attempt was made to determine which sources of DIF and which item types could be revised most effectively. The revision process is expensive, so considerations of cost-effectiveness should be taken into account. Revisions to reduce DIF in translated items may follow the steps used in this study.

It has been rarely studied whether observed disparity in health-related quality-of-life (HRQoL) scores between patients with diabetes and healthy individuals is due to differential item functioning (DIF) or a true difference in the underlying construct. This study aimed to examine DIF in the SF-36 questionnaire and its effect on comparing HRQoL scores between patients with diabetes and healthy people. The sample consisted of 230 patients with type 2 diabetes and 642 healthy individuals who filled out the Persian version of the SF-36 questionnaire. To detect DIF across patients with diabetes and healthy individuals, multiple-group multiple-indicator multiple-causes model was used. In addition, item calibration strategy was used to determine whether the effect of item-level DIF was transferred to the scale level. Nine out of thirty-six (25%) items were detected as DIF, of which one item (11%) was flagged as uniform and eight items (89%) as non-uniform DIF. Most of the DIF items were detected in the mental health component which includes vitality, perceived mental health and social functioning subscales rather than in physical health component. Moreover, nonsignificant latent mean differences for general health perception and social functioning subscales became significant after DIF calibration. The findings of the present study show that patients with diabetes and healthy individuals perceived some items in the SF-36 questionnaire differently. More importantly, in some subscales, the effect of item-level DIF was transferred to the scale level. Consequently, considerable caution should be taken in comparing HRQoL scores between patients with diabetes and healthy individuals.

In this study the authors used differential item functioning analysis to examine if there were items in the Mental, Emotional, and Bodily Toughness Inventory functioning differently across gender and athletic membership. A total of 444 male (56.3%) and female (43.7%) participants (30.9% athletes and 69.1% non-athletes) responded to the Mental, Emotional, and Bodily Toughness Inventory items. Using Mantel–Haenszel and simultaneous item bias test methods, 43 items were analyzed for differential item functioning. Four Mental, Emotional, and Bodily Toughness Inventory items were identified as large differential item functioning items by both Mantel–Haenszel and simultaneous item bias test methods, where item 21 favored non-athletes, item 40 favored athletes, item 2 favored males, and item 17 favored females. Athletic membership differential item functioning disappeared, whereas gender differential item functioning still existed at the scale level. Overall, there are gender and athletic membership differential item functioning items in the Mental, Emotional, and Bodily Toughness Inventory, but only gender differential item functioning still exists at the scale level. Thus, conclusions regarding gender differences in mental toughness should be made with caution when using total Mental, Emotional, and Bodily Toughness Inventory scores.

A study of Quality Assessment of Science Instructional Management in Thailand: An analysis of Differential Item Functioning and Test Functioning in Mixed format Tests