Evaluating the Magnitude of Differential Item Functioning in Polytomous Items

This study aims to identify the sources of Differential Item Functioning (DIF) using the Mixture Ordinal Logistic Regression (Mixture OLR) method, a contemporary approach for detecting DIF. To analyze mathematics self-efficacy, data from a scale comprising 9 items were obtained from 5000 8th-grade students as part of the ABIDE-2016 project. The study compared the presence and extent of DIF by gender using two methods and examined the sources of DIF for items displaying DIF with Mixture OLR. The OLR analysis revealed that five items exhibited DIF at level A, but no DIF was observed with Mixture OLR. Furthermore, it was found that the magnitude of DIF (B) for an item showing DIF at level A changed due to Mixture OLR. The results indicate that the homogeneity of the data affects both the number of items displaying DIF and the magnitude of DIF. Three items did not exhibit significant DIF according to both methods. One significant finding in the study highlights the moderating effect of latent class on item 8, where DIF was observed. However, the source of DIF was not related to gender but rather stemmed from different ecological variables. An analysis of latent class characteristics revealed that students with significant DIF effects had lower absenteeism and fewer siblings. Additionally, students in this class had greater access to books at home and participated in more out-of-school mathematics courses. Surprisingly, these students were found to engage less in social activities. Various factors can influence how students respond to test items, potentially leading to DIF. These factors may include cultural background, gender, social environment, school, teacher, family interest/attitude toward the child, and home climate. Therefore, when developing and administering tests, it is crucial to test for data homogeneity and consider the impact of these variables, in addition to gender, to identify any sources of DIF in test items.

The Center for Epidemiological Studies-Depression Scale (CES-D) is the most widely used instrument to assess depressive symptoms in people living with HIV. However, its differential item functioning (DIF) by HIV status and sexual orientation has yet to be explored. This study examined DIF and measurement invariance of the CES-D using an item response theory (IRT) framework, and a more traditional factor analytic approach. Data from 841 HIV-infected and HIV-uninfected individuals, from Miami, Florida, were analyzed. Uniform DIF by HIV status was detected in Items 4, 12, and 16 from the Positive Affect factor. Nonuniform DIF was detected in Items 13 and 17. Uniform DIF by sexual orientation was detected in Items 2, 15, and 19, two of them from the Interpersonal factor. Nonuniform DIF was detected in Item 2. Using a factor analytic approach, the CES-D was invariant at the configural and metric levels by HIV and sexual orientation. These findings indicate that overall, however, using IRT, the magnitudes of DIF were negligible, the CED-D was somewhat invariant using factor analytic methods; the CES-D may be reliably used to compare by HIV status or sexual orientation.

The authors demonstrated the empirical relationship between Cohen's chi-square effect size, w, and differential item functioning (DIF), defined as group differences in item response theory (IRT) item difficulty. In Experiment 1, in which the lower asymptote was 0, the authors argued that Cohen's designation of small, medium, and large effects connotes reasonably well for that definition of DIF. In Experiment 2, the lower asymptote of the item response function was raised from 0 to 0.2 and the item discrimination parameter was held to 1.0. Doing so admitted non-crossing nonuniform DIF to the model, violating an underlying assumption of the Mantel–Haenszel procedure that the odds ratio is constant across studied levels of the matching criterion. Smaller difficulty parameter difference resulted, which produced larger effects with an inflation in effect size of about 15%. In Experiment 3, the authors used the 1-parameter logistic model to examine the effect that group differences in the matching criterion has on the apparent effect size. A 1-standard-deviation difference in group means on the matching criterion resulted in an approximately 30% increase in effect size for a given magnitude of DIF. The implications of the inflation of apparent effect size caused by (a) group differences in the matching criterion and (b) choice of IRT model on studies of DIF are discussed.

The Assessment of Culturally and Contextually Relevant Supports (ACCReS) was developed in response to the need for well-constructed instruments to measure teachers’ cultural responsiveness and guide decision-making related to professional development needs. The current study sought to evaluate the presence of differential item functioning (DIF) in ACCReS items and the magnitude of DIF, if detected. With a national sample of 999 grade K-12 teachers in the United States, we examined measurement invariance of ACCReS items in relation to responses from (a) racially and ethnically minoritized (REM) youth and white teachers (teacher race), (b) teachers in schools with 0–50% and 51–100% REM youth (student race), and (c) teachers with <1–5 years of teaching experience and teachers with >5 years of experience. Findings suggested that ACCReS items exhibited negligible levels of DIF. The lack of DIF found provides additional evidence for the validity of scores from the ACCReS to assess teachers’ cultural responsiveness. Furthermore, descriptive analyses revealed that teachers were more likely to agree with items pertaining to their own classroom practice than items related to access to adequate training and support. Results inform implications for future educational and measurement research.

ABSTRACTThe detection of differential item functioning (DIF) has become an important psychometric research topic in recent years. A number of item response theory (IRT) methods for solving this problem have been suggested. A common approach is to calculate some function of the area between item response curves estimated from the subpopulations of interest. While these methods relay overall item level DIF information, they do not indicate the location and magnitude of DIF along the ability continuum. In order to provide these important details, this paper presents a method for producing simultaneous confidence bands for the difference between item response curves. After these bands have been plotted, the size and regions of DIF are easily identified. Implementation considerations and illustrative examples are also given.

A differential item functioning (DIF) detection method for testlet-based data was proposed and evaluated in this study. The proposed DIF model is an extension of a bifactor multidimensional item response theory (MIRT) model for testlets. Unlike traditional item response theory (IRT) DIF models, the proposed model takes testlet effects into account, thus estimating DIF magnitude appropriately when a test is composed of testlets. A fully Bayesian estimation method was adopted for parameter estimation. The recovery of parameters was evaluated for the proposed DIF model. Simulation results revealed that the proposed bifactor MIRT DIF model produced better estimates of DIF magnitude and higher DIF detection rates than the traditional IRT DIF model for all simulation conditions. A real data analysis was also conducted by applying the proposed DIF model to a statewide reading assessment data set.

The Washington Assessment of Risk and Needs of Students (WARNS) is a brief self-report measure designed for schools, courts, and youth service providers to identify student behaviors and contexts related to school truancy. Empirical support for WARNS item invariance between ethnic groups is lacking. This study examined differential item functioning (DIF) to ensure that items on the WARNS function similarly across groups, especially for groups where truancy rates are highest. The item response theory graded response model was used to examine DIF between Caucasian, African American, and Latinx students. DIF was identified in six items across WARNS domains. The DIF amount and magnitude likely will not influence decisions based on total scores. Implications for practice and suggestions for an ecological framework to explain the DIF results are discussed.

Identifying items with differential item functioning (DIF) in an assessment is a crucial step for achieving equitable measurement. One critical issue that has not been fully addressed with existing studies is how DIF items can be detected when data are multilevel. In the present study, we introduced a Lord's Wald test-based procedure for detecting both uniform and non-uniform DIF with polytomous items in the presence of the ubiquitous multilevel data structure. The proposed approach is a multilevel extension of a two-stage procedure, which identifies anchor items in its first stage and formally evaluates candidate items in the second stage. We applied the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm to estimate multilevel polytomous item response theory (IRT) models and to obtain accurate covariance matrices. To evaluate the performance of the proposed approach, we conducted a preliminary simulation study that considered various conditions to mimic real-world scenarios. The simulation results indicated that the proposed approach has great power for identifying DIF items and well controls the Type I error rate. Limitations and future research directions were also discussed.

There has been growing use of ideal point models to develop scales measuring important psychological constructs. For meaningful comparisons across groups, it is important to identify items on such scales that exhibit differential item functioning (DIF). In this study, the authors examined several methods for assessing DIF on polytomous items generated by an ideal point process. Two paradigms (i.e., null hypothesis significance testing [NHST] and effect size quantification) were utilized, and three test statistics (i.e., the log-likelihood ratio [LR], the Akaike information criterion [AIC], and Lord’s chi-square) and two approaches to DIF testing (i.e., the constrained and free baseline methods) were evaluated. In addition, the authors investigated three levels of impact. The results revealed that DIF effect sizes were moderately large for the .50 uniform DIF conditions and small for nonuniform DIF; moreover, the LR test in general yielded the best results. When there was small to moderate impact, the free baseline approach combined with an item linking implementation produced the most satisfactory results.

"As estatísticas englobadas sob a denominação métodos Mantel-Haenszel (MH), por suas simplicidades, baixo custo computacional e bons resultados, são os métodos mais empregadas para detectar o funcionamento diferencial dos itens (DIF). Os métodos MH podem ser usados para detectar o DIF tanto em itens dicotômicos como em itens politômicos, permitindo comparar dois ou mais grupos simultaneamente, e aplicam-se tanto no âmbito da Teoria Clássica dos Testes (TCT) como da Teoria da Resposta ao Item (TRI). Este artigo fornece uma visão completa e integrada dos métodos MH e apresenta um programa que permite aplicar essas estatísticas no estudo do DIF. O programa é gratuito e está disponível em Espanhol, Inglês e Português."

Practitioners are increasingly becoming interested in Differential Item Functioning (DIF) forimproving the validity of test and scale interpretations. Among the statistical procedures availableto assess DIF in dichotomous and polytomous items, the Mantel-Haenszel chi-square (1959) andother standardization proceduresmay beparticularly attractive to practitioners.Some of the statis-tics associated with Mantel-Haenszel procedure can be performed using specific software such asEZ-DIF (Waller, 1998), DIFAS (Penfield, 2005), and MH-DIF (Fidalgo, 1994). However, usingthese programs requires being familiar with the statistics of DIF procedures in order to understandoutput. In addition, key characteristics of the Mantel-Haenszel procedures, such as the matchingstrategy (thick or thin), purification of the matching criteria, and so on, are not currently availablein all statistical software packages. As a result, EASY-DIF was developed to provide easy-to-usesoftware for performing the most common and useful MH and standardization procedures, witha view to guiding practitioners through the analyses and helping them interpret the output.EASY-DIFanalyzesuniformandnonuniformDIFforthetotalsampleandseparatelyforlow-performing and high-performing groups (Clauser, Mazor, & Hambleton, 1994). Users canexplore possible cancellation and amplification DIF effects by establishing different cut scoresfor each group. Up to six different matching strategies based on total score distributions can beimplemented: thinmatching,equal interval, percentage of totalsample, percentage of focalsam-ple, censored matching, and minimum cell frequency (Donoghue & Allen, 1993). In addition,users can use a procedure purification of matching criteria through the selection of a validsubtest.For dichotomous items, EASY-DIF computes the Mantel-Haenszel chi-square (Holland T Mantel & Haenszel, 1959), the Mantel-Haenszel common odds ratio (CamilliSMantel&Haenszel,1959),theMH-DeltaDIF,thestandarderrorofMH-DeltaComputer Program Exchange

Liu and Agresti (1996) proposed a Mantel and Haenszel‐type (1959) estimator of a common odds ratio for several 2 × J tables, where the J columns are ordinal levels of a response variable. This article applies the Liu‐Agresti estimator to the case of assessing differential item functioning (DIF) in items having an ordinal response variable. A simulation study was conducted to investigate the accuracy of the Liu‐Agresti estimator in relation to other statistical DIF detection procedures. The results of the simulation study indicate that the Liu‐Agresti estimator is a viable alternative to other DIF detection statistics.

Assessing differential item functioning (DIF) using the ordinal logistic regression (OLR) model highly depends on the asymptotic sampling distribution of the maximum likelihood (ML) estimators. The ML estimation method, which is often used to estimate the parameters of the OLR model for DIF detection, may be substantially biased with small samples. This study is aimed at proposing a new application of the elastic net regularized OLR model, as a special type of machine learning method, for assessing DIF between two groups with small samples. Accordingly, a simulation study was conducted to compare the powers and type I error rates of the regularized and nonregularized OLR models in detecting DIF under various conditions including moderate and severe magnitudes of DIF (DIF = 0.4 and 0.8), sample size (N), sample size ratio (R), scale length (I), and weighting parameter (w). The simulation results revealed that for I = 5 and regardless of R, the elastic net regularized OLR model with w = 0.1, as compared with the nonregularized OLR model, increased the power of detecting moderate uniform DIF (DIF = 0.4) approximately 35% and 21% for N = 100 and 150, respectively. Moreover, for I = 10 and severe uniform DIF (DIF = 0.8), the average power of the elastic net regularized OLR model with 0.03 ≤ w ≤ 0.06, as compared with the nonregularized OLR model, increased approximately 29.3% and 11.2% for N = 100 and 150, respectively. In these cases, the type I error rates of the regularized and nonregularized OLR models were below or close to the nominal level of 0.05. In general, this simulation study showed that the elastic net regularized OLR model outperformed the nonregularized OLR model especially in extremely small sample size groups. Furthermore, the present research provided a guideline and some recommendations for researchers who conduct DIF studies with small sample sizes.

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel method of differential item functioning (DIF) analysis in computerized adaptive tests (CATs). Each simulated examinee received 25 items from a 75-item pool. A three-parameter logistic item response theory (IRT) model was assumed, and examinees were matched on expected true scores based on their CAT responses and estimated item parameters. The CAT-based DIF statistics were found to be highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. Average DIF statistics and average standard errors also were examined for items with various characteristics. Finally, a study was conducted of the accuracy with which the modified Mantel-Haenszel procedure could identify CAT items with substantial DIF using a classification system now implemented by some testing programs. These additional analyses provided further evidence that the CAT-based DIF procedures performed well. More generally, the results supported the use of IRT-based matching variables in DIF analysis. Index terms: adaptive testing, computerized adaptive testing, differential item functioning, item bias, item response theory.

BackgroundSelf-reported health status measures, like the Short Form 36-item Health Survey (SF-36), can provide rich information about the overall health of a population and its components, such as physical, mental, and social health. However, differential item functioning (DIF), which arises when population sub-groups with the same underlying (i.e., latent) level of health have different measured item response probabilities, may compromise the comparability of these measures. The purpose of this study was to test for DIF on the SF-36 physical functioning (PF) and mental health (MH) sub-scale items in a Canadian population-based sample.MethodsStudy data were from the prospective Canadian Multicentre Osteoporosis Study (CaMos), which collected baseline data in 1996–1997. DIF was tested using a multiple indicators multiple causes (MIMIC) method. Confirmatory factor analysis defined the latent variable measurement model for the item responses and latent variable regression with demographic and health status covariates (i.e., sex, age group, body weight, self-perceived general health) produced estimates of the magnitude of DIF effects.ResultsThe CaMos cohort consisted of 9423 respondents; 69.4% were female and 51.7% were less than 65 years. Eight of 10 items on the PF sub-scale and four of five items on the MH sub-scale exhibited DIF. Large DIF effects were observed on PF sub-scale items about vigorous and moderate activities, lifting and carrying groceries, walking one block, and bathing or dressing. On the MH sub-scale items, all DIF effects were small or moderate in size.ConclusionsSF-36 PF and MH sub-scale scores were not comparable across population sub-groups defined by demographic and health status variables due to the effects of DIF, although the magnitude of this bias was not large for most items. We recommend testing and adjusting for DIF to ensure comparability of the SF-36 in population-based investigations.