Detecting Differential Item Functioning of Polytomous Items for an Ideal Point Response Process

The purpose of this study is to examine the distractors of items that exhibit differential item functioning (DIF) across gender to explain the possible sources of DIF in the context of large-scale tests. To this end, two non-linear logistic regression (NLR) models-based DIF methods (three parameters, 3PL-NLR and four-parameter, 4PL-NLR) were first used to detect DIF items, and the Mantel-Haenszel Delta (MH-Delta) DIF method was used to calculate the DIF effect size for each DIF item. Then, the multinomial log-linear regression (MLR) model and 2-PL nested logit model (2PL-NLM) were applied to items exhibiting DIF with moderate and large DIF effect sizes. The ultimate goals are (a) to examine behaviors of distractors across gender and (b) to investigate if distractors have any impact on DIF effects. DIF results of the Art Section of the General Aptitude Test (GAT-ART) based on both 3PL-NLR and 4PL-NLR methods indicate that only 10 DIF items had moderate to large DIF effects sizes. According to MLR differential distractor functioning (DDF) results, all items exhibited DDF across gender except for one item. An interesting finding of this study is that DIF items related to the verbal analogy and context analysis were in favor of female students, while all DIF items related to the reading comprehension subdomain were in favor of male students, which may signal the existence of content specific DIF or true ability difference across gender. DDF results show that distractors have a significant effect on DIF results. Therefore, DDF analysis is suggested along with DIF analysis since it signals the possible causes of DIF.

This study compared three procedures—the Mantel- Haenszel (MH), the simultaneous item bias (SIB), and the logistic regression (LR) procedures—with respect to their Type I error rates and power to detect nonuniform dif ferential item functioning (DIF). Data were simulated to reflect a variety of conditions: The factors manipulated included sample size, ability distribution differences between the focal and the reference groups, proportion of DIF items in the test, DIF effect sizes, and type of item. 384 conditions were studied. Both the SIB and LR proce dures were equally powerful in detecting nonuniform DIF under most conditions. The MH procedure was not very effective in identifying nonuniform DIF items that had disordinal interactions. The Type I error rates were within the expected limits for the MH procedure and were higher than expected for the SIB and LR proce dures ; the SIB results showed an overall increase of approximately 1% over the LR results. Index terms: differential item functioning, logistic regression statistic, Mantel-Haenszel statistic, nondirectional DIF, simulta neous item bias statistic, SIBTEST, Type I error rate, unidirectional DIF.

Irritability is one of the most common transdiagnostic symptoms. It spans across mood, anxiety, and other disorder classes and is listed as a top problem in outpatient settings. Irritable mood is a symptom for at least 10 disorders in the DSM-5, and it is also developmentally normal across the lifespan. Given this background, we investigated how irritability maps across disorders and ages of children and adolescents.Participants (N = 186) ages 8-16 were enrolled in two ongoing IRB-approved studies investigating the nature of irritability in a transdiagnostic sample of children and adolescents. Participants completed both a child and parent version of the Affective Reactivity Index (ARI-C and ARI-P, respectively) and the Kiddie Schedule for Affective Disorders and Schizophrenia (KSADS) was administered by a trained clinician. Multiple indicators multiple causes (MIMIC) modeling was used to assess for differential item functioning among items in the ARI-P/C compared to diagnoses of major depressive disorder (MDD), generalized anxiety disorder (GAD), oppositional defiant disorder (ODD), and attention-deficit hyperactive disorder (ADHD).ARI-P showed differential item functioning (DIF) with MDD, GAD, and ADHD and participant age. ARI-C showed DIF with ADHD. However, the DIF effect sizes were small.These results indicate child and parent reports of child irritability manifest similarly regardless of specific diagnosis or age. Further understanding of how irritability presents itself across disorders and the lifespan can allow for improved assessment and treatment.

In this article we present results of a Differential Item Functioning (DIF) study using Shealy and Stout's (1993) multidimensionality-based DIF analysis framework. In this framework, differences in test score distributions across different groups of examinees may be a result of multidimensionality if secondary dimensions (not the primary dimension measured by the test) differentially affect examinee performance. Thus, this framework requires both statistical and substantive judgments for identifying potential DIF items and substantiating the causes of DIF, which, as a result, will enhance a comprehensive construct validity argument. In this article, we illustrate step-by-step procedures of multidimensionality-based DIF analyses using LanguEdge reading comprehension test data. Qualitative data from think-aloud verbal protocols were used to generate DIF hypotheses about differential functioning of vocabulary items between two groups of Indo-European and non-Indo-European L2 learners. Statistical Simultaneous Item Bias Test (SIBTEST; Shealy & Stout, 1993) was used to test the DIF hypotheses. The DIF results supported the hypotheses by flagging four uniform DIF items and one crossing DIF item. Post-hoc analyses of the DIF-flagged items were performed by visual inspection of group differences using TestGraf (Ramsay, 2001) and revisiting the qualitative verbal data and analyses of cognate types. The results showed that DIF items with large effect sizes were associated with the following: (a) translation-equivalent cognates; (b) word meaning determined independent of context, and (c) less frequent, more difficult distracter words than those in stems. In light of empirical evidence, the article discusses implications for test development and validation processes.

The purpose of this study was to examine differential item functioning (DIF) in a rubric used to assess middle-school solo and ensemble performances. This study was guided by the following research questions: (a) does measurement equivalence for all items exist when used to measure subgroups of students based on their musical instrument? (b) what patterns of differential item functioning effects exist for items when used to measure subgroups of students based on their musical instrument? and (c) what size of differential item functioning effects exists for items when used to measure subgroups of students based on their musical instrument? In total, 17 adjudicators evaluated 138 middle-school instrumental students (ages 11–13) in the context of a live, formal solo and ensemble performance assessment. Using the Many Facets Rasch Partial Credit measurement model, measurement equivalence for all items did not exist when used to measure subgroups of students based on their musical instrument ([Formula: see text]2 (252) = 634.00, p < .01). Of the 252 total pairwise interactions examined between items and instruments, 57 (22.62%) significant interaction terms were identified. Overall, 26 (10.3%) significant interaction terms demonstrated a moderate to large effect (|DIF| >/= .63 logits), nine (3.6%) significant interaction terms demonstrated a slight to moderate effect (|DIF| = (.43, .63 logits)), and 22 (8.7%) significant interaction terms demonstrated a negligible effect (|DIF| < .43 logits). Implications for the fairness of music performance assessments and improved assessment protocols are discussed.

This study elaborates the Rasch differential item functioning (DIF) model formulation under the marginal maximum likelihood estimation context. Also, the Rasch DIF model performance was examined and compared with the Mantel–Haenszel (MH) procedure in small sample and short test length conditions through simulations. The theoretically known relationship of the DIF estimators between the Rasch DIF model and the MH procedure was confirmed. In general, the MH method showed a conservative tendency for DIF detection rates compared with the Rasch DIF model approach. When there is DIF, the z test (when the standard error of the DIF estimator is estimated properly) and the likelihood ratio test in the Rasch DIF model approach showed higher DIF detection rates than the MH chi-square test for sample sizes of 100 to 300 per group and test lengths ranging from 4 to 39. In addition, this study discusses proposed Rasch DIF classification rules that accommodate statistical inference on the direction of DIF.

Differential Item Functioning (DIF) and bias measurement are often used as synonyms in standardized tests fairness evaluation between individuals belonging to different groups. Recently, Zumbo et al. (2016, 2017) have provided a redefinition of DIF/bias term and proposed a new methodology for DIF/bias detection analysis. The new definition of bias requires attributional reasoning; therefore, there is a need to find a way to control for possible confounding factors. Only by balancing groups with respect to covariates, it is possible to attribute DIF to group membership. Propensity score matching techniques allow to carry out groups balancing and bias is detected if item is flagged as DIF, after balancing groups. The conditional logistic regression is proposed for DIF detection analysis after matching because it allows to consider the data structure generated by matching. The aim of this work is twofold. Firstly, we assess the efficacy and performance of the new methodology in imbalanced groups, comparing its performance to performance of traditional DIF detection methods (Mantel-Haenszel statistic, logistic regression and Lord's χ2). Our research, through a simulation study, shows that the new methodology outperforms traditional DIF detection methods in imbalanced groups in situations of large samples and DIF items presence. Nevertheless, the new methodology suffers to I error inflation for large samples and simulation results suggest that the use of an effect size measure (ΔR2) reduces significantly this issue. Secondly, the proposal methodology is applied to data coming from the large-scale standardized test administered by the National Evaluation Institute for the School System (INVALSI) to evaluate pupils' Italian language and mathematics competencies. The idea is to detect possible DIF items among pupils from different academic tracks. The results reveal that very few items are flagged as DIF, indicating the fairness of INVALSI tests.

Differential item functioning (DIF) is a pernicious statistical issue that can mask true group differences on a target latent construct. A considerable amount of research has focused on evaluating methods for testing DIF, such as using likelihood ratio tests in item response theory (IRT). Most of this research has focused on the asymptotic properties of DIF testing, in part because many latent variable methods require large samples to obtain stable parameter estimates. Much less research has evaluated these methods in small sample sizes despite the fact that many social and behavioral scientists frequently encounter small samples in practice. In this article, we examine the extent to which model complexity—the number of model parameters estimated simultaneously—affects the recovery of DIF in small samples. We compare three models that vary in complexity: logistic regression with sum scores, the 1-parameter logistic IRT model, and the 2-parameter logistic IRT model. We expected that logistic regression with sum scores and the 1-parameter logistic IRT model would more accurately estimate DIF because these models yielded more stable estimates despite being misspecified. Indeed, a simulation study and empirical example of adolescent substance use show that, even when data are generated from / assumed to be a 2-parameter logistic IRT, using parsimonious models in small samples leads to more powerful tests of DIF while adequately controlling for Type I error. We also provide evidence for minimum sample sizes needed to detect DIF, and we evaluate whether applying corrections for multiple testing is advisable. Finally, we provide recommendations for applied researchers who conduct DIF analyses in small samples.

Purpose Differential Item Functioning (DIF), an item malfunctioning, causes Differential Test Functioning (DTF), thus biasing questionnaire measures. The current study evaluates the relationship between DIF and DTF for the Barthel Index and the Functional Independence Measure, likely the most used disability measures. The aim is to understand under which conditions DIF can be ignored as its DTF is negligible. Methods A simulation study was run. Disability measures were obtained for the Barthel Index and FIM motor domain using Rasch analysis with previously published item calibrations. Several DIF scenarios have been assessed. DTF was tolerable if ≤0.50 logits. Results Simulations showed that the larger the DIF, the larger the DTF and that, keeping the overall DIF constant, the total number of items with DIF does not affect DTF. DIF of the items with the lowest or highest calibrations is the most dangerous. The DIF of central items should be so massive to matter in DTF terms that it is unlikely to happen in practice. The FIM robustness to DIF is better than that of the Barthel Index. Conclusions The FIM and the Barthel Index show remarkable robustness to DIF. Thanks to this feature, sample invariant, generalisable disability measures are available.

ABSTRACTThis study examined the relationship of differential item functioning (DIF) to item difficulty on the Scholastic Aptitude Test (SAT). The data comprise verbal and mathematical item statistics from nine recent administrations of the SAT. In general, item difficulty is related to DIF. The nature of that relationship appears to be independent of the choice of DIF index (either the Mantel‐Haenszel or the standardization approach) as well as of test form. However, the relationship was dependent on the particular group comparison and on both the test sections and the item type being analyzed.The relationship was strong for each of the racial and ethnic group contrasts—in which black, Hispanic, and Asian American examinees were compared in turn with white examinees—but was weak for the female and male examinee contrast. The relationship also appeared stronger on the verbal sections than on the mathematical sections. The relationship is such that more difficult items tended to exhibit positive DIF (DIF favored the focal group over the white reference group). On the verbal sections, only the reading comprehension item type (with the smallest observed range in item difficulty) failed to exhibit a strong relationship.Another index, the standardized difference in percentage omit (DIFPOM), correlated very highly (negatively) with DIF. Differential omission refers to a relative difference in omit rates between groups matched in ability. In fact, DIFPOM was consistently a better predictor of DIF in most models than was item difficulty. The relationship between DIF and DIFPOM held up across all four comparisons, including gender. It was also present in the mathematical sections with nearly the same magnitude exhibited in the verbal sections.Although DIF and DIFPOM are mathematically dependent measures, it was proposed that DIFPOM may be partly responsible for the relationship between DIF and item difficulty. To what extent DIF is a consequence of differential omission and to what extent differential omission is a manifestation of DIF is problematic. Nonetheless, the presence of differential omission on a test has the potential to influence DIF indices and therefore should be an important concern especially for formula‐scored tests, where omission occurs often on difficult items.Among other findings is that Hispanic and black focal groups tended to omit differentially less than did the white reference groups. For Asian American examinees, the reverse holds. For females and males, the direction depends on the test sections. In general, groups that omitted differentially less experienced a relative advantage (high‐positive DIF values) on the more difficult items, as measured by the DIF indices studied here (which treat omits as wrong in their calculation).

This study aims to identify the sources of Differential Item Functioning (DIF) using the Mixture Ordinal Logistic Regression (Mixture OLR) method, a contemporary approach for detecting DIF. To analyze mathematics self-efficacy, data from a scale comprising 9 items were obtained from 5000 8th-grade students as part of the ABIDE-2016 project. The study compared the presence and extent of DIF by gender using two methods and examined the sources of DIF for items displaying DIF with Mixture OLR. The OLR analysis revealed that five items exhibited DIF at level A, but no DIF was observed with Mixture OLR. Furthermore, it was found that the magnitude of DIF (B) for an item showing DIF at level A changed due to Mixture OLR. The results indicate that the homogeneity of the data affects both the number of items displaying DIF and the magnitude of DIF. Three items did not exhibit significant DIF according to both methods. One significant finding in the study highlights the moderating effect of latent class on item 8, where DIF was observed. However, the source of DIF was not related to gender but rather stemmed from different ecological variables. An analysis of latent class characteristics revealed that students with significant DIF effects had lower absenteeism and fewer siblings. Additionally, students in this class had greater access to books at home and participated in more out-of-school mathematics courses. Surprisingly, these students were found to engage less in social activities. Various factors can influence how students respond to test items, potentially leading to DIF. These factors may include cultural background, gender, social environment, school, teacher, family interest/attitude toward the child, and home climate. Therefore, when developing and administering tests, it is crucial to test for data homogeneity and consider the impact of these variables, in addition to gender, to identify any sources of DIF in test items.

Assessing differential item functioning (DIF) using the ordinal logistic regression (OLR) model highly depends on the asymptotic sampling distribution of the maximum likelihood (ML) estimators. The ML estimation method, which is often used to estimate the parameters of the OLR model for DIF detection, may be substantially biased with small samples. This study is aimed at proposing a new application of the elastic net regularized OLR model, as a special type of machine learning method, for assessing DIF between two groups with small samples. Accordingly, a simulation study was conducted to compare the powers and type I error rates of the regularized and nonregularized OLR models in detecting DIF under various conditions including moderate and severe magnitudes of DIF (DIF = 0.4 and 0.8), sample size (N), sample size ratio (R), scale length (I), and weighting parameter (w). The simulation results revealed that for I = 5 and regardless of R, the elastic net regularized OLR model with w = 0.1, as compared with the nonregularized OLR model, increased the power of detecting moderate uniform DIF (DIF = 0.4) approximately 35% and 21% for N = 100 and 150, respectively. Moreover, for I = 10 and severe uniform DIF (DIF = 0.8), the average power of the elastic net regularized OLR model with 0.03 ≤ w ≤ 0.06, as compared with the nonregularized OLR model, increased approximately 29.3% and 11.2% for N = 100 and 150, respectively. In these cases, the type I error rates of the regularized and nonregularized OLR models were below or close to the nominal level of 0.05. In general, this simulation study showed that the elastic net regularized OLR model outperformed the nonregularized OLR model especially in extremely small sample size groups. Furthermore, the present research provided a guideline and some recommendations for researchers who conduct DIF studies with small sample sizes.

"As estatísticas englobadas sob a denominação métodos Mantel-Haenszel (MH), por suas simplicidades, baixo custo computacional e bons resultados, são os métodos mais empregadas para detectar o funcionamento diferencial dos itens (DIF). Os métodos MH podem ser usados para detectar o DIF tanto em itens dicotômicos como em itens politômicos, permitindo comparar dois ou mais grupos simultaneamente, e aplicam-se tanto no âmbito da Teoria Clássica dos Testes (TCT) como da Teoria da Resposta ao Item (TRI). Este artigo fornece uma visão completa e integrada dos métodos MH e apresenta um programa que permite aplicar essas estatísticas no estudo do DIF. O programa é gratuito e está disponível em Espanhol, Inglês e Português."

Transgender and gender diverse (TGD) individuals demonstrate elevated risk for eating disorders (EDs) and ED symptoms compared to cisgender individuals. Rates of ED symptoms in TGD populations may be inaccurately estimated due to item bias or interpretation differences on assessments of ED diagnostic criteria. Differential item functioning (DIF) can indicate if individuals respond differently to assessment items at the same latent level of ED psychopathology. We aimed to assess for DIF in the ED diagnostic criteria, represented by items on the Eating Disorder Diagnostic Scale (EDDS), among cisgender men/women and TGD individuals. We expected small-to-moderate DIF between cisgender men/women. We hypothesized that items assessing body image, shape, and fear of fatness would demonstrate significant DIF with moderate-to-large effect sizes in TGD individuals relative to cisgender men/women. College students identifying as cisgender men (n = 1240), cisgender women (n = 3750), and TGD individuals (n = 271) completed the EDDS. DIF was tested using likelihood ratio tests. The magnitude of the effect was estimated to identify moderate-to-large effects. Results indicated large, significant response bias to items assessing weight concerns, psychosocial impairment, compensatory behavior frequency, and select binge-eating symptoms. Specifically, cisgender men and TGD individuals weresignificantly less likely to endorse these ED symptoms than cisgender women at the same latent level of ED psychopathology. This study was the first to investigate DIF in an assessment of ED diagnostic criteria in TGD individuals. Our findings identified a need for unbiased assessments of ED diagnostic criteria to more accurately assess ED symptoms across gender groups.

Studies on the differential item functioning (DIF) are usually considered in the context of manifest groups. Recently, with the increase in the number of analyses conducted with mixture models, investigating the situations that cause differences between groups has come to the forefront. In addition, it is considered important to examine the DIF with mixture models in which levels are also handled. In this study, it is aimed to compare the results of the multilevel mixture item response theory (MMIRT) model and the mixture item response theory (MIRT) model and the results of the DIF analyses based on the manifest groups. The research sample consists of students who answered the second booklet in the electronic Trends in International Mathematics and Science Study (eTIMSS) 2019 and coded their gender. The answers given to 15 items were analyzed with the Mantel Haenszel (MH) method for the gender variable according to the manifest groups, and with the selection of the most appropriate models by varying the number of groups and the number of levels according to the MIRT model and the MMIRT model. DIF analyses of the obtained latent groups were also performed with the MH method. In the light of the findings, the number of items displaying DIF in both the MIRT model and the MMIRT model is higher than the manifest groups. While only one item displayed DIF in the analysis according to gender, 14 items displayed DIF according to the MIRT model and seven items displayed DIF according to the MMIRT model. There is not a complete overlap in the number of DIF items and DIF effect sizes found as a result of the MIRT model and MMIRT model analyses. For this reason, a level analysis should be conducted before the analyses and if there is multi-levelness, the analyses should be conducted by taking this situation into consideration.