Comparing Differential Item Functioning Based On Multilevel Mixture Item Response Theory, Mixture Item Response Theory And Manifest Groups

This study investigates the impact of item parameter drift (IPD) on parameter and ability estimation when the underlying measurement model fits a mixture distribution, thereby violating the item invariance property of unidimensional item response theory (IRT) models. An empirical study was conducted to demonstrate the occurrence of both IPD and an underlying mixture distribution using real-world data. Twenty-one trended anchor items from the 1999, 2003, and 2007 administrations of Trends in International Mathematics and Science Study (TIMSS) were analyzed using unidimensional and mixture IRT models. TIMSS treats trended anchor items as invariant over testing administrations and uses pre-calibrated item parameters based on unidimensional IRT. However, empirical results showed evidence of two latent subgroups with IPD. Results also showed changes in the distribution of examinee ability between latent classes over the three administrations. A simulation study was conducted to examine the impact of IPD on the estimation of ability and item parameters, when data have underlying mixture distributions. Simulations used data generated from a mixture IRT model and estimated using unidimensional IRT. Results showed that data reflecting IPD using mixture IRT model led to IPD in the unidimensional IRT model. Changes in the distribution of examinee ability also affected item parameters. Moreover, drift with respect to item discrimination and distribution of examinee ability affected estimates of examinee ability. These findings demonstrate the need to caution and evaluate IPD using a mixture IRT framework to understand its effects on item parameters and examinee ability.

BackgroundIn low-stakes educational assessments, test takers might show a performance decline (PD) on end-of-test items. PD is a concern in educational assessments, especially when groups of students are to be compared on the proficiency variable because item responses gathered in the groups could be differently affected by PD. In order to account for PD, mixture item response theory (IRT) models have been proposed in the literature.MethodsIn this article, multigroup extensions of three existing mixture models that assess PD are compared. The models were applied to the mathematics test in a large-scale study targeting school track differences in proficiency.ResultsDespite the differences in the specification of PD, all three models showed rather similar item parameter estimates that were, however, different from the estimates given by a standard two parameter IRT model. In addition, all models indicated that the amount of PD differed between tracks, in that school track differences in proficiency were slightly reduced when PD was accounted for. Nevertheless, the models gave different estimates of the proportion of students showing PD, and differed somewhat from each other in the adjustment of proficiency scores for PD.ConclusionsMultigroup mixture models can be used to study how PD interacts with proficiency and other variables to provide a better understanding of the mechanisms behind PD. Differences between the presented models with regard to their assumptions about the relationship between PD and item responses are discussed.

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel method of differential item functioning (DIF) analysis in computerized adaptive tests (CATs). Each simulated examinee received 25 items from a 75-item pool. A three-parameter logistic item response theory (IRT) model was assumed, and examinees were matched on expected true scores based on their CAT responses and estimated item parameters. The CAT-based DIF statistics were found to be highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. Average DIF statistics and average standard errors also were examined for items with various characteristics. Finally, a study was conducted of the accuracy with which the modified Mantel-Haenszel procedure could identify CAT items with substantial DIF using a classification system now implemented by some testing programs. These additional analyses provided further evidence that the CAT-based DIF procedures performed well. More generally, the results supported the use of IRT-based matching variables in DIF analysis. Index terms: adaptive testing, computerized adaptive testing, differential item functioning, item bias, item response theory.

The purpose of this study is to carry out differential item functioning (DIF) analysis for content areas of a reading comprehension subtest using four area indices within Item Response Theory (IRT) framework. The differences in the magnitudes of the area indices were compared based on the subject areas. The DIF analysis was carried out across gender groups only. The item level data of the English reading comprehension subtest were gathered from English Language Achievement exam done in School of Foreign Languages, Ege University, Turkey, in 2013. A sample of 2,117 examinees (1,011 males and 1,116 females) was randomly selected. For the DIF analysis, (a) an IRT model for the item characteristic curves was specified; (b) model-data-fit was investigated for the selected IRT model;(c) Item characteristic curves were separately computed for each group on a common scale; finally, (d) indices indicating the degree of DIF on each item were computed. The results of the study indicated that both un-weighted and weighted area indices showed non-uniformity in DIF in the item characteristic curves in reading comprehension subtest in most cases. A significant correlation was observed between un-weighted and weighted area indices. Key words: Item characteristic curve, item bias, differential item functioning (DIF).

Aptitude test scores are typically interpreted similarly for examinees with the same overall score. However, research has found evidence of examinee differences in strategies, as well as in the continued application of appropriate procedures during testing. Such differences can impact the correlates of test scores, making similar interpretations for equivalent scores questionable. This study presents some item response theory (IRT) models that are relevant to identifying examinee differences in strategies and understanding of test-taking procedures. First, mixture IRT models that identify latent classes of examinees with different patterns of item responses are considered; these models have long been available but unfortunately are not routinely applied. Strategy differences between the classes can then be studied separately by modeling the response patterns with cognitive complexity variables within each class. Secondly, novel psychometric approaches that leverage response time information (in particular, response time residuals) in order to identify both inter and intraindividual variability in response processes are considered. In doing so, a general method for evaluating threats to validity is proposed. The utility of the approach, in terms of providing more interpretable performance estimates and improving the administration of psychological measurement instruments, is then demonstrated with an empirical example.

Differential item functioning (DIF) analysis can be used to great advantage in second language (L2) assessments. This study examined the differences in performance on L2 test items between groups from different first language backgrounds and suggested ways of improving L2 assessments. The study examined DIF on L2 (Hebrew) test items for two language groups: Arabic speakers and Russian speakers. Many items (42%) functioned differentially, posing a potential threat to validity. Most problematic were the sentence completion items, which are more loaded on vocabulary and grammar then the other item types. To validate our study results, two special test forms—non-DIF and large DIF—were constructed, based on the results of the study. These test forms were administered during the years 2004–2005. The findings validated the results of the main study. To reduce the number of DIF items in operational tests, we suggest performing DIF analysis on pilot items and changing the ratios of item types included in the test, while preserving the defined test construct.

Reading comprehension skills are critical predictors of students’ academic performance. This study, based on PIRLS 2021 paper-based test data, investigates whether multiple-choice reading comprehension items function differently according to students’ socioeconomic status (SES) in the samples of Türkiye, Ireland, and South Africa. A total of 21 multiple-choice items drawn from the following passages were analyzed: Where is the Honey?, Pemba Sherpa, and Marie Curie. The analyses of Differential Item Functioning (DIF) were conducted by using Mantel–Haenszel (MH) test, Logistic Regression (LR), and Lord’s Chi-Square method. The findings revealed that the highest number of items exhibiting Differential Item Functioning (DIF) was observed in Turkey, while the lowest was found in South Africa. Additionally, Logistic Regression was identified as the method that detected the greatest number of DIF items. Observed differences in item functioning by SES varied significantly across countries, passages, and DIF detection methods. Notably, the number of DIF items identified in the Turkish sample was relatively high. Furthermore, Logistic Regression (LR) demonstrated higher sensitivity in detecting DIF compared to the other methods employed. The results underscore the importance of evaluating measurement fairness not only from a statistical standpoint but also within contextual, cultural, and cognitive dimensions.

Detection of differential item functioning (DIF) is needed in the development of tests to obtain useful items. The Mantel-Haenszel method and standardization are tools for DIF detection based on classical theory assumptions. The study was conducted to highlight the sensitivity and accuracy between the Mantel-Haenszel method and the standardization method in DIF detection. Simulation design (a) test participants consisted of 1000 responses in both the reference and focus groups, (b) the size of the proportion of DIF (0.1; 0.25; 0.50; and 0.75), and (c) the length of the multiple choice test with 40 choices the answer. Research shows that the Mantel-Haenszel method has the same sensitivity as the standardization method in DIF proportions of 10% and 25%, however, when the ratio of DIF proportions above 25% the standardization method is less sensitive, and conversely the sensitivity of the Mantel-Haenszel method increases. The standardization method has higher accuracy than the Mantel-Haenszel method in the DIF proportion of 10%, however, when the size of the DIF proportion above 10% the accuracy of the standardization method decreases, the accuracy of the Mantel-Haenszel method is higher than the standardization method. Thus, if the ratio of DIF is detected by the standardization method of (≤0.10), then the results of the standardization method are preferred as a reference. Conversely, if the proportion of DIF detected by the standardization method is (≥0.10), then the result of the Mantel-Haenszel method is chosen as a reference.

In this study the authors used differential item functioning analysis to examine if there were items in the Mental, Emotional, and Bodily Toughness Inventory functioning differently across gender and athletic membership. A total of 444 male (56.3%) and female (43.7%) participants (30.9% athletes and 69.1% non-athletes) responded to the Mental, Emotional, and Bodily Toughness Inventory items. Using Mantel–Haenszel and simultaneous item bias test methods, 43 items were analyzed for differential item functioning. Four Mental, Emotional, and Bodily Toughness Inventory items were identified as large differential item functioning items by both Mantel–Haenszel and simultaneous item bias test methods, where item 21 favored non-athletes, item 40 favored athletes, item 2 favored males, and item 17 favored females. Athletic membership differential item functioning disappeared, whereas gender differential item functioning still existed at the scale level. Overall, there are gender and athletic membership differential item functioning items in the Mental, Emotional, and Bodily Toughness Inventory, but only gender differential item functioning still exists at the scale level. Thus, conclusions regarding gender differences in mental toughness should be made with caution when using total Mental, Emotional, and Bodily Toughness Inventory scores.

"As estatísticas englobadas sob a denominação métodos Mantel-Haenszel (MH), por suas simplicidades, baixo custo computacional e bons resultados, são os métodos mais empregadas para detectar o funcionamento diferencial dos itens (DIF). Os métodos MH podem ser usados para detectar o DIF tanto em itens dicotômicos como em itens politômicos, permitindo comparar dois ou mais grupos simultaneamente, e aplicam-se tanto no âmbito da Teoria Clássica dos Testes (TCT) como da Teoria da Resposta ao Item (TRI). Este artigo fornece uma visão completa e integrada dos métodos MH e apresenta um programa que permite aplicar essas estatísticas no estudo do DIF. O programa é gratuito e está disponível em Espanhol, Inglês e Português."

The purpose of this study is to examine the distractors of items that exhibit differential item functioning (DIF) across gender to explain the possible sources of DIF in the context of large-scale tests. To this end, two non-linear logistic regression (NLR) models-based DIF methods (three parameters, 3PL-NLR and four-parameter, 4PL-NLR) were first used to detect DIF items, and the Mantel-Haenszel Delta (MH-Delta) DIF method was used to calculate the DIF effect size for each DIF item. Then, the multinomial log-linear regression (MLR) model and 2-PL nested logit model (2PL-NLM) were applied to items exhibiting DIF with moderate and large DIF effect sizes. The ultimate goals are (a) to examine behaviors of distractors across gender and (b) to investigate if distractors have any impact on DIF effects. DIF results of the Art Section of the General Aptitude Test (GAT-ART) based on both 3PL-NLR and 4PL-NLR methods indicate that only 10 DIF items had moderate to large DIF effects sizes. According to MLR differential distractor functioning (DDF) results, all items exhibited DDF across gender except for one item. An interesting finding of this study is that DIF items related to the verbal analogy and context analysis were in favor of female students, while all DIF items related to the reading comprehension subdomain were in favor of male students, which may signal the existence of content specific DIF or true ability difference across gender. DDF results show that distractors have a significant effect on DIF results. Therefore, DDF analysis is suggested along with DIF analysis since it signals the possible causes of DIF.

Differential item functioning (DIF) detection was handled based on “Mantel-Haenszel (MH)”, “Simultaneous item bias test (SIBTEST)”, “Lord's chi-square”, “Raju's area” methods when item purification was performed or item purification was not performed using real data in current study. After detecting gender-related DIF, expert opinions were taken for bias study. It is important to conduct the gender bias research in the English test when purification is performed and when purification is not performed, as there were DIF studies, but there were not completely similar bias studies in the literature. The sample of the research consists of 7389 students who took the “Transition from Primary to Secondary Education Exam (TPSEE, referred to as “TEOG” in Turkey)” administered in April 2017. When gender-related DIF analysis was performed with the four methods, the results were found to differ partially. DIF analysis results differed in the different conditions item purification was performed or not. Detection of DIF was indicative of possible bias. In the second stage of the study, the opinions of seven experts were taken for item 11, for which DIF was detected at least at B level based on MH, SIBTEST. As a result of expert opinion, it was found that there was no item bias according to gender in any item in the English test. It is recommended that similar bias studies can be conducted for test developers to be aware of the features that may lead to item bias and to construct unbiased items.

Inattentive responses can threaten measurement quality, yet they are common in rating- or Likert-scale data. In this study, we proposed a new mixture item response theory model to distinguish inattentive responses from normal responses so that test validity can be ascertained. Simulation studies demonstrated that the parameters of the new model were recovered fairly well using the Bayesian methods implemented in the freeware WinBUGS, and fitting the new model to data that lacked inattentive responses did not result in severely biased parameter estimates. In contrast, ignoring inattentive responses by fitting standard item response theory models to data containing inattentive responses yielded seriously biased parameter estimates and a failure to distinguish inattentive participants from normal participants; the person-fit statistic lz was also unsatisfactory in identifying inattentive responses. Two empirical examples demonstrate the applications of the new model.

The validity of inferences from patient-reported outcome measure (PROM) scores can be confounded by differential item functioning (DIF). DIF occurs when there is heterogeneity in how patients respond to and interpret questions about their health, despite having the same underlying health status. Ignoring the effects of DIF could lead to inaccurate interpretations and misinformed clinical decisions resulting in compromised healthcare delivery. Tree-based item response theory (IRT) models are recommended as an alternative class of methods for analyzing PROMs because they offer a robust approach for identifying DIF when covariates associated with DIF are unknown a priori. This paper introduces a web application developed using R Shiny, which enables users to implement tree-based IRT models for DIF assessment in potentially heterogeneous populations. The app provides flexible model specifications, visualization tools, and customizable settings to accommodate various data types and research needs. A practical tutorial is included, guiding users through the application interface, data preparation, model selection, and interpretation of results. The web application (https://ucalgary-pcma-lab.shinyapps.io/tree_based_dif_analysis/) offers interactive data upload in .CSV and .XLSX data formats. Recommendations are provided for selecting model parameters within the app based on the results of previous simulation studies. The web app tests for DIF on dichotomous- and polytomous-scored items. The coefficients, item parameters, and plots provide insights into potential sources of DIF. This web application provides a user-friendly, interactive, innovative, easily accessible, and valuable tool for clinicians, applied health researchers, and analysts seeking to understand sample heterogeneity due to DIF in PROM data.

The asymptotic power of the Mantel-Haenszel (MH) test for the differential item function (DIF) is derived. The formula describes the behavior of the power when the number of items is large, so that the measured latent trait can be considered as the matching variable in the MH test. As shown in the derived formula, the power is related to the sample size, effect size of DIF, the item response function (IRF), and the distribution of the latent trait in the reference and the focal groups. The formula provides an approximation of the power of the MH test in practice and thus provides a guideline for DIF detection in practice. It also suggests analytical explanations of the behavior of the MH test as observed in many previous simulation studies. Based on the formula, this study shows how to conduct the sample size calculation. The power of MH test under some practical models such as the two-parameter logistic (2PL) and three-parameter logistic (3PL) item response theory (IRT) models is discussed.