Detecting Differential Item and Step Functioning with Rating Scale and Partial Credit Trees

Ensuring fairness in educational and psychological assessments is critical, particularly in detecting differential item functioning (DIF), where items perform differently across subgroups. The Rasch tree method, a model-based recursive partitioning approach, is an innovative and flexible DIF detection tool that does not require the pre-specification of focal and reference groups. However, research systematically examining its performance under realistic measurement conditions, such as when multiple DIF items do not consistently favor one subgroup, is limited. This study builds on prior research, evaluating the Rasch tree method's ability to detect DIF by investigating the impact of DIF balance, along with other key factors such as DIF magnitude, sample size, test length, and contamination levels. Additionally, we incorporate the Educational Testing Service effect size heuristic as a criterion to compare the DIF detection rate performance with only statistical significance. Results indicate that the Rasch tree has better true DIF detection rates under balanced DIF conditions and large DIF magnitudes. However, its accuracy declines when DIF is unbalanced and the percentage of DIF contamination increases. The use of an effect size reduces the detection of negligible DIF. Caution is recommended with smaller samples, where detection rates are the lowest, especially for larger DIF magnitudes and increased DIF contamination percentages in unbalanced conditions. The study highlights the strengths and limitations of the Rasch tree method under a variety of conditions, underscores the importance of the impact of DIF group imbalance, and provides recommendations for optimizing DIF detection in practical assessment scenarios.

Type I error rates of the likelihood ratio test for the detection of differential item functioning (DIF) were investigated using monte carlo simulations. The graded response model with five ordered categories was used to generate datasets of a 30-item test for samples of 300 and 1,000 simulated examinees. All DIF comparisons were simulated by randomly pairing two groups of examinees. Three different sample size combinations of reference and focal groups were simulated under two ability matching conditions. For each of the six combinations of sample sizes by ability matching conditions, 100 replications of DIF detection comparisons were simulated. Item parameter estimates and likelihood values were obtained by marginal maximum likelihood estimation using the computer program MULTILOG. Irpe I eryor rates of the likelihood ratio test statistics for all six combinations of sample sizes and ability matching conditions were within theoretically expected values at each of the nominal alpha levels considered.

This study compares three procedures for the detection of differential item functioning (DIF) under item response theory (IRT): (a) Lord's chi-square, (b) Raju's area measures, and (c) the likelihood ratio test. Relations among the three procedures and some practical considerations, such as linking metrics and scale purification, are discussed. Data from two forms of a university mathematics placement test were analyzed to examine the congruence among the three procedures. Results indicated that there was close agreement among the three DIF detection procedures.

The study of measurement invariance in polytomous items that targets individual score levels is known as differential step functioning (DSF). The analysis of DSF requires the creation of a set of dichotomizations of the item response variable. There are two primary approaches for creating the set of dichotomizations to conduct a DSF analysis: the adjacent categories approach, and the cumulative approach. To date, there is limited research on how these two approaches compare within the context of DSF, particularly as applied to a real data set. This study evaluated the results of a DSF analysis using both dichotomization schemes in order to determine if the two approaches yield similar results. The results revealed that the two approaches generally led to consistent results, particularly in the case where DSF effects were negligible. However, when significant DSF effects were present, the two approaches occasionally led to differing conclusions.

Liu and Agresti (1996) proposed a Mantel and Haenszel‐type (1959) estimator of a common odds ratio for several 2 × J tables, where the J columns are ordinal levels of a response variable. This article applies the Liu‐Agresti estimator to the case of assessing differential item functioning (DIF) in items having an ordinal response variable. A simulation study was conducted to investigate the accuracy of the Liu‐Agresti estimator in relation to other statistical DIF detection procedures. The results of the simulation study indicate that the Liu‐Agresti estimator is a viable alternative to other DIF detection statistics.

The study's objective was to identify the differential item functioning (DIF) or bias using item response theory. This research is exploratory research with a quantitative approach. In this study, the data used is a fundamental physics test, specifically motion one and two dimensions, dynamics and work and energy concept, with 25 items in the two-tier multiple choices in form. Two hundred and fifty-four prospective physics teachers at West Java and Banten provinces, Indonesia, consisting of 103 males (reference group) and 151 females (focal group). The data analysis includes two stages: conformity of the logistic model parameters and item characteristics, followed by DIF detection. The model suitability test shows the appropriate parameter is 2PL. DIF detection has three methods: simple area indices, Raju and the Lord method. Research results show two things. First, the model fit test shows that the data best fit the 2-parameter or 2-PL logistic parameter model, and all items have good characteristics. Second, the most significant item containing uniform DIF was number 5, followed by nonuniform DIF items 9 and 25. These three items were biased because the chances of answering correctly for male and female students differed. Significantly, this means that this item favors one group. This study illustrates the importance of constructing instrument items so that all test takers, both male and female, have an equal chance to answer the question correctly. Keywords: Differential item functioning, physics test, simple area indices, Raju method, Lord method;

This software program computes the Mantel-Haenszel statistic that is used for detecting differential item functioning (DIF). The main features of the program are (a) the capability of using total test score or a "purified" test score as the internal criterion for matching examinees, (b) using an external criterion for matching examinees if one is available, (c) complete descriptive statistics on the test data, (d) complete reference and focal group performance data for items flagged as DIF, (e) additional DIF statistics, and(f) availability of mainframe and PC versions. The program can currently handle up to 100 items. There is no limit on the number of examinees per group.

Background: Early detection of community health risk factors such as stress is of great interest to health policymakers, but representative data collection is often expensive and time-consuming. It is important to investigate the use of alternative means of data collection such as crowdsourcing platforms.Methods: An online sample of Amazon Mechanical Turk (MTurk) workers (N = 500) filled out, for themselves and their child, demographic information and the 10-item Perceived Stress Scale (PSS-10), designed to measure the degree to which situations in one’s life are appraised as stressful. Internal consistency reliability of the PSS-10 was examined via Cronbach’s alpha. Analysis of variance (ANOVA) was utilized to explore trends in the average perceived stress of both adults and their children. Last, Rasch trees were utilized to detect differential item functioning (DIF) in the set of PSS-10 items.Results: The PSS-10 showed adequate internal consistency reliability (Cronbach’s alpha = 0.73). ANOVA results suggested that stress scores significantly differed by education (p = 0.024), employment status (p = 0.0004), and social media usage (p = 0.015). Rasch trees, a recursive partitioning technique based on the Rasch model, indicated that items on the PSS-10 displayed DIF attributable to physical health for adults and social media usage for children.Conclusion: The key conclusion is that this data collection scheme shows promise, allowing public health officials to examine health risk factors such as perceived stress quickly and cost effectively.

In 2009, DeMars stated that when impact exists there will be Type I error inflation, especially with larger sample sizes and larger discrimination parameters for items. One purpose of this study is to present the patterns of Type I error rates using Mantel–Haenszel (MH) and logistic regression (LR) procedures when the mean ability between the focal and reference groups varies from zero to one standard deviation. The findings can be used as guides for alpha adjustment when using MH or LR methods when impact exists. A second purpose is to better understand the conditions that cause Type I error rates to inflate. The results indicate that inflation can be controlled even in the presence of large ability differences and with large samples.

The direct assessment of the writing ability of 2000 randomly selected secondary school students was performed using the partial credit model. The effects on item parameter estimates of rating scale and type of writing sample were investigated. The rating scales used were a sum and an averaging of the individual raters' ratings and the types of items were expository and narrative. Results showed that expository items tended to provide more information for high ability examinees than did narrative items and that the sum holistic rating method yielded greater information than did the average holistic rating scale. Additional analyses of the rating scales used as well as implications for test construction are discussed.

Often tests includes partial credit items. The computation of “measures” for a partial credit test is harder to understand for those learning Rasch than when analysing a multiple choice test or a rating scale survey. This chapter presents both the thinking of what it means to measure with partial credit items and the steps needed to conduct a Rasch analysis of partial credit test data. Links are made to Rasch Analysis in the Human Sciences analyses of dichotomous test data, analyses of rating scale data, and Wright Maps for both dichotomous tests and rating scale data. In this chapter partial credit analyses are presented in which all partial credit items of a potential maximum point total have the same rating scale structure. Also partial credit analyses are presented in which items are allowed to have their own unique rating scale structure.

Two step-down tests for equality of covariance matrices

The assessment of programming skills must be able to measure holistically the theoretical abilities and programming skills. Performance assessment is seen as an appropriate assessment to measure programming ability because it is able to evaluate authentic ability through the process carried out or the product produced. The problem that often arises is performance assessment testing to measure programming capabilities. On this occasion, the Partial Credit Model (PCM) was tried to test performance assessments to measure programming capabilities. PCM is a development of the Rasch model, which applies one parameter, namely the item difficulty index. If the Rasch model is applied to dichotomous items, then PCM is applied to politomic items. PCM assumes that all items have the same difference power. The category score on the PCM shows the per-step score for correctly completing the item according to the scoring rubric developed. The probability of each test taker is estimated by calculating the probability of answering each step in completing an item. There were five items that were tested in two parallel classes in relatively different times. The test results showed a difference in probability in the two classes, but the difference was not too far away. So, the performance assessment was precise enough to measure programming capability, and the test results were quite precise when tested with PCM.

Maclean et al. (1976) applied a specific Box-Cox transformation to test for mixtures of distributions against a single distribution. Their null hypothesis is that a sample of n observations is from a normal distribution with unknown mean and variance after a restricted Box-Cox transformation. The alternative is that the sample is from a mixture of two normal distributions, each with unknown mean and unknown, but equal, variance after another restricted Box-Cox transformation. We developed a computer program that calculated the maximum likelihood estimates (MLEs) and likelihood ratio test (LRT) statistic for the above. Our algorithm for the calculation of the MLEs of the unknown parameters used multiple starting points to protect against convergence to a local rather than global maximum. We then simulated the distribution of the LRT for samples drawn from a normal distribution and five Box-Cox transformations of a normal distribution. The null distribution appeared to be the same for the Box-Cox transformations studied and appeared to be distributed as a chi-square random variable for samples of 25 or more. The degrees of freedom parameter appeared to be a monotonically decreasing function of the sample size. The null distribution of this LRT appeared to converge to a chi-square distribution with 2.5 degrees of freedom. We estimated the critical values for the 0.10, 0.05, and 0.01 levels of significance.

The aim of this study is to validate the 36 items related to emotional problems from the Korean Children and Youth Panel Survey (KCYPS) using Rasch models. Specifically, the analysis focused on the 2018 data from high school sophomores in the 5th wave of KCYPS, employing both the Partial Credit Model (PCM) and the Rating Scale Model (RSM). The findings are as follows. First, all 36 items were found to fit both the PCM and RSM, indicating that the 4-point response categories were appropriate for both models. Second, the PCM, which assumes non-equidistant thresholds, demonstrated a statistically better fit than the RSM, which assumes equidistant thresholds. Third, items related to somatization and depression were particularly effective in capturing threshold shifts of emotional problems. Based on these results, the study discusses modeling options for threshold estimation and proposes future research topics, including the application of longitudinal item response theory to KCYPS data.