Cross-Cultural Validation of the Mathematics Construct and Attribute Profiles: A Differential Item Functioning Approach

The purpose of this study is to examine the distractors of items that exhibit differential item functioning (DIF) across gender to explain the possible sources of DIF in the context of large-scale tests. To this end, two non-linear logistic regression (NLR) models-based DIF methods (three parameters, 3PL-NLR and four-parameter, 4PL-NLR) were first used to detect DIF items, and the Mantel-Haenszel Delta (MH-Delta) DIF method was used to calculate the DIF effect size for each DIF item. Then, the multinomial log-linear regression (MLR) model and 2-PL nested logit model (2PL-NLM) were applied to items exhibiting DIF with moderate and large DIF effect sizes. The ultimate goals are (a) to examine behaviors of distractors across gender and (b) to investigate if distractors have any impact on DIF effects. DIF results of the Art Section of the General Aptitude Test (GAT-ART) based on both 3PL-NLR and 4PL-NLR methods indicate that only 10 DIF items had moderate to large DIF effects sizes. According to MLR differential distractor functioning (DDF) results, all items exhibited DDF across gender except for one item. An interesting finding of this study is that DIF items related to the verbal analogy and context analysis were in favor of female students, while all DIF items related to the reading comprehension subdomain were in favor of male students, which may signal the existence of content specific DIF or true ability difference across gender. DDF results show that distractors have a significant effect on DIF results. Therefore, DDF analysis is suggested along with DIF analysis since it signals the possible causes of DIF.

In this article we present results of a Differential Item Functioning (DIF) study using Shealy and Stout's (1993) multidimensionality-based DIF analysis framework. In this framework, differences in test score distributions across different groups of examinees may be a result of multidimensionality if secondary dimensions (not the primary dimension measured by the test) differentially affect examinee performance. Thus, this framework requires both statistical and substantive judgments for identifying potential DIF items and substantiating the causes of DIF, which, as a result, will enhance a comprehensive construct validity argument. In this article, we illustrate step-by-step procedures of multidimensionality-based DIF analyses using LanguEdge reading comprehension test data. Qualitative data from think-aloud verbal protocols were used to generate DIF hypotheses about differential functioning of vocabulary items between two groups of Indo-European and non-Indo-European L2 learners. Statistical Simultaneous Item Bias Test (SIBTEST; Shealy & Stout, 1993) was used to test the DIF hypotheses. The DIF results supported the hypotheses by flagging four uniform DIF items and one crossing DIF item. Post-hoc analyses of the DIF-flagged items were performed by visual inspection of group differences using TestGraf (Ramsay, 2001) and revisiting the qualitative verbal data and analyses of cognate types. The results showed that DIF items with large effect sizes were associated with the following: (a) translation-equivalent cognates; (b) word meaning determined independent of context, and (c) less frequent, more difficult distracter words than those in stems. In light of empirical evidence, the article discusses implications for test development and validation processes.

e17613 Background: Accurate evaluation of patient satisfaction is an important dimension of quality of cancer screening and treatment. We applied ordinal logistic regression to assess uniform, non-uniform and total Differential Item Functioning (DIF) for the Patient Satisfaction with Cancer-related Care (PSCC) measure items in English vs. Spanish speaking cancer patients. Methods: The sample included 18 to 86 years old patients who completed the PSCC in English (N = 1067; 82.5% female) and Spanish (N = 229; 100% female 100%). We used total scale score and latent satisfaction estimate separately as matching variable to assess uniform, non-uniform, and total DIF for the PSCC items using ordinal logistic regression analysis. We used language as the grouping variable, i.e., the English speaking group is the reference group and the Spanish speaking group the focal group. Results: Using the total scale score (TSS), we found uniform DIF for 10 out of 29 PSCC items. For example, item 1 showed uniform DIF (χ2 = 5....

This study aimed to investigate Differential Item Functioning (DIF) based on gender and ability level for Post-Graduate General Aptitude Test (PGAT) items in Saudi Arabia, using classical methods (MH χ2, MH-LOR, BD χ2, and CDR). The study samples consisted of (4,000) students distributed equally between males and females. For overall sample, 56 (54%) out of 104 items showed DIF: with (48%) of them favoring females and (41%) favoring males. For high ability sample, percentage of DIF items decreased across subtests, particularly for verbal sub-test. DIF items favoring females decreased (to 40%) and the ones favoring males increased (to 55%). ANOVA results showed that for overall sample, females outperformed males on total score and verbal ability, while males outperformed females on quantitative and logical abilities, significantly (p < 0.01). When DIF items were removed for overall sample, gender gap was reduced except for verbal ability, favoring females. For high ability sample, differences on total and sub-scores were not statistically significant except for quantitative ability, that favored males (p < 0.01). When DIF items were removed for high ability sample, gender differences were not statistically significant (p > 0.05). Thus, it was recommended to conduct stratified DIF analysis for ability admission tests based on ability area and level, gender and their interaction; and to report DIF size and direction for ability groups based on cut scores.

The study's objective was to identify the differential item functioning (DIF) or bias using item response theory. This research is exploratory research with a quantitative approach. In this study, the data used is a fundamental physics test, specifically motion one and two dimensions, dynamics and work and energy concept, with 25 items in the two-tier multiple choices in form. Two hundred and fifty-four prospective physics teachers at West Java and Banten provinces, Indonesia, consisting of 103 males (reference group) and 151 females (focal group). The data analysis includes two stages: conformity of the logistic model parameters and item characteristics, followed by DIF detection. The model suitability test shows the appropriate parameter is 2PL. DIF detection has three methods: simple area indices, Raju and the Lord method. Research results show two things. First, the model fit test shows that the data best fit the 2-parameter or 2-PL logistic parameter model, and all items have good characteristics. Second, the most significant item containing uniform DIF was number 5, followed by nonuniform DIF items 9 and 25. These three items were biased because the chances of answering correctly for male and female students differed. Significantly, this means that this item favors one group. This study illustrates the importance of constructing instrument items so that all test takers, both male and female, have an equal chance to answer the question correctly. Keywords: Differential item functioning, physics test, simple area indices, Raju method, Lord method;

We fine‐tuned and compared several encoder‐based Transformer large language models (LLM) to predict differential item functioning (DIF) from the item text. We then applied explainable artificial intelligence (XAI) methods to identify specific words associated with the DIF prediction. The data included 42,180 items designed for English language arts and mathematics summative state assessments among students in grades 3 to 11. Prediction ranged from .04 to .32 among eight focal and reference group pairs. Our findings suggest that many words associated with DIF reflect minor subdomains included in the test blueprint by design, rather than construct‐irrelevant content that may need to be removed from assessments. This may explain why qualitative reviews of DIF items often yield inconclusive results. Our approach can be used to (1) screen words associated with DIF during the item‐writing process for immediate revision to reduce preventable adverse DIF, (2) assist traditional DIF item reviews by highlighting key words, or (3) use DIF prediction as an alternative when obtaining sufficient sample size for traditional DIF analyses is impossible. Extensions of this research can enhance the assessment fairness, especially programs that lack resources to build high‐quality items, and among smaller subpopulations with insufficient sample sizes for traditional DIF analyses. See source code here .

The Center for Epidemiological Studies-Depression Scale (CES-D) is the most widely used instrument to assess depressive symptoms in people living with HIV. However, its differential item functioning (DIF) by HIV status and sexual orientation has yet to be explored. This study examined DIF and measurement invariance of the CES-D using an item response theory (IRT) framework, and a more traditional factor analytic approach. Data from 841 HIV-infected and HIV-uninfected individuals, from Miami, Florida, were analyzed. Uniform DIF by HIV status was detected in Items 4, 12, and 16 from the Positive Affect factor. Nonuniform DIF was detected in Items 13 and 17. Uniform DIF by sexual orientation was detected in Items 2, 15, and 19, two of them from the Interpersonal factor. Nonuniform DIF was detected in Item 2. Using a factor analytic approach, the CES-D was invariant at the configural and metric levels by HIV and sexual orientation. These findings indicate that overall, however, using IRT, the magnitudes of DIF were negligible, the CED-D was somewhat invariant using factor analytic methods; the CES-D may be reliably used to compare by HIV status or sexual orientation.

This study compared three procedures—the Mantel- Haenszel (MH), the simultaneous item bias (SIB), and the logistic regression (LR) procedures—with respect to their Type I error rates and power to detect nonuniform dif ferential item functioning (DIF). Data were simulated to reflect a variety of conditions: The factors manipulated included sample size, ability distribution differences between the focal and the reference groups, proportion of DIF items in the test, DIF effect sizes, and type of item. 384 conditions were studied. Both the SIB and LR proce dures were equally powerful in detecting nonuniform DIF under most conditions. The MH procedure was not very effective in identifying nonuniform DIF items that had disordinal interactions. The Type I error rates were within the expected limits for the MH procedure and were higher than expected for the SIB and LR proce dures ; the SIB results showed an overall increase of approximately 1% over the LR results. Index terms: differential item functioning, logistic regression statistic, Mantel-Haenszel statistic, nondirectional DIF, simulta neous item bias statistic, SIBTEST, Type I error rate, unidirectional DIF.

OP0102-HPR Cross-Cultural Validity of the Animated Activity Questionnaire (AAQ) that Assesses Activity Limitations in Patients with HIP or Knee Osteoarthritis

Identifying items with differential item functioning (DIF) in an assessment is a crucial step for achieving equitable measurement. One critical issue that has not been fully addressed with existing studies is how DIF items can be detected when data are multilevel. In the present study, we introduced a Lord's Wald test-based procedure for detecting both uniform and non-uniform DIF with polytomous items in the presence of the ubiquitous multilevel data structure. The proposed approach is a multilevel extension of a two-stage procedure, which identifies anchor items in its first stage and formally evaluates candidate items in the second stage. We applied the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm to estimate multilevel polytomous item response theory (IRT) models and to obtain accurate covariance matrices. To evaluate the performance of the proposed approach, we conducted a preliminary simulation study that considered various conditions to mimic real-world scenarios. The simulation results indicated that the proposed approach has great power for identifying DIF items and well controls the Type I error rate. Limitations and future research directions were also discussed.

Assessing differential item functioning (DIF) using the ordinal logistic regression (OLR) model highly depends on the asymptotic sampling distribution of the maximum likelihood (ML) estimators. The ML estimation method, which is often used to estimate the parameters of the OLR model for DIF detection, may be substantially biased with small samples. This study is aimed at proposing a new application of the elastic net regularized OLR model, as a special type of machine learning method, for assessing DIF between two groups with small samples. Accordingly, a simulation study was conducted to compare the powers and type I error rates of the regularized and nonregularized OLR models in detecting DIF under various conditions including moderate and severe magnitudes of DIF (DIF = 0.4 and 0.8), sample size (N), sample size ratio (R), scale length (I), and weighting parameter (w). The simulation results revealed that for I = 5 and regardless of R, the elastic net regularized OLR model with w = 0.1, as compared with the nonregularized OLR model, increased the power of detecting moderate uniform DIF (DIF = 0.4) approximately 35% and 21% for N = 100 and 150, respectively. Moreover, for I = 10 and severe uniform DIF (DIF = 0.8), the average power of the elastic net regularized OLR model with 0.03 ≤ w ≤ 0.06, as compared with the nonregularized OLR model, increased approximately 29.3% and 11.2% for N = 100 and 150, respectively. In these cases, the type I error rates of the regularized and nonregularized OLR models were below or close to the nominal level of 0.05. In general, this simulation study showed that the elastic net regularized OLR model outperformed the nonregularized OLR model especially in extremely small sample size groups. Furthermore, the present research provided a guideline and some recommendations for researchers who conduct DIF studies with small sample sizes.

Differential Item Functioning (DIF) and bias measurement are often used as synonyms in standardized tests fairness evaluation between individuals belonging to different groups. Recently, Zumbo et al. (2016, 2017) have provided a redefinition of DIF/bias term and proposed a new methodology for DIF/bias detection analysis. The new definition of bias requires attributional reasoning; therefore, there is a need to find a way to control for possible confounding factors. Only by balancing groups with respect to covariates, it is possible to attribute DIF to group membership. Propensity score matching techniques allow to carry out groups balancing and bias is detected if item is flagged as DIF, after balancing groups. The conditional logistic regression is proposed for DIF detection analysis after matching because it allows to consider the data structure generated by matching. The aim of this work is twofold. Firstly, we assess the efficacy and performance of the new methodology in imbalanced groups, comparing its performance to performance of traditional DIF detection methods (Mantel-Haenszel statistic, logistic regression and Lord's χ2). Our research, through a simulation study, shows that the new methodology outperforms traditional DIF detection methods in imbalanced groups in situations of large samples and DIF items presence. Nevertheless, the new methodology suffers to I error inflation for large samples and simulation results suggest that the use of an effect size measure (ΔR2) reduces significantly this issue. Secondly, the proposal methodology is applied to data coming from the large-scale standardized test administered by the National Evaluation Institute for the School System (INVALSI) to evaluate pupils' Italian language and mathematics competencies. The idea is to detect possible DIF items among pupils from different academic tracks. The results reveal that very few items are flagged as DIF, indicating the fairness of INVALSI tests.

The aims of this study were to investigate whether the 6 items of the Physical Appearance Scale (Marsh, Richards, Johnson, Roche, & Tremayne, 1994) show differential item functioning (DIF) across gender groups of adolescents, and to show how this can be done using the multigroup mean and covariance structure (MG-MACS) analysis model. Two samples composed of 402 boys and 445 girls were analyzed. Two DIF items were detected. One of them showed uniform DIF in the unexpected direction, whereas the other showed nonuniform DIF in the expected direction. The practical significance of the DIF detected was trivial. Thus, we concluded that the differences between girls' and boys' average scores on the analyzed scale reflect valid differences on the latent trait.

Analyzing examinees’ responses using cognitive diagnostic models (CDMs) has the advantage of providing diagnostic information. To ensure the validity of the results from these models, differential item functioning (DIF) in CDMs needs to be investigated. In this article, the Wald test is proposed to examine DIF in the context of CDMs. This study explored the effectiveness of the Wald test in detecting both uniform and nonuniform DIF in the DINA model through a simulation study. Results of this study suggest that for relatively discriminating items, the Wald test had Type I error rates close to the nominal level. Moreover, its viability was underscored by the medium to high power rates for most investigated DIF types when DIF size was large. Furthermore, the performance of the Wald test in detecting uniform DIF was compared to that of the traditional Mantel‐Haenszel (MH) and SIBTEST procedures. The results of the comparison study showed that the Wald test was comparable to or outperformed the MH and SIBTEST procedures. Finally, the strengths and limitations of the proposed method and suggestions for future studies are discussed.

BackgroundOver 2 billion people suffer from vision impairment or blindness globally, and access to validated visual measurement tools in imperative in accurately describing and managing the burden of eye disease. The present study applies contemporary psychometric validation techniques to the widely used 33-item Indian Visual Function Questionnaire (IND-VFQ-33).MethodsWe first estimated the polychoric correlation between each pair of items. Next, an unrotated and oblique Promax rotated factor analysis, item response theory (IRT, using a graded response model (GRM)), and differential item functioning (DIF) testing were applied to the IND-VFQ-33. We subsequently propose a validated IND-VFQ-33 questionnaire after psychometric testing, data reduction, and adjustment.ResultsExploratory unrotated factor analysis identified two factors; one with a particularly high eigenvalue (18.1) and a second with a lower eigenvalue still above our threshold (1.1). A subsequent oblique Promax factor rotation was undertaken for a 2-factor solution, revealing two moderately correlated factors (+ 0.68) with clinically discrete item loadings onto either Factor 1 (21 items; collectively labelled “daily activities”) or Factor 2 (5 items; collectively labelled “bright lights”). IRT confirmed high item discrimination for all remaining items with good separation between difficulty thresholds. We found significant DIF on depression for six items in Factor 1 (all uniform DIF, except item 21 (non-uniform DIF) with no substantive difference in beta thresholds for any item and no substantive difference in expected individual or sum score, by depression at baseline. For Factor 2, only one item demonstrated significant uniform DIF on gender, similarly without major differences in beta thresholds or expected total score between gender at baseline. Consequently, no further item recalibration or reduction was undertaken after IRT and DIF analysis.ConclusionApplying IRT and DIF validation techniques to the IND-VFQ-33 identified 2 discrete factors with 26 uniquely-loading items, clinically representative of difficulty performing daily activities and experiencing difficulty due to bright lights/glare respectively. The proposed modified scale may be useful in evaluating symptomatic disease progression or response to treatment in an Indian population.