Item Selection in Computerized Classification Testing

Computerized classification testing (CCT) commonly chooses items maximizing information at the cut score, which yields the most information for decision-making. However, a corollary problem is that all examinees will be given the same set of items, resulting in high test overlap rate and unbalanced item bank usage, which threatens test security. Moreover, another pivotal issue for CCT is time control. Since both the extremely long response time (RT) and large RT variability across examinees intensify time-induced anxiety, it is crucial to reduce the number of examinees exceeding the time limitation and the differences between examinees' test-taking times. To satisfy these practical needs, this paper proposes the novel idea of stage adaptiveness to tailor the item selection process to the decision-making requirement in each step and generate fresh insight into the existing response time selection method. Results indicate that a balanced item usage as well as short and stable test times across examinees can be achieved via the new methods.

<p id="p00005">Computerized classification testing (CCT) can adaptively classify test-takers into two or more different categories, and it has been widely used in qualifying tests and clinical psychology or medical diagnosis. As an essential part of CCT, the termination rule determines when the test is to be stopped and to which category the test-taker is ultimately classified into, directly affecting the test efficiency and classification accuracy. According to the theoretical basis of the termination rules, existing rules can be roughly divided into the likelihood ratio, Bayesian decision theory, and confidence interval rules. And their core ideas are constructing hypothesis tests, designing loss functions, and comparing the relative positions of confidence intervals, respectively. At the same time, when constructing specific termination rules, the requirement of different test scenarios (e.g., the number of categories and the number of tests’ dimensions) should also be considered. <br/>There are advantages and disadvantages to each of the three types of termination rules. Specifically, the likelihood ratio rule is based on the likelihood ratio test, with better theoretical properties. However, the method requires prior determination of the indifference interval and the type I and II error rates, introducing the impact of subjective factors. Also, it is more challenging to extend the method in complex test situations, such as multidimensional and multicategory CCT. Bayesian decision theory rules make classification decisions based on the loss function. It can dynamically optimize the decision from a more global perspective since it works backward from the final stage of the test. In addition, the variety of loss functions makes the method very flexible in form and makes it easy to be applied to different test situations. However, in practice, the flexibility will inevitably result in the uncertainty of the choice of loss function, and the inappropriate loss function may be biased. The confidence interval method is the most straightforward because of its relatively simple principle and low computational effort. However, this method is less robust and has a relatively low test efficiency. <br/>Currently, CCT is mainly applied in eligibility tests and clinical medicine questionnaires. In eligibility tests, all three types of termination rules have the potential to be widely applied. However, in practice, the principles of the likelihood ratio rule and the Bayesian decision theory rule are not easily understood by the general public, and these methods are also accompanied by the problem of over-exposure of items for their preference of cut-point based item selection methods. Therefore, the confidence interval rule, which is relatively simple in principle and has alleviated item exposure, has been widely used in existing qualifying tests. Bayesian decision theory rules are more applicable in clinical questionnaires because of their finer control over various classification losses. <br/>The following can be considered for future research on CCT termination rules. First, Bayesian decision theory rules can be improved by considering non-statistical constraints with the help of the flexibility of its loss function. Second, termination rules can be developed for multidimensional and multicategory CCT to meet more practical needs. Third, termination rules that integrate response time can be developed to improve test efficiency and classification accuracy. Fourth, it is possible to construct termination rules under the framework of machine learning.

Assessment can serve different goals. If the aim of testing is to classify respondents into one of multiple levels instead of obtaining a precise estimate of the respondent’s ability, computerized classification testing can be used. This type of testing requires algorithms for item selection and making the classification decision. The result of the test administration is provided in a report about the decision with sometimes additional feedback. The design of all these components of the test should be in line with the testing goal. Several goals have been defined for assessment which make a judgment about: pupils, the learning process, groups of students and schools, and the quality of education. The possibilities for use of computerized classification testing for different testing goals are investigated in the current paper

This study aims to validate the Online Learning Scale (OLS) and estimate the OLS’s cut score. The sample was 1,459 undergraduate students across different universities and field studies in Thailand. The OLS was newly developed by synthesizing related literature, focus groups, and observations and was designed to measure online learning quality from student-centered perspectives. The 24 OLS items were validated using Classical Test Theory (CTT) and Item Response Theory (IRT). The OLS’ cut score was estimated by analyzing the scale’s lowest points of the test information function. The effectiveness of OLS’ cut score was investigated by dividing groups of samples by cut score and analyzing their difference using Chi-square test and ANOVA with other variables. The results indicated that the OLS showed appropriate psychometric properties: 1) The Cronbach’s alpha reliability was .94 and discrimination indices were .62 to .79, 2) CFA fit indices indicated good fit (CFI = .953, RMSEA = .053), and 3) The IRT difficulty parameters were arranged in order and in the range of -3.0 to 3.0, while the discrimination parameters exceeded 0.65 in all items. The estimated two cut scores (29 and 48 points) effectively distinguished the online learning quality delivered to the students.

In education testing, the items that examinees receive may be selected for a variety of reasons, resulting in missing data for items that were not selected. Item selection is internal when based on prior performance on the test, such as in adaptive testing designs or for branching items. Item selection is external when based on an auxiliary variable collected independently to performance on the test, such as education level in a targeting testing design or geographical location in a nonequivalent anchor test equating design. This paper describes the implications of this distinction for Item Response Theory (IRT) estimation, drawing upon missing-data theory (e.g., Mislevy &amp; Sheehan, 1989, https://doi.org/10.1007/BF02296402; Rubin, 1976, https://doi.org/10.1093/biomet/63.3.581), and selection theory (Meredith, 1993, https://doi.org/10.1007/BF02294825). Through mathematical analyses and simulations, we demonstrate that this internal versus external item selection framework provides a general guide in applying missing-data and selection theory to choose a valid analysis model for datasets with missing data.

It is not uncommon to use unidimensional item response theory (IRT) models to estimate ability in multidimensional data. Therefore it is important to understand the implications of summarizing multiple dimensions of ability into a single parameter estimate, especially if effects are confounded when applied to computerized adaptive testing (CAT). Previous studies have investigated the effects of different IRT models and ability estimators by manipulating the relationships between item and person parameters. However, in all cases, the maximum information criterion was used as the item selection method. Because maximum information is heavily influenced by the item discrimination parameter, investigating a-stratified item selection methods is tenable. The current Monte Carlo study compared maximum information, a-stratification, and a-stratification with b blocking item selection methods, alone, as well as in combination with the Sympson-Hetter exposure control strategy. The six testing conditions were conditioned on three levels of interdimensional item difficulty correlations and four levels of interdimensional examinee ability correlations. Measures of fidelity, estimation bias, error, and item usage were used to evaluate the effectiveness of the methods. Results showed either stratified item selection strategy is warranted if the goal is to obtain precise estimates of ability when using unidimensional CAT in the presence of two-dimensional data. If the goal also includes limiting bias of the estimate, Sympson-Hetter exposure control should be included. Results also confirmed that Sympson-Hetter is effective in optimizing item pool usage. Given these results, existing unidimensional CAT implementations might consider employing a stratified item selection routine plus Sympson-Hetter exposure control, rather than recalibrate the item pool under a multidimensional model.

PurposeThis study aims to focus on providing a computerized classification testing (CCT) system that can easily be embedded as a self-assessment feature into the existing legacy environment of a higher education institution, empowering students with self-assessments to monitor their learning progress and following strict data protection regulations. The purpose of this study is to investigate the use of two different versions (without dashboard vs with dashboard) of the CCT system during the course of a semester; to examine changes in the intended use and perceived usefulness of two different versions (without dashboard vs with dashboard) of the CCT system; and to compare the self-reported confidence levels of two different versions (without dashboard vs with dashboard) of the CCT system.Design/methodology/approachA total of N = 194 students from a higher education institution in the area of economic and business education participated in the study. The participants were provided access to the CCT system as an opportunity to self-assess their domain knowledge in five areas throughout the semester. An algorithm was implemented to classify learners into master and nonmaster. A total of nine metrics were implemented for classifying the performance of learners. Instruments for collecting co-variates included the study interest questionnaire (Cronbach’s a = 0. 90), the achievement motivation inventory (Cronbach’s a = 0. 94), measures focusing on perceived usefulness and demographic data.FindingsThe findings indicate that the students used the CCT system intensively throughout the semester. Students in a cohort with a dashboard available interacted more with the CCT system than students in a cohort without a dashboard. Further, findings showed that students with a dashboard available reported significantly higher confidence levels in the CCT system than participants without a dashboard.Originality/valueThe design of digitally supported learning environments requires valid formative (self-)assessment data to better support the current needs of the learner. While the findings of the current study are limited concerning one study cohort and a limited number of self-assessment areas, the CCT system is being further developed for seamless integration of self-assessment and related feedback to further reveal unforeseen opportunities for future student cohorts.

This article illustrates five different methods for estimating Angoff cut scores using item response theory (IRT) models. These include maximum likelihood (ML), expected a priori (EAP), modal a priori (MAP), and weighted maximum likelihood (WML) estimators, as well as the most commonly used approach based on translating ratings through the test characteristic curve (i.e., the IRT true‐score (TS) estimator). The five methods are compared using a simulation study and a real data example. Results indicated that the application of different methods can sometimes lead to different estimated cut scores, and that there can be some key differences in impact data when using the IRT TS estimator compared to other methods. It is suggested that one should carefully think about their choice of methods to estimate ability and cut scores because different methods have distinct features and properties. An important consideration in the application of Bayesian methods relates to the choice of the prior and the potential bias that priors may introduce into estimates.

In the assessment field, the synergy between artificial intelligence (AI) and item response theory (IRT) opens new revolutionary perspectives.AI brings unprecedented data analysis and processing capabilities, while IRT provides a robust theoretical framework for modeling learners' responses to test items.This combination enables addressing the complex challenges of assessment more effectively and precisely than ever before.This research explores the application of IRT in conjunction with AI in the assessment area.Focusing on score regeneration following the removal of improper items, our study employs AI to effectively identify and remove such items, thereby influencing candidate selection.Using key parameters of IRT, such as difficulty, discrimination, and chance, this research aims to develop a deep learning (DL) model for classifying and selecting items.The study specifically applies the Artificial Neural Network (ANN) method via binary classification models for this task.Experimental results show that the proposed model performs very well, with a high accuracy rate for item selection.

Preface SECTION ONE: EDUCATIONAL ASSESSMENT IN AMERICA: HISTORY, GOALS, AND APPLICATIONS 1. Issues and Measurement Practices in the Schools Issues and Themes Learning Objectives Accountability, Assessment, and Testing Why do we Test? Types and Varieties of Tests Assumptions Made by Test Developers Trends in Testing Law and Testing Case Law Educational Ethics and Testing Summary Discussion Questions 2. History of Testing in American Education Issues and Themes Learning Objectives Scientific Measurement and Racism The Next Generation Single Factor Models Multiple Factor Models Contemporary Public Schools Winds of Change Summary Discussion Questions SECTION TWO: THE MEANING AND QUALITY OF TEST SCORES 3. The Measurement and Description of Variables Issues and Themes Learning Objectives Variables Central Tendency Normal Distribution Variation and Variance Standard Scores Variation Correlation Summary Discussion Questions 4. Reliability as a Measure of Test Quality Issues and Themes Learning Objectives Reliability Internal Consistency Gains and Losses Standard Error of Measurement Factors with an Impact on Reliability Reliability of Cut Scores Reliability of Performance Tasks and Criterion Tests Summary Discussion Questions 5. Validity as a Measure of Test Quality Issues and Themes Learning Objectives Validity: The Traditional Context of Validity Contemporary View Comparison Groups Test Bias and Fairness Summary Discussion Questions SECTION THREE: TESTING OF STUDENTS BY CLASSROOM TEACHERS 6. Planning and Constructing Classroom Tests Issues and Themes Learning Objectives Testing and Teaching Formative Tests Summative Evaluations Testing Formats Summary Discussion Questions 7. Classroom Development and Use of Extended Answer Tests Issues and Themes Learning Objectives Homework Completion or Short Answer The Essay Test Bluffing Essay Test Format Essay Grading The SAT Essay Test Summary Discussion Questions 8. Item Selection and Analysis Issues and Themes Classroom Testing A Priori Qualitative Assessment and Analysis of Tests Item Difficulty Index Distracter Analysis Item Analysis with Constructed or Supply Type Items Discrimination Computer Adaptive Testing Item Response Theory (IRT) Summary Discussion Questions 9. Performance Assessments Issues and Themes Learning Objectives Assessment and Instruction Performance Based Assessments Scoring Portfolios Summary Discussion Questions 10. Grades, Progress Reports, and Report Cards Issues and Themes Learning Objectives Background and History Purposes of Report Card Grades Student Grades Report Card Comments On-line Report Cards Report Card Grades for the Gifted Grades and Students with Special Needs The Case of Transfer Students Grade Point Average and Class Rank Teacher-Parent Conference Grade Inflation Promotion and Graduation Summary Discussion Questions SECTION FOUR: TESTING FOR STUDENT LEARNING, TALENT, APTITUDE, AND SPECIAL NEEDS 11. Standardized Measures of Learning Issues and Themes Learning Objectives Explaining Test Scores to Parents Major Achievement Test Batteries State Mandated Assessments Summary Discussion Questions 12. Testing of Aptitude and Selection Issues and Themes Learning Objectives Aptitude Scholastic Aptitude and Intelligence Training of Test Administrators Tests of Intelligence Administered One-on-One Tests of Intelligence Administered to Groups Group Administered Aptitude Tests Assessment of Talented and Gifted Children Assessment of Aptitude for College Study Summary Discussion Questions 13. Identification of Learning Problems Issues and Themes Learning Objectives Incidence Informal Screening Instructional Support Team Referral, Assessment, and the IEP Committee Assessment Process Formal Assessments of Attention and Focus AD/HD Measurement Scales Assessments of Reading Problems Early Reading Tests Assessments of Language and Speech Problems Measures for the Identification of Problems Learning Mathematics Individualized Educational Plan Accommodations for Children with Disabilities Summary Discussion Questions SECTION FIVE: NO CHILD, TEACHER, OR SCHOOL LEFT BEHIND 14. International, National, and Statewide Testing Programs Issues and Themes Learning Objectives International Comparisons Nation's Report Card School Assessments No Child Left Behind Summary Discussion Questions 15. Student, Family, and Teacher Factors in Test Scores Issues and Themes Learning Objectives Students Families Teachers Summary Discussion Questions 16. Test Preparation for Successful Schools Issues and Themes Learning Objectives Accountability Angst Legal and Ethical Requirements Marginal Ethics Successful Schools Classroom Instruction Architecture and School Design Administration Summary Discussion Questions 17. Accountability and Evaluation Issues and Themes Learning Objectives Evaluation Overview Systematic Program Evaluation Steps in a Systematic Evaluation Value-added Evaluation Summary Discussion Questions References Glossary Index About the Author

Even when the scoring of an examination is based on item response theory (IRT), standard-setting methods seldom use this information directly when determining the minimum passing score (MPS) for an examination from an Angoff-based standard-setting study. Often, when IRT scoring is used, the MPS value for a test is converted to an IRT-based theta ability estimate by using the test characteristic curve. The purpose of this study is to investigate some approaches to determining the MPS from an Angoff-based standard-setting study when the test questions have been calibrated using IRT methods. Although the methods studied appear to converge on the same cut score, the cut scores derived from the IRT-based approaches are all within one standard error of the Angoff-based MPS value.

Over the past thirty years, obtaining diagnostic information from examinees’ item responses has become an increasingly important feature of educational and psychological testing. The objective can be achieved by sequentially selecting multidimensional items to fit the class of latent traits being assessed, and therefore Multidimensional Computerized Adaptive Testing (MCAT) is one reasonable approach to such task. This study conducts a rigorous investigation on the relationships among four promising item selection methods: D-optimality, KL information index, continuous entropy, and mutual information. Some theoretical connections among the methods are demonstrated to show how information about the unknown vector θ can be gained from different perspectives. Two simulation studies were carried out to compare the performance of the four methods. The simulation results showed that mutual information not only improved the overall estimation accuracy but also yielded the smallest conditional mean squared error in most region of θ. In the end, the overlap rates were calculated to empirically show the similarity and difference among the four methods.

Chapter 15 - Perspectives

It is essential to control self-reported trait measurements for response style effects to ensure a valid interpretation of estimates. Traditional psychometric models facilitating such control consider item responses as the result of two kinds of response processes—based on the substantive trait, or based on response styles—and they assume that both of these processes have a constant influence across the items of a questionnaire. However, this homogeneity over items is not always given, for instance, if the respondents’ motivation declines throughout the questionnaire so that heuristic responding driven by response styles may gradually take over from cognitively effortful trait-based responding. The present study proposes two dynamic IRTree models, which account for systematic continuous changes and additional random fluctuations of response strategies, by defining item position-dependent trait and response style effects. Simulation analyses demonstrate that the proposed models accurately capture dynamic trajectories of response processes, as well as reliably detect the absence of dynamics, that is, identify constant response strategies. The continuous version of the dynamic model formalizes the underlying response strategies in a parsimonious way and is highly suitable as a cognitive model for investigating response strategy changes over items. The extended model with random fluctuations of strategies can adapt more closely to the item-specific effects of different response processes and thus is a well-fitting model with high flexibility. By using an empirical data set, the benefits of the proposed dynamic approaches over traditional IRTree models are illustrated under realistic conditions.

BackgroundIncreased violent events happen in the general hospitals in China and yet non-psychiatric departments do not have tools for violence-tendency screening.MethodsThe current study developed and evaluated two Inpatient Psychological Experience Questionnaires (IPEQs) for the screening of violence-related six mental health disorders: (Inpatient Psychological Experience Questionnaire-1 (IPEQ-1): anxiety, depression and suicidality; Inpatient Psychological Experience Questionnaire-2 (IPEQ-2): paranoid personality disorder, emotionally unstable personality disorder and histrionic personality disorder). Two initial IPEQs (IPEQ-1: 37 items and IPEQ-2: 30 items) were developed and assessed by domain experts. Then 1210 inpatients were recruited and divided into three groups (160, 450 and 600 samples, respectively) for IPEQs item selection and evaluation. During the two-stage item selection, three statistical methods including Pearson’s correlation coefficient, exploratory factor analysis and item response theory were applied. For the item evaluation, Cronbach’s alpha coefficient, test-retest reliability, criterion-related validity and construct validity of the final questionnaires were measured.ResultsTwelve items were selected for each IPEQs. Cronbach’s alpha coefficients were 0.91 and 0.78 for IPEQ-1 and IPEQ-2, respectively. Test-retest replication ratios were 0.95 and 0.87 for IPEQ-1 and IPEQ-2, respectively. Correlation coefficients between different disorders and their related-tools scores were [0.51, 0.44] and [0.40, 0.44] for IPEQ-1 and IPEQ-2, respectively and were significant (P < 0.01). Confirmatory factor analysis supported the validity of the final IPEQs (P < 0.05), and the model fit index met the criterion generally.ConclusionThe IPEQs developed in this study could be effective and easy-to-use tools for screening inpatients with violence-intendancy in non-psychosomatic departments.