A general framework for modeling missing data due to item selection with item response theory

The selection of items from a calibrated item bank for fixed-form tests is an optimal test design problem; this problem has been handled in the literature by mathematical programming models. A similar problem, however, arises when items are selected for an adaptive test or for testlets. This paper focuses on the similarities of optimal design of fixed-form tests, adaptive tests, and testlets within the framework of the general theory of optimal designs. A sequential design procedure is proposed that uses these similarities. This procedure not only enables optimal design of fixed-form tests, adaptive tests, and testlets, but is also very flexible. The procedure is easy to apply, and consistent estimates for the trait level distribution are obtained. Index terms: adaptive tests, consistency, efficiency, optimal test design, sequential procedure, test design, testlets.

It is not uncommon to use unidimensional item response theory (IRT) models to estimate ability in multidimensional data. Therefore it is important to understand the implications of summarizing multiple dimensions of ability into a single parameter estimate, especially if effects are confounded when applied to computerized adaptive testing (CAT). Previous studies have investigated the effects of different IRT models and ability estimators by manipulating the relationships between item and person parameters. However, in all cases, the maximum information criterion was used as the item selection method. Because maximum information is heavily influenced by the item discrimination parameter, investigating a-stratified item selection methods is tenable. The current Monte Carlo study compared maximum information, a-stratification, and a-stratification with b blocking item selection methods, alone, as well as in combination with the Sympson-Hetter exposure control strategy. The six testing conditions were conditioned on three levels of interdimensional item difficulty correlations and four levels of interdimensional examinee ability correlations. Measures of fidelity, estimation bias, error, and item usage were used to evaluate the effectiveness of the methods. Results showed either stratified item selection strategy is warranted if the goal is to obtain precise estimates of ability when using unidimensional CAT in the presence of two-dimensional data. If the goal also includes limiting bias of the estimate, Sympson-Hetter exposure control should be included. Results also confirmed that Sympson-Hetter is effective in optimizing item pool usage. Given these results, existing unidimensional CAT implementations might consider employing a stratified item selection routine plus Sympson-Hetter exposure control, rather than recalibrate the item pool under a multidimensional model.

Several alternatives for item selection algorithms based on item response theory in computerized classification testing (CCT) have been suggested, with no conclusive evidence on the substantial superiority of a single method. It is argued that the lack of sizable effect is because some of the methods actually assess items very similarly through different calculations and will usually select the same item. Consideration of methods that assess information across a wider range is often unnecessary under realistic conditions, although it might be advantageous to utilize them only early in a test. In addition, the efficiency of item selection approaches depend on the termination criteria that are used, which is demonstrated through didactic example and Monte Carlo simulation. Item selection at the cut score, which seems conceptually appropriate for CCT, is not always the most efficient option. A broad framework for item selection in CCT is presented that incorporates these points.

In the assessment field, the synergy between artificial intelligence (AI) and item response theory (IRT) opens new revolutionary perspectives.AI brings unprecedented data analysis and processing capabilities, while IRT provides a robust theoretical framework for modeling learners' responses to test items.This combination enables addressing the complex challenges of assessment more effectively and precisely than ever before.This research explores the application of IRT in conjunction with AI in the assessment area.Focusing on score regeneration following the removal of improper items, our study employs AI to effectively identify and remove such items, thereby influencing candidate selection.Using key parameters of IRT, such as difficulty, discrimination, and chance, this research aims to develop a deep learning (DL) model for classifying and selecting items.The study specifically applies the Artificial Neural Network (ANN) method via binary classification models for this task.Experimental results show that the proposed model performs very well, with a high accuracy rate for item selection.

Over the past thirty years, obtaining diagnostic information from examinees’ item responses has become an increasingly important feature of educational and psychological testing. The objective can be achieved by sequentially selecting multidimensional items to fit the class of latent traits being assessed, and therefore Multidimensional Computerized Adaptive Testing (MCAT) is one reasonable approach to such task. This study conducts a rigorous investigation on the relationships among four promising item selection methods: D-optimality, KL information index, continuous entropy, and mutual information. Some theoretical connections among the methods are demonstrated to show how information about the unknown vector θ can be gained from different perspectives. Two simulation studies were carried out to compare the performance of the four methods. The simulation results showed that mutual information not only improved the overall estimation accuracy but also yielded the smallest conditional mean squared error in most region of θ. In the end, the overlap rates were calculated to empirically show the similarity and difference among the four methods.

Computerized adaptive language testing (CALT), supported by item response theory (IRT) and Internet technologies, has become an important aspect in modern testing. This paper first reviews the basic theories in testing and then presents the process of a CALT. Later, key issues of a CALT, such as construction of item banks, item selection and terminations of a test, have been addressed in detail. Besides, the advantages and disadvantages of a CALT are discussed in accordance with the current CALT practice around the world. By doing so, the paper points out that CALT is the new trend in language testing and will be applied massively in the future in China.

ABSTRACTThis paper examines item response theory (IRT) scale transformations and IRT scale linking methods used in the Non‐Equivalent Groups with Anchor Test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean‐mean, mean‐var linking, concurrent calibration, Stocking and Lord and Haebara characteristic curves approaches, and fixed‐item parameters scale linkage. The main idea is to view any linking procedure as a restriction on the item parameter space. Then a rewriting of the log‐likelihood function together with an appropriately implemented maximization procedure of the log‐likelihood function under linear (or nonlinear restrictions) will accomplish the linking. The proposed method is general enough to cover both the dichotomous item response models (the one parameter logistic (1PL) model, 2PL, and 3PL) and the polytomous unidimensional IRT models like the generalized partial credit model.

The use of multistage adaptive testing (MST) has gradually increased in large‐scale testing programs as MST achieves a balanced compromise between linear test design and item‐level adaptive testing. MST works on the premise that each examinee gives their best effort when attempting the items, and their responses truly reflect what they know or can do. However, research shows that large‐scale assessments may suffer from a lack of test‐taking engagement, especially if they are low stakes. Examinees with low test‐taking engagement are likely to show noneffortful responding (e.g., answering the items very rapidly without reading the item stem or response options). To alleviate the impact of noneffortful responses on the measurement accuracy of MST, test‐taking engagement can be operationalized as a latent trait based on response times and incorporated into the on‐the‐fly module assembly procedure. To demonstrate the proposed approach, a Monte‐Carlo simulation study was conducted based on item parameters from an international large‐scale assessment. The results indicated that the on‐the‐fly module assembly considering both ability and test‐taking engagement could minimize the impact of noneffortful responses, yielding more accurate ability estimates and classifications. Implications for practice and directions for future research were discussed.

The item response theory (IRT) is widely used in many fields recently because it provides us the abilities of examinees and the difficulties of items simultaneously. Online IRT test systems are often equipped with the adaptive testing. Such a testing method selects the most appropriate items to examinees automatically. However, the item difficulty values remain the same even if the features of the examinees and the size of the items change. Calibration of the difficulty values is required in such cases, and the dually adaptive online IRT testing system works, where “dually adaptive” means that one is targeted to the adequate item selection and the other is targeted to the adjustment of the item difficulty values. Using this, the database in the system will automatically be growing and be updated. In this paper, we introduce such a novel system with applications.

Classical Test Theory (CTT), also known as the true score theory, refers to the analysis of test results based on test scores. The statistics produced under CTT include measures of item difficulty, item discrimination, measurement error and test reliability. The term “Classical” is used in contrast to “Modern” test theory which usually refers to item response theory (IRT). The fact that CTT was developed before IRT does not mean that CTT is outdated or replaced by IRT. Both CTT and IRT provide useful statistics to help us analyse test data. Generally, CTT and IRT provide complementary results. For many item analyses, CTT may be sufficient to provide the information we need. There are, however, theoretical differences between CTT and IRT, and many researchers prefer IRT because of enhanced measurement properties under IRT. IRT also provides a framework that facilitates test equating, computer adaptive testing and test score interpretation. While this book devotes a large part to IRT, we stress that CTT is an important part of the methodologies for educational and psychological measurement. In particular, the exposition of the concept of reliability in CTT sets the basis for evaluating measuring instruments. A good understanding of CTT lays the foundations for measurement principles. There are other approaches to measurement such as generalizability theory and structural equation modelling, but these are not the focus of attention in this book.

Chapter 15 - Perspectives

Preface SECTION ONE: EDUCATIONAL ASSESSMENT IN AMERICA: HISTORY, GOALS, AND APPLICATIONS 1. Issues and Measurement Practices in the Schools Issues and Themes Learning Objectives Accountability, Assessment, and Testing Why do we Test? Types and Varieties of Tests Assumptions Made by Test Developers Trends in Testing Law and Testing Case Law Educational Ethics and Testing Summary Discussion Questions 2. History of Testing in American Education Issues and Themes Learning Objectives Scientific Measurement and Racism The Next Generation Single Factor Models Multiple Factor Models Contemporary Public Schools Winds of Change Summary Discussion Questions SECTION TWO: THE MEANING AND QUALITY OF TEST SCORES 3. The Measurement and Description of Variables Issues and Themes Learning Objectives Variables Central Tendency Normal Distribution Variation and Variance Standard Scores Variation Correlation Summary Discussion Questions 4. Reliability as a Measure of Test Quality Issues and Themes Learning Objectives Reliability Internal Consistency Gains and Losses Standard Error of Measurement Factors with an Impact on Reliability Reliability of Cut Scores Reliability of Performance Tasks and Criterion Tests Summary Discussion Questions 5. Validity as a Measure of Test Quality Issues and Themes Learning Objectives Validity: The Traditional Context of Validity Contemporary View Comparison Groups Test Bias and Fairness Summary Discussion Questions SECTION THREE: TESTING OF STUDENTS BY CLASSROOM TEACHERS 6. Planning and Constructing Classroom Tests Issues and Themes Learning Objectives Testing and Teaching Formative Tests Summative Evaluations Testing Formats Summary Discussion Questions 7. Classroom Development and Use of Extended Answer Tests Issues and Themes Learning Objectives Homework Completion or Short Answer The Essay Test Bluffing Essay Test Format Essay Grading The SAT Essay Test Summary Discussion Questions 8. Item Selection and Analysis Issues and Themes Classroom Testing A Priori Qualitative Assessment and Analysis of Tests Item Difficulty Index Distracter Analysis Item Analysis with Constructed or Supply Type Items Discrimination Computer Adaptive Testing Item Response Theory (IRT) Summary Discussion Questions 9. Performance Assessments Issues and Themes Learning Objectives Assessment and Instruction Performance Based Assessments Scoring Portfolios Summary Discussion Questions 10. Grades, Progress Reports, and Report Cards Issues and Themes Learning Objectives Background and History Purposes of Report Card Grades Student Grades Report Card Comments On-line Report Cards Report Card Grades for the Gifted Grades and Students with Special Needs The Case of Transfer Students Grade Point Average and Class Rank Teacher-Parent Conference Grade Inflation Promotion and Graduation Summary Discussion Questions SECTION FOUR: TESTING FOR STUDENT LEARNING, TALENT, APTITUDE, AND SPECIAL NEEDS 11. Standardized Measures of Learning Issues and Themes Learning Objectives Explaining Test Scores to Parents Major Achievement Test Batteries State Mandated Assessments Summary Discussion Questions 12. Testing of Aptitude and Selection Issues and Themes Learning Objectives Aptitude Scholastic Aptitude and Intelligence Training of Test Administrators Tests of Intelligence Administered One-on-One Tests of Intelligence Administered to Groups Group Administered Aptitude Tests Assessment of Talented and Gifted Children Assessment of Aptitude for College Study Summary Discussion Questions 13. Identification of Learning Problems Issues and Themes Learning Objectives Incidence Informal Screening Instructional Support Team Referral, Assessment, and the IEP Committee Assessment Process Formal Assessments of Attention and Focus AD/HD Measurement Scales Assessments of Reading Problems Early Reading Tests Assessments of Language and Speech Problems Measures for the Identification of Problems Learning Mathematics Individualized Educational Plan Accommodations for Children with Disabilities Summary Discussion Questions SECTION FIVE: NO CHILD, TEACHER, OR SCHOOL LEFT BEHIND 14. International, National, and Statewide Testing Programs Issues and Themes Learning Objectives International Comparisons Nation's Report Card School Assessments No Child Left Behind Summary Discussion Questions 15. Student, Family, and Teacher Factors in Test Scores Issues and Themes Learning Objectives Students Families Teachers Summary Discussion Questions 16. Test Preparation for Successful Schools Issues and Themes Learning Objectives Accountability Angst Legal and Ethical Requirements Marginal Ethics Successful Schools Classroom Instruction Architecture and School Design Administration Summary Discussion Questions 17. Accountability and Evaluation Issues and Themes Learning Objectives Evaluation Overview Systematic Program Evaluation Steps in a Systematic Evaluation Value-added Evaluation Summary Discussion Questions References Glossary Index About the Author

An examination of the psychometric properties of the physical self-description questionnaire using a polytomous item response model

Tests used for such purposes as determining educational quality, defining educational needs, hiring an employee, student selection and placement and performing guidance and clinic services have an important place in education and psychology. Of course, they should have certain psychometric features related to test scores' validity and reliability. Various test theories have helped to create more valid and reliable measurements and, as a result, to make better decisions regarding individuals. In education and psychology, Classical Test Theory (CTT) and Item Response Theory (IRT) are both widely used. CTT assumes that an individual's observed score is the total of the true score and the error score, while IRT estimates an individual's ability or latent trait from responses to test items (Embretson & Reise, 2000).When IRT assumptions and model-data fit are ensured, item and ability parameters' invariance occurs; this is known as the most important advantage IRT has over CTT. Item and ability parameters' invariance means estimating ability parameters independently of item sample and estimating item parameters independently of ability sample. IRT's invariance feature makes it very practicable in many applications, for instance, test development, computerized adaptive testing, bias studies, test equating and item mapping (Hambleton & Swaminathan, 1985). IRT is classified under two main categories as parametric IRT (PIRT) and nonparametric IRT (NIRT) (Olivares, 2005; Sijtsma & Molenaar, 2002).To analyse ordered items, such as Likert-type attitude items, partial credit cognitive items or not ordered graded items such as multiple-choice test items, item response models are developed towards polytomous items in IRT (Ostini & Nering, 2006). In these models developed for polytomous items, a non-linear relationship between an individual's latent trait and the possibility of choosing a certain category of item answer is explained (Embretson & Reise, 2000). Graded Response Model (GRM), part of IRT models developed for polytomous items, is often preferred by researchers for applications since it is more useful in presentations, portfolios, essays and Likert-type items with ordered item categories (DeMars, 2010; Ostini & Nering, 2006). To scale tests that consist of polytomous items by making true estimates according to GRM, evaluating PIRT's assumptions and model-data fit is necessary. And to provide these assumptions and model-data fit, large samples are needed. At this point, NIRT models draw attention because they provide a practical advantage in determining psychometric properties of tests with fewer items and respondents (Stout, 2001).NIRT models are defined as statistical scaling methods that require fewer assumptions than PIRT models for measuring persons and items (Stochl, 2007). With their wide application area, NIRT models are used in ordinal scales, applied research areas, sociology, marketing research and health research on quality of life (Sijtsma, 2005). The literature reveals that two models, namely, the Mokken model and nonparametric regression estimation models, are employed. These two models are themselves divided into sub-models. The Mokken model consists of the sub-models Monotone Homogeneity Model (MHM) and the Double Monotonicity Model (DMM). Nonparametric regression estimation models consist of such sub-models as the Kernel Smoothing Approach Model (KSAM), the Isotonic Regression Estimation and the Smoothed Isotonic Regression Estimation models (Lee, 2007; Sijtsma & Molenaar, 2002). Along with theoretical studies being conducted, new sub-models are being added to nonparametric regression estimation models.As a NIRT model, MHM requires unidimensionality, local independence and monotonicity assumptions, and it defines the relationship that latent variables and items with homogeneous (unidimensional) and monotone item characteristic curve (ICC) have (Meijer & Baneke, 2004; Sijtsma & Molenaar, 2002). …

BackgroundIncreased violent events happen in the general hospitals in China and yet non-psychiatric departments do not have tools for violence-tendency screening.MethodsThe current study developed and evaluated two Inpatient Psychological Experience Questionnaires (IPEQs) for the screening of violence-related six mental health disorders: (Inpatient Psychological Experience Questionnaire-1 (IPEQ-1): anxiety, depression and suicidality; Inpatient Psychological Experience Questionnaire-2 (IPEQ-2): paranoid personality disorder, emotionally unstable personality disorder and histrionic personality disorder). Two initial IPEQs (IPEQ-1: 37 items and IPEQ-2: 30 items) were developed and assessed by domain experts. Then 1210 inpatients were recruited and divided into three groups (160, 450 and 600 samples, respectively) for IPEQs item selection and evaluation. During the two-stage item selection, three statistical methods including Pearson’s correlation coefficient, exploratory factor analysis and item response theory were applied. For the item evaluation, Cronbach’s alpha coefficient, test-retest reliability, criterion-related validity and construct validity of the final questionnaires were measured.ResultsTwelve items were selected for each IPEQs. Cronbach’s alpha coefficients were 0.91 and 0.78 for IPEQ-1 and IPEQ-2, respectively. Test-retest replication ratios were 0.95 and 0.87 for IPEQ-1 and IPEQ-2, respectively. Correlation coefficients between different disorders and their related-tools scores were [0.51, 0.44] and [0.40, 0.44] for IPEQ-1 and IPEQ-2, respectively and were significant (P < 0.01). Confirmatory factor analysis supported the validity of the final IPEQs (P < 0.05), and the model fit index met the criterion generally.ConclusionThe IPEQs developed in this study could be effective and easy-to-use tools for screening inpatients with violence-intendancy in non-psychosomatic departments.