Abstract In recent years, machine learning (ML) techniques have received more attention in detecting aberrant test‐taking behaviors due to advantages when compared to traditional data forensics methods. However, defining “True Test Cheaters” is challenging—different than other fraud detection tasks such as flagging forged bank checks or credit card frauds, testing organizations are often lack of physical evidences to identify “True Test Cheaters” to train ML models. This study proposed a statistically defensible method of labeling “True Test Cheaters” in the data, demonstrated the effectiveness of using ML approaches to identify irregular statistical patterns in exam data, and established an analytical framework for evaluating and conducting real‐time ML‐based test data forensics. Classification accuracy and false negative/positive results are evaluated across different supervised‐ML techniques. The reliability and feasibility of operationally using this approach for an IT certification exam are evaluated using real data.

Abstract Research on Automated Essay Scoring has become increasing important because it serves as a method for evaluating students’ written responses at scale. Scalable methods for scoring written responses are needed as students migrate to online learning environments resulting in the need to evaluate large numbers of written‐response assessments. The purpose of this study is to describe and evaluate three active learning methods that can be used to minimize the number of essays that must be scored by human raters while still providing the data needed to train a modern Automated Essay Scoring system. The three active learning methods are the uncertainty‐based, the topological‐based, and the hybrid method. These three methods were used to select essays included in the Automated Student Assessment Prize competition that were then classified using a scoring model that was trained with the bidirectional encoder representations from a transformer language model. All three active learning methods produced strong results, with the topological‐based method producing the most efficient classification. Growth rate accuracy was also evaluated. The active learning methods produced different levels of efficiency under different sample size allocations but, overall, all three methods were highly efficient and produced classifications that were similar to one another.

Abstract This module is designed for educators, educational researchers, and psychometricians who would like to develop an understanding of the basic concepts of validity theory, test validation, and documenting a “validity argument.” It also describes how an in‐depth understanding of the purposes and uses of educational tests sets the foundation for validation. We describe the benefits and limitations of educational tests, the concept of validity and why it is important, and the types of validity evidence that should be used to support the use of a test for a particular purpose. We also discuss the need for assessment programs to provide such evidence and how it should be interpreted and documented to use educational assessments to best serve education.

It was April of 2021. My son, Timothy, and I drove to a nearby city for him to take a college admission test. The test center closest to our home had quickly filled up due to reduced capacity (for social distancing) and increased demand. Securing a seat at a testing center half an hour away seemed to be luxury, considering the circumstances. This was a good enough reason for neither of us to complain about our lack of sleep. After dropping Timothy off, I decided to wait for him in the parking lot of the test center. I had brought a book with me and was excited about this excuse to take a brief break from the busyness of life and enjoy some quiet time while reading. I was not too far along in my reading when I looked up and saw Timothy returning to the car. “Wow. Time really flies when you're enjoying a good book,” I thought. However, I quickly realized that a college admission test would probably take longer than me reading a few pages of my book. After getting into the car, my son explained that the door to the testing center was locked and had never opened for him and a small group of other students waiting to take the test. Apparently, the test center was closed, but the students had not received any notifications about the closure. We were later told that an email about the closure had been dispersed; however, no one in this particular group of test-takers had received that email. This is just one example of how COVID-19 has had an impact on my life and the lives of those around me. Since the World Health Organization (WHO) declared a pandemic on March 11, 2020, the world it represents has been dramatically disrupted. Events have been canceled; restaurants were closed; cities were locked down; and the list goes on. The educational measurement world was not spared. For example, exams were canceled; schools were closed; and research studies were delayed. EM:IP published a special section on the impact of COVID-19 on the field of educational measurement in the Fall 2020 issue. Since then, so much has changed. The encouraging statistics of COVID-19 vaccines initially brought hope that the global disaster would end soon. The hope, however, shattered into pieces when new variants were detected one after another. What we know, now, is there are more variants in existence than the most frequently discussed Alpha, Delta, and Omicron variants. What we do not know is whether there are enough letters in the Greek alphabet to account for the present and future variants of COVID-19. When planning for this special section, I decided to design a cover graphic to complement the COVID-19 subject-matter. Initially, I was going to portray a rearview mirror to signal leaving behind the pandemic and continuing on our journey ahead. However, this idea was out of touch with the reality. Yes, the global disruption is about two years old, but we still have not seen the end of the tunnel. I finally settled on the idea of “Turning the Page.” The coronavirus may not disappear soon (or at all), but we should be ready to move forward. Now that we have more data on the lingering impact of COVID-19, the authors in this section (who are spread across three continents) speculate on different aspects of the impact on educational measurement. Some authors discuss college admissions and inequalities—hot topic issues from before 2020 that have been exacerbated by the pandemic. Others share their thoughts and experiences related to the changes to educational assessments, licensure exams, and certification exams. Still, others cover longitudinal analysis and meta-analysis. Online testing or remote assessment continues to be of great interest, as shown by several authors. Lastly, disrupted data are inevitable, especially during a pandemic. Two articles present some thoughts on decision-making using disrupted data. I hope you find this special section interesting and thought-provoking. I also hope, as Bennett's title suggests, you can look at “The Good Side of COVID-19.”

Abstract Technical documentation for educational tests focuses primarily on properties of individual scores at single points in time. Reliability, standard errors of measurement, item parameter estimates, fit statistics, and linking constants are standard technical features that external stakeholders use to evaluate items and individual scale scores. However, these cross‐sectional, “point‐in‐time” features can mask threats to the validity of score interpretations, including those for aggregate scores and trends over time. We use test score data collected before and during the COVID‐19 pandemic to show that longitudinal analyses, not just point‐in‐time analyses, are necessary to detect threats to desired inferences. We propose that educational agencies require and vendors include longitudinal data features, including “match rates” and correlations, as standard exhibits in technical documentation.

Abstract Module Overview: In this digital ITEMS module, Dr. Jodi M. Casabianca provides a primer on the hierarchical rater model (HRM) framework and the recent expansions to the model for analyzing raters and ratings of constructed responses. In the first part of the module, she establishes an understanding of the nature of constructed responses, the rating process and the common rater errors or rater effects originating from the rating process. She then compares traditional measures for analyzing raters and ratings to item response theory (IRT) measures, including the HRM. In the second part of the module, she provides details on the HRM, including its modularity with respect to using different combinations of IRT models, signal detection models, and latent trait structures. In the third section, she discusses some extensions including approaches for longitudinal and multidimensional data. The module contains audio‐narrated slides, quizzes with feedback, and additional resources such as a glossary and reference documents.