Process Data Analysis in ILSAs

This paper presents the concepts and observations in the author’s keynote address at the May 2019 “Opportunity versus Challenge: Exploring Usage of Log-File and Process Data in International Large-Scale Assessments” conference in Dublin, Ireland. This paper recaps briefly some key points that emerged at the December 2018 ETS symposium on process data in Washington, DC, and suggests some common terminology and concepts for facilitating future meaningful discussion. It then borrows some concepts from evolutionary biologists to take a critical look at process data and some of its ethical implications before turning to discuss major applications for and challenges of using process data and logfiles. The second half of this paper then maps out the terrain in which process data emerges as a “byproduct” of the shift from paper-and-pencil testing to digital-based assessments (DBA) and looks closely at how process data interacts with the development and operationalization of assessment systems. This interaction suggests new, fresh ways of improving national and international assessment system processes, and ultimately the measurement of cognitive skills, while venturing into the new frontiers opened with process data.

International large-scale assessments, such as PISA and TIMSS, have had a profound effect on educational policy and school accountability. In an open letter to the OECD in 2014, a large number of academics expressed their concerns over the continuous cycle of global testing and how it negatively affects student’s well-being and impoverishes classrooms, as it inevitably involves more and longer batteries of multiple-choice testing, more scripted “vendor”-made lessons, and less autonomy for teachers. Others have pointed to the ways in which international large-scale assessments have opened up education systems and built country’s assessment capacities. This chapter reviews the ways in which international standardized assessments permeate national policy and school-level decision-making, drawing on the sociology of numbers to understand the appeal of standardization and benchmarking. The OECD initiative “PISA for Schools” is described and how it potentially extends the influence of international assessments on the classroom. An alternative project within the OECD (the “Strategic Education Governance” project) is presented and how the approach of case studies and learning seminars allows countries to learn more holistically about how to hold schools accountable and improve education. Openness, situational awareness, and a consideration for the complexity of successful reform are presented as necessary “habits of mind” to successfully improve learning outcomes. The final conclusion reflects on the limitations and potential usage of international large-scale assessments for effective education reform.KeywordsAccountabilityCommensurationStandardizationEducational governance

Why are contractors keen to develop, implement, and analyse International Large-Scale Assessments (ILSAs) when they appear to make no financial gains? What makes ILSA contracts so attractive? This paper takes as its starting point the fact that almost all ILSA contractors interviewed stated their ILSA work is an investment, either in the form of a break-even or a loss contract. Applying policy borrowing and lending theory to non-state actors, and analysing 35 interviews with OECD and IEA staff and ILSA contractors, the paper discusses why contractors carry out ILSA contracts or donate to ILSAs. Rationales relate to interests in methodological developments and innovation; research; social responsibility; learning; exposure; prestige; credibility; networks; business opportunities; and individual rationales. The paper identifies parallels with the motives of corporate philanthropy in education. The paper concludes that ILSA contractors are using ILSAs to forward their political agendas, which include growing the learning assessment market.

International large-scale assessments such as PISA, TIMSS, PIRLS, and TALIS are among the most important tools available to policymakers and educational stakeholders for informing evidence-based decision-making in quality assurance processes. However, little is known about how pre-service teachers understand and evaluate the usefulness of this research. This study investigated pre-service teachers' attitudes towards the role of international large-scale assessments in quality assurance processes, as well as their factual knowledge in this field. We conducted a web-based survey of 85 master’s students, which included an attitude scale and a knowledge test. The results showed that participants generally expressed positive attitudes towards using international large-scale assessment results in quality processes, but their factual knowledge was low. Furthermore, we found that factual knowledge did not differ significantly between students with very positive and those with less positive attitudes, as the Mann–Whitney test was not statistically significant (p = .186) and the effect size was small (r = .16; PS = .582). These findings indicate a gap between endorsement of international large-scale assessments and the knowledge required for informed use, suggesting that initial teacher education should integrate compulsory, practice-oriented international large-scale assessment literacy (e.g., interpreting national reports, subgroup gaps, and translating results into lesson planning and school self-evaluation).

This chapter discusses the role of aesthetic education with a focus on educational drama and theatre. It investigates the lack of international large-scale assessment (ILSA) studies in the field of aesthetic education and exemplifies how to measure competence development in one of the aesthetic subjects: drama, based on the international mixed method large-scale assessment study DICE (Drama Improves Lisbon Key Competences in Education). The aim is to gain new understanding of the role of aesthetics in schooling, relating traditional philosophical arts theory from Aristotle and Dewey to relevant contemporary conceptualizations, such as twenty-first century skills (OECD), Lisbon Key Competences (EU), and Education for Sustainable Development (UNESCO). The discussion considers three main questions: Why does only a few international large-scale quantitative assessments of drama education exist? Why are researchers and practitioners in drama education skeptical about quantitative measurements? Can we design large-scale assessment studies in drama education?KeywordsEducational dramaApplied theatreArts educationAesthetic subjectsDICE

International large-scale assessments (ILSAs) are attracting global attention, but examining reasons for this sharp increase in demand is an under-explored area. Drawing on key concepts in policy borrowing research, this chapter synthesizes key explanations why ILSAs are so attractive to policy makers. From a demand side perspective, ILSAs captivate countries by projecting as follows: (i) the comparative advantage of numbers over narratives, (ii) the quest for a credible source of information in an era marked by a surplus of evidence, (iii) the weak link between national curriculum and some ILSAs, and as (iv) transnational accreditation of public education. On the supply side, two brand-new developments are identified: (i) the preoccupation with linking test accountability to “education is in crisis” and (ii) new ILSA derivative tools that urge countries to reconsider partial and non-participation. In the age of ILSA expansion and test-based accountability, countries are saturated in a surplus of assessments; yet, the predominant policy advice remains singular, and countries are discouraged from alternative non-standardized paths for measuring learning.

This simulation study was prompted by the current increased interest in linking national studies to international large-scale assessments (ILSAs) such as IEA’s TIMSS, IEA’s PIRLS, and OECD’s PISA. Linkage in this scenario is achieved by including items from the international assessments in the national assessments on the premise that the average achievement scores from the latter can be linked to the international metric. In addition to raising issues associated with different testing conditions, administrative procedures, and the like, this approach also poses psychometric challenges. This paper endeavors to shed some light on the effects that can be expected, the linkage errors in particular, by countries using this practice. The ILSA selected for this simulation study was IEA TIMSS 2011, and the three countries used as the national assessment cases were Botswana, Honduras, and Tunisia, all of which participated in TIMSS 2011. The items selected as items common to the simulated national tests and the international test came from the Grade 4 TIMSS 2011 mathematics items that IEA released into the public domain after completion of this assessment. The findings of the current study show that linkage errors seemed to achieve acceptable levels if 30 or more items were used for the linkage, although the errors were still significantly higher compared to the TIMSS’ cutoffs. Comparison of the estimated country averages based on the simulated national surveys and the averages based on the international TIMSS assessment revealed only one instance across the three countries of the estimates approaching parity. Also, the percentages of students in these countries who actually reached the defined benchmarks on the TIMSS achievement scale differed significantly from the results based on TIMSS and the results for the simulated national assessments. As a conclusion, we advise against using groups of released items from international assessments in national assessments in order to link the results of the former to the latter.

The aim of this special issue, “Learning Assessments for Sustainability?”, is to examine the interaction between the environmental and sustainability education (ESE) movement and the international large-scale assessments (ILSAs) movement. Both global educational movements emerged in the 1960s and their simultaneous work have affected each other since then. While the articles in this special issue highlight the potential benefits of ILSAs as a source of data for secondary analysis, they also demonstrate the limitations of ILSAs and their negative consequences to ESE. As such, we call for more research on the interaction between ESE and ILSAs and for a serious consideration of how test-based accountability practices might work against meaningful engagement with ESE. This introductory article includes three sections. The first section provides context about the movements. The second section presents an overview of the articles and alternative ways for reading them. The third section discusses lessons learned from the collection of articles. We conclude with a call for further research and reflection.

Meta-analyses and international large-scale assessments (ILSA) are key sources for informing educational policy, research, and practice. While many critical research questions could be addressed by drawing evidence from both of these sources, meta-analysts seldom integrate ILSAs, and current integration practices lack methodological guidance. The aim of this methodological review is therefore to synthesize and illustrate the principles and practices of including ILSA data in meta-analyses. Specifically, we (a) review four ILSA data inclusion approaches (analytic steps, potential, challenges); (b) examine whether and how existing meta-analyses included ILSA data; and (c) provide a hands-on illustrative example of how to implement the four approaches. Seeing the need for meta-analyses on educational inequalities, we situated the review and illustration in the context of gender differences and socioeconomic gaps in student achievement. Ultimately, we outline the steps meta-analysts could take to utilize the potential and address the challenges of ILSA data for meta-analyses in education.

This chapter synthesizes the existing evidence on the role of socioeconomic status (SES) in determining educational performance, attainment, and a large variety of adult outcomes. Special attention is given to studies that have used International Large-Scale Assessments (ILSAs) to document the intensity of the associations between these outcomes and SES, to allow their comparison across countries, or to highlight conceptual and methodological issues associated with the SES construct. An analysis of the associations between various education and labor market outcomes by one proxy of SES – mothers’ educational attainment – using two ILSAs – PISA 2021 and PIAAC 2021 – is provided to highlight that significant inequities exist for different age groups and across selected countries, and, given these, to suggest that different countries may have provided more equal opportunities to the various generations at different points in time. The chapter stresses the value of adopting an equity-in-inputs angle, which discusses SES’s direct influence on outcomes and its association with other factors that predict outcomes, when examining gaps in outcomes in school and adulthood. In combination with the wealth of data in ILSAs, this approach can support policy research in the quest for strategies to improve equity in our education systems and in our societies.

For over two centuries, education scholars and policy communities have sought to measure and compare education. However, it is only in recent decades that comparative measurements of learning began rising to prominence above all other educational data. Comparative learning measures are now so established that gathering comparative learning data has become an educational purpose in itself. This Commentary Paper provides a timely critical reflection on the impact of International Large-Scale Assessments (ILSAs) 30 years on from a key moment in their emergence. The first part of the paper offers a brief history of the rise of ILSAs and overview of ILSA research. The second part presents reflections from scholars who have made a significant contribution in this field – Rutkowski, Waldow, Gorur, Takayama, and Grek. They discuss what ILSAs have done for education, whilst reflecting on how ILSAs have created absences in the educational debate, and where attention might be productively re-directed in future scholarship.

International large-scale assessments (ILSAs) require national governments to invest significant resources in both time and money. With such investment national governments expect the results of ILSAs to provide policy and research communities with evaluative information on their educational system. Armed with this evaluative information, policymakers in many participating countries have used the results to stimulate reform. However, systematically tracking this influence and judging the validity of the claims has proven difficult for both the policy and research communities. There can be an erroneous expectation in the wider education community, and sometimes amongst policymakers themselves, that ILSA data automatically suggests policy solutions. Because of this error, a means for better systematizing the policymaking process responding to ILSA results is required. The model developed here can assist the policy and research community to better understand whether ILSAs are providing valid evidence to support their use in policy formation enactment and can be used to analyze ongoing consequences of that influence. Two worked examples demonstrate the utility of the model.

International large-scale assessments like international computer and information literacy study (ICILS) (Fraillon et al. in International Association for the Evaluation of Educational Achievement (IEA), 2015) provide important empirically-based knowledge through the proficiency scales, of what characterizes tasks at different difficulty levels, and what that says about students at different ability levels. In international comparisons, one of the threats against validity is country differential item functioning (DIF), also called item-by-country interaction. DIF is a measure of how much harder or easier an item is for a respondent of a given group as compared to respondents from other groups of equal ability. If students from one country find a specific item much harder or easier than students from other countries, it can impair the comparison of countries. Therefore, great efforts are directed towards analyzing for DIF and removing or changing items that show DIF. From another angle, however, this phenomenon can be seen not only as a threat to validity, but also as an insight into what distinguishes students from different countries, and possibly their education, on a content level, providing even more pedagogically useful information. Therefore, in this paper, the data from ICILS 2013 is re-analyzed to address the research question: Which kinds of tasks do Danish, Norwegian, and German students find difficult and/or easy in comparison with students of equal ability from other countries participating in ICILS 2013? The analyses show that Norwegian and Danish students find items related to computer literacy easier than their peers from other countries. On the other hand, Danish and, to a certain degree, Norwegian students find items related to information literacy more difficult. Opposed to this, German students do not find computer literacy easier, but they do seem to be comparably better at designing and laying out posters, web pages etc. This paper shows that essential results can be identified by comparing the distribution of difficulties of items in international large-scale assessments. This is a more constructive approach to the challenge of DIF, but it does not eliminate the serious threat to the validity of the comparison of countries.

Germany has relied on its well-established tradition of preparing the workforce through the so-called dual system of practical in-firm training (apprenticeships) and theoretical foundations conveyed by vocational schools. Believing in the high quality of academic school programmes that prepare a small elite for university studies, the German public remained convinced until well into the 1990s that there was little need for innovation and improvement in German education. Although there had been earlier signs of concern, this conviction remained unchanged until the publication of the outcomes of the International Association for the Evaluation of Educational Achievement (IEA) Third International Mathematics and Science Study (TIMSS) in 1996. The ‘TIMSS shock’, reinforced by later studies, has resulted in a host of changes in educational policy, perhaps best characterised by the late Hermann Lange as the ‘turn towards evidence-based educational policy making’. Hallmarks of this trend were the decision by all 16 federal states in Germany to participate regularly in international large-scale assessments (ILSAs), including interstate comparisons (taken in 1997); the decision to define standards of educational performance (corresponding to the federally recognised school leaving certificates); the initiation of academic study programmes intended to foster the development of the required advanced technical skills; and the founding and/or expansion of agencies for quality management (including systems of reports on the state of education, both at the federal and the state levels). In this article, the role of ILSAs (including both IEA and others) in these policy changes are described and examined from a critical perspective.

This chapter outlines the motivation behind the creation of this book on foundations, innovations, and quality assurance in digital-based international large-scale assessments (ILSAs). It examines the critical role of technology as a driving force in the evolution of ILSAs, highlighting how advancements in digital tools and methodologies have transformed assessment practices in the past decades. The chapter also provides a comprehensive overview of the book’s content, detailing each part’s key themes and contributions. Additionally, it presents a perspective on the future of digital-based ILSAs, discussing emerging trends and potential directions for research and practice in this rapidly changing field.