Editor's Introduction

International assessments provide important knowledge about science education and help inform decisions about policies, programmes, and practices in participating countries. In 2006, science was the primary domain for the Programme for International Student Assessment (PISA), supported by the Organisation for Economic Cooperation and Development (OECD) and conducted by the Australian Council for Educational Research (ACER). Compared to the school curriculum orientation of Trends in International Math and Science Study (TIMSS), PISA provides a perspective that emphasises the application of knowledge to science and technology‐related life situations. The orientation of PISA includes both knowledge and attitudes as these contribute to students’ competencies that are central to scientific literacy. In addition to students’ knowledge and competencies, the 2006 PISA survey gathered data on students’ interest in science, support for scientific enquiry, and responsibility towards resources and environments. The survey used both a non‐contextualised student questionnaire and contextualised questions. The latter is an innovative approach which embedded attitudinal questions at the conclusion of about two‐thirds of the test units. The results presented in this article make connections between students’ attitudes and interests in science and scientific literacy.

A number of East Asian education systems (e.g. Hong Kong, Japan, South Korea, Shanghai, Singapore and Taiwan) have gained international prominence over the years, due to their rapid climbs to top positions in international tests such as the Programme for International Student Assessment (PISA), Progress in International Reading Literacy Study (PIRLS) and Trends in International Mathematics and Science Study (TIMSS). For example, Hong Kong was placed 17th in PIRLS reading literacy, and Singapore ranked 15th in 2001, but within 5 years, they garnered 2nd and 4th places in 2006, respectively. In the 2015 PISA results, Singapore, Hong Kong, Taipei, Japan and Beijing-Shanghai-Jiangsu-Guangzhou (B-S-J-G), China, were top achievers (OECD, Report on PISA 2015: results in focus. Organisation for Economic Co-operation and Development, 2016).

Over the last 20 years, since the launch of its flagship study the Programme for International Student Assessment (PISA), the Organization for Economic Co-operation and Development (OECD) has become a behemoth of transnational influence on education policy making. To better understand the evolution of the OECD’s perspective on education and the importance of PISA within the OECD’s overall publications, I analysed over 900 OECD publications on education with a topic modelling approach. The analysis revealed that contrary to popular belief, the OECD is a multi-centric organization with various agendas in education policy making, focusing on economic changes, higher education, management, planning, and budgeting. PISA made up only 12% of the overall text corpus and was therefore far less discussed than expected. This is in sharp contrast to the general impression held, namely that the OECD’s work in education mostly relates to PISA. Therefore, this study reveals that the OECD has a far broader agenda than anticipated. This paper acts as a systematic review of OECD literature to gain insights into its institutional objectives.

The results of the PISA (Program for International Student Assessment) survey in 2012-2015 on the achievement of mathematical competencies of Indonesian students showed a significant increase, but the overall achievement was still below the average of the countries belonging to the Organization for Economic Co-operation and Development (OECD). Furthermore, the results of the Trends in International Mathematics and Science Study (TIMSS) report showed that many students like and feel good about mathematics, but their confidence in their mathematical abilities was quite low. Many studies reveal the close association of Mathematics Self-Efficacy (MSE) with the performance/achievement of students' mathematical competencies. In 2015, the PISA survey was done using computerization except in 15 countries, one of them was in Indonesia. Therefore, the results of this study are to produce the first computer-based MSE scale developed in Indonesia.

A decade on, the situation portrayed in the influential report of a deteriorating US science and technology ecosystem has, if anything, worsened.

Student scores from the Third International Mathematics and Science Study (TIMSS) have been employed as a benchmark to measure school effectiveness in many districts across the United States. Bracey (2000) discussed the TIMSS results at the high school level. This article focuses on an in-depth examination of student performance from the TIMSS primary and middle school studies. The data analyses show substantial uncertainty of the country rankings from the released TIMSS reports. Other technical issues hinge on the instrument construction, curricular inequivalency, and statistical outliers. These analyses raise concerns about the use of the TIMSS benchmark for school reforms in the U.S. Taken together, these concerns have important implications for the interpretation and use of the TIMSS results.

Middle school data from the Third International Mathematics and Science Study (TIMSS) are analyzed in this article to compare achievement difference between seventh and eighth grades. The statistical computing involves a selective subtraction of item scores in more than 40 countries. The research findings indicate that not all TIMSS items have resulted in a higher mean score at the upper grade level. Item features are discussed to characterize part of the released TIMSS instrument that generates a higher average score at the lower grade. This empirical study may help enrich understanding of the TIMSS benchmark among mathematics and science educators.

The OECD Programme for International Student Assessment (PISA), launched by governments of the Organisation for Economic Co-operation and Development (OECD) in 1997, aims at assessing some of the key competencies that contribute to the success of 15-year-old individuals, on a regular basis and within an internationally accepted framework. PISA seeks to provide a basis for policy dialogue and for collaboration in defining and implementing educational goals, in innovative ways that reflect judgements about the skills that are relevant to adult life. PISA defines competence as the ability to successfully meet complex demands in varied contexts through the mobilisation of psychosocial resources, including knowledge and skills, motivation, attitudes, emotions, and other social and behavioural components. Measuring and comparing competencies across languages and cultures is a difficult challenge and is being pursued by PISA progressively. PISA focused its first three assessments on literacy skills, defined as the capacity of young adults to access, manage, integrate and evaluate information, to think imaginatively, to hypothesise and discover, and to communicate their thoughts and ideas effectively. The reasoning behind shifting the emphasis from assessing whether students can reproduce what they have learned towards whether they can extrapolate from what they have learned and apply their competencies in novel situations, derives from the nature of knowledge and skills required in modern life. For example, the tasks that can be solved through simple memorisation or with pre-set algorithms are those that are also easiest to digitise, automatise and offshore, and will thus be less relevant in a modern knowledge society. Since there is no overarching cross-national and cross-cultural agreement on what fundamental competencies 15-year-olds should possess, an international assessment such as PISA can only capture a selection of competencies. Moreover, since various methodological constraints limit the nature of competencies that are currently amenable to large scale assessment, PISA cannot capture the entirety of competencies that will make young people successful. However, the findings presented in this article suggest that the competencies that PISA does assess are highly predictive for the future success of students. In addition, PISA provides policy makers and practitioners with useful tools to improve quality, equity and efficiency in education, by revealing some common characteristics of students, schools and education systems that do well. In a modern world, comparative assessments are an essential tool for educational improvement and research shows that the existence of standardised assessments and examinations is one of the most powerful predictors for the success of an education system. That is not hard to understand, because without such assessments, all students, schools and education systems look the same, it is impossible for teachers and school administrators to detect institutional and systemic strengths and weaknesses, and to support and intervene where expectations are not met. Without reliable and comparable information on learning outcomes, teachers and governments alike rely on input-based incentives and policies that are all too often mirrored in large quality variation between schools as well as a strong dependency between learning success and the socio-economic context of students and schools. Last but not least, it is important to keep in mind that the absence of the reading, mathematical and scientific competencies measured by PISA does not automatically imply the presence of all those important competencies that have not been measured.

The research examines the role of using ICT to raise students’ achievement in Italian technical and professional schools. The hypothesis developed in this research is that students who obtain better learning results are those students who use ICT more than those students who are low performers. Data from the Programme for International Student Assessment (PISA) 2009 by OECD (Organisation for Economic Co-operation and Development) have been analyzed. The relationship between using ICT at school and students’ digital skills and attitudes has been analyzed by comparing the lowest performers and strong performers in reading literacy in Italian technical and professional schools. Results show that in Italian technical and professional schools, strong performers in reading literacy in PISA 2009 are more capable to do tasks at computer than low performers; however reading online and using ICT have greater positive effects towards achievement among low performers. Key words: ICT, PISA, professional school, reading literacy, technical school.

During the last decade, in many European countries large investments were made to foster the use of information and communication technology (ICT) in education with the expectation that ICT would make teaching and learning more effective. This would, for example, become evident in scores obtained by students in the Programme for International Student Assessment (PISA). PISA studies show, however, that PISA achievements do not increase with ICT use. The question for research is why high-frequency ICT use tends to be related to lower PISA achievements. To answer this question, the author has looked at PISA results published by the Organisation for Economic Cooperation and Development (OECD) as well as at the results of his own analyses of PISA data. While it seems that ICT use as well as PISA achievements are related to cultural capital, the findings are not clear-cut and vary among the three European countries chosen for investigation.

One of the overarching goals of international large-scale assessments (ILSA) is to inform public discourse about the quality of education in different countries. To fulfil this function, the Organisation for Economic Co-operation and Development (OECD), for example, raises awareness of the Program for International Student Assessment (PISA) results through different forms of traditional and social media (e.g. press releases and other activities under the slogan PISA Day). Scholars have responded to the rapid growth of ILSA by examining public discourse through newspaper articles, policy documents, and other outlets. However, we know very little about whether and to what extent the general public is actually affected by PISA results. In order to address this gap, this study uses data regarding public trust in education from the 2011 wave of the International Social Survey Program (ISSP). Drawing on survey data from 30 countries and Hierarchical Linear Models (HLM), the study shows that PISA rankings have a significant effect on public perceptions. We find that in high performing countries the general public expresses higher levels of confidence in the education system. We discuss these patterns in the context of growing politicisation of education policy making and the use of ILSA as evidence.

If you are aware of the results given in the media reports about the Third International Mathematics and Science Study (TIMSS), you probably know that fourth graders from the United States (U.S.) scored above the international average in mathematics and that eighth and twelfth graders scored below average (Mullis et al. 1997). As an educator, you are aware of the dangers of looking only at averages of test scores. Rich information can be gleaned from the TIMSS data that will help us learn more about what our students know and are able to do. The data from a large-scale study, such as the TIMSS, often raise questions about what the numbers really mean. This article addresses one such question that arose from examining part of the third- and fourth-grade TIMSS data. The process that we used may be as valuable as the information that we found. Perhaps this process will help you answer questions that arise as you reflect on the TIMSS results.

Predecessor items and performance level

A GREAT deal has been made of student achievement data in international comparisons. Less information has been gathered about in various countries. We have a few field studies, such as those reported in The Teaching Gap, by James Stigler and James Hiebert, and in Knowing and Teaching Elementary Mathematics, by Liping Ma. Neither depicts a particularly robust portrait of what American know about mathematics or how to teach it. With the new emphasis on highly qualified teachers from the U.S. Department of Education, teacher quality has moved to center stage. Two new studies add a bit to our knowledge of here and abroad. The first is Teaching Mathematics in Seven Countries: Results from the TIMSS 1999 Video Study, in which videotapes of actually teaching eighth-grade mathematics in seven countries were studied. The report (NCES 2003013) was put together by Hiebert and a host of other researchers and is available at www.nces.ed.gov/timss. The second is Preparing Teachers Around the World, from Aubrey Wang, Ashaki Coleman, Richard Coley, and Richard Phelps of the ETS Policy Information Center. It is available at www.ets.org/research/pic/teachprep.pdf. In 1995, as part of the original TIMSS (Third International Mathematics and Science Study), videos of were made in three nations: Germany, Japan, and the United States. In general, U.S. and German focused on teaching procedures, while Japanese emphasized conceptual matters. One accidental finding cropped up when people coding the videos had a What the hell was that? reaction to noises on the tapes. Those noises were mostly interruptions by the intercom, visitors, etc. More than 30% of American classes suffered interruptions. Zero percent of Japanese classes suffered an intrusion. The TIMSS 1999 Video Study extends the original research to seven countries, adding Australia, the Czech Republic, Hong Kong, the Netherlands, and Switzerland and omitting Germany. (A more detailed summary of this study, by the TIMSS Video Mathematics Research Group, can be found in the June 2003 Kappan.) These countries all scored high or relatively high in TIMSS and TIMSS-R (Repeat). In TIMSS-R, only the Czech Republic did not have a score that was significantly higher than that of the U.S., which scored 502 as compared to the Czech Republic's 520. (The international average was 487.) Getting coders to agree on what they were seeing was difficult in the original study, and it was exponentially more difficult to get coders to agree about what they were seeing when viewing videos from seven countries. In addition, they might have missed something important, as overall judgments of quality did not correlate with judgments on the specific indicators of quality. (These results are from a presentation made at the annual meeting of the American Educational Research Association in April, not from the report itself.) Coders first looked at how much time in different countries spent introducing new content, practicing new content, or reviewing earlier content. American spent more time reviewing old material than anyone else except the Czechs. None of the countries spent a great deal of time practicing new material, but Japanese did allot 60% of their time to introducing new content. Only Japan's lessons were considered to be of high or moderate complexity, 39% and 45% respectively. The U.S. had the fewest lessons considered to be of high complexity (6%) and the third-highest number judged to be of low complexity (67%), behind Australia (77%) and the Netherlands (69%). Only Japanese offered many proofs. Twenty- six percent of the problems in any given lesson contained proofs, and 39% of the lessons contained at least one proof. Second-place Switzerland posted just 3% and 11% respectively. The researchers asked whether the lessons consisted of stating concepts, using procedures, or making connections between mathematical facts. …

THE GOLDEN treasury of selected, spun, and distorted statistics from 1983 that we know as A Nation at Risk said, only to keep and improve on the slim competitive edge we retain in world markets, we must dedicate ourselves to the reform of our educational system. document tightly yoked our economic position in the world to how well or poorly elementary and secondary students bubbled-in answer sheets. This was utter nonsense, but few recognized it as such. distinguished historian of education, Lawrence Cremin, was one who did. He pointed out that competitiveness depended mostly on the actions taken by the President, Congress, and several federal departments -- not to mention things that happen in other nations. Cremin wrote: To conclude that the problems of competitiveness can be solved by educational reform, especially educational reform defined solely in terms of school reform, is not merely utopian and millennialist, it is at best a foolish and at worst a crass effort to direct attention away from those truly responsible for doing something about competitiveness and to lay the burden on the schools. It is a device that has been used repeatedly in the history of American education. Cremin notwithstanding, A Nation at Risk's spurious argument was widely accepted. theory enjoyed great popularity during the recession of 1990-91. Many variations on the lousy-schools-are-producing-a-lousy- work-force-and-that-is-killing-us-in-the-global-marketplace argument could be heard. Fortunately, the economy wasn't paying attention, and by late 1993 and early 1994, many publications featured banner headlines about the recovery that was to become the longest sustained period of growth in the nation's history. The American Economy: Back on Top was one such headline in the New York Times. Well, if the schools took the rap when the economy went south, surely they would get the credit when the economy turned around, right? Hardly. Just three months after that headline in the Times, IBM CEO Louis Gerstner, Jr., took to that paper's op-ed page with an essay headlined Our Schools Are Broken. They are broken, Gerstner charged, because they are not producing students who can compete with their peers. Recently, I quantified the relationship between test scores and competitiveness. I correlated the Current Competitiveness Index (CCI), as calculated by the World Economic Forum in Geneva, with test scores in mathematics from the Third International Mathematics and Science Study (TIMSS). World Economic Forum ranked 75 nations according to their CCI; 41 countries took part in TIMSS. Thirty-eight countries had ranks on both. results are shown in Table 1. briefest glance reveals that there is little relationship between the two columns. For example, the United States is 29th in TIMSS mathematics, but second in competitiveness. Korea is third in mathematics, but 27th in competitiveness. And so on. Overall, the correlation between test score ranks and competitiveness ranks is +.23. This is tiny as correlations go. It means that test scores account for just 5% of what makes up a nation's competitiveness. (The square of the correlation coefficient is the amount of one variable accounted for by another.) And, actually, this figure is misleadingly high. bottom seven countries are low on both variables and so create what little relationship there is. If these seven nations are removed from the calculation, the correlation between test scores and competitiveness actually becomes negative. Indeed, the CCI does not ignore schools, and, in fact, the TIMSS scores from both mathematics and science are included, as are survey rankings from executives that the World Economic Forum claims have international perspectives. (The inclusion of TIMSS scores in the CCI makes the low correlations even more remarkable. …