Multilevel Modeling with HLM: Taking a Second Look at PISA

In the most recent edition of the Handbook of Reading Research, Ebony Elizabeth Thomas and colleagues (2020) identify the need to recontextualize critical literacy pedagogy and research in ways that center Black and Indigenous communities. Although critical literacy has a rich tradition in emancipatory work (e.g. Freire, 1996), Thomas et al. argue for the need to begin elsewhere, with the knowledge and traditions of Black and Indigenous communities, to produce literacy research and pedagogy that is more responsive to current social issues and iterations of racism. In this article, I combine their insights with those of early literacy researchers, such as Candace R. Kuby and Tara Gutshall Rucker, (e.g. Kuby, 2013; Kuby and Rucker, 2020) to suggest that the shifting of theoretical framings may be a useful way of broadening the context of critical literacies research and scholarship. Kuby and Rucker’s (2020) examination explores offerings from postructural and posthumanist theories for expanding conceptions of literacy in efforts to challenge our ideas of literacies, inequalities, and justice in early literacy research. Similarly, I analyze the affordances of Pro-Black and marginalized theories in critical literacies research, such as Black Feminist Thought (Collins, 2002), for understanding and expanding how we attend to positionality in early literacy educational research. Grounded in Black Feminist theories (Collins, 2002) and methodologies (Evans-Winters, 2019) as well my own positionality as a Black woman, this article takes up one specific concern from the call for this special issue, that is to address the lack of Pro-Black research in early literacy education. In so doing, I aim to broaden the ways that a historically praxis-oriented body of research in early childhood and elementary education, critical literacies research, is theorized and enacted by integrating more thoroughly Black women’s ways of knowing. I explore how to leverage our individual and combined perspectives, which are grounded in our rich history of resistance and thriving in the face of adversity, to produce knowledge and literacy practices useful for justice-oriented education. Through an analysis of academic literature related to critical literacies research and Black women educators in early childhood and elementary education, I address the question: How are Black women’s ways of knowing integrated in early childhood and elementary critical literacies research with participants who are Black women educators? I offer a sense of the extent to which Black women educators’ ways of knowing are associated with the term critical literacy and also identify fruitful strategies that critical literacies researchers can use to integrate Black women’s ways of knowing into the knowledge base and practices of critical literacies in early literacy research. Some strategies include integrating Black women educators' emotions and spiritual knowledge into the research and co-researching with Black women who are also participants.

I shall attempt to demonstrate here that the general categories of hermeneutics can be applied just as well to the natural sciences as to the humanities, and I shall explore the implications of this demonstration for both types of disciplines. In doing so, it will be necessary for me to raise some unfashionable issues about the objectivity and rationality of critical interpretations, and, as a result, I may appear to be cast in the role of advocatus diaboli. The value of lowering the critical barriers that have been set up between the human and natural sciences, however, will outweigh any incidental paradoxes that may crop up en route.

The US Institute of Education Sciences (IES) currently provides a progressive series of education research grants that can support the development of innovative and creative educational programs, starting with exploratory grants, and progressing through developmental grants, efficacy trials, and finally, effectiveness studies. The IES further requires an evaluation component for each of these program developmental stages. In this article, the authors report on their experiences as evaluators of the developmental stage of a digital educational program called Scholar. Their reflections concentrate on what might be uniquely relevant to digital educational programs and to programs at the IES developmental stage. The key issues discussed include: (a) the challenge of assessing implementation fidelity and variability for a developmental project; (b) the challenge of assessing the program's reach to diverse program participants, especially those least well served in a given context; and (c) the challenges of securing sufficient and relevant implementation data, and of addressing pressures for gathering preliminary efficacy data. Interwoven through these discussions are snapshots of selected program participants' voices describing their own program experiences and views. The article concludes with the authors' reflections and commentary on future directions for evaluating comparable digital educational programs at the IES developmental stage.

Previous articleNext article No AccessA New Use of Randomization in Experimental Curriculum EvaluationHerbert J. Walberg and Wayne W. WelchHerbert J. Walberg Search for more articles by this author and Wayne W. Welch Search for more articles by this author PDFPDF PLUS Add to favoritesDownload CitationTrack CitationsPermissionsReprints Share onFacebookTwitterLinkedInRedditEmail SectionsMoreDetailsFiguresReferencesCited by Volume 75, Number 4Winter, 1967 Article DOIhttps://doi.org/10.1086/442821 Views: 3Total views on this site Citations: 26Citations are reported from Crossref Journal History This article was published in The School Review (1893-1979), which is continued by the American Journal of Education (1979-present). Copyright 1967 University of ChicagoPDF download Crossref reports the following articles citing this article:Joy Cumming, Adrian Coulston, John Elkins Planning for incomplete data collection in educational research, Research Papers in Education 2, no.11 (Jul 2006): 31–46.https://doi.org/10.1080/0267152870020103Barry J. Fraser Learning environment in curriculum evaluation: A review, Evaluation in Education 5, no.11 (Jan 1981): 1–93.https://doi.org/10.1016/0191-765X(81)90014-8VICTOR L. WILLSON Research Techniques in AERJ Articles: 1969 to 1978, Educational Researcher 9, no.66 (Jul 2016): 5–10.https://doi.org/10.3102/0013189X009006005Frances Lawrenz The stability of student perception of the classroom learning environment, Journal of Research in Science Teaching 14, no.11 (Jan 1977): 77–81.https://doi.org/10.1002/tea.3660140113Frances Lawrenz Student perception of the classroom learning environment in biology, chemistry, and physics courses, Journal of Research in Science Teaching 13, no.44 (Jul 1976): 315–323.https://doi.org/10.1002/tea.3660130405William H. Ward A test of the association of class size to students' attitudes toward science, Journal of Research in Science Teaching 13, no.22 (Mar 1976): 137–143.https://doi.org/10.1002/tea.3660130206Frances Lawrenz The relationship between science teacher characteristics and student achievement and attitude, Journal of Research in Science Teaching 12, no.44 (Oct 1975): 433–437.https://doi.org/10.1002/tea.3660120415Wayne W. Welch The process of evaluation, Journal of Research in Science Teaching 11, no.33 (Sep 1974): 175–184.https://doi.org/10.1002/tea.3660110303Wayne W. Welch Review of the research and evaluation program of harvard project physics, Journal of Research in Science Teaching 10, no.44 (Dec 1973): 365–378.https://doi.org/10.1002/tea.3660100411Wayne W. Welch, Arlen R. Gullickson A Strategy for Evaluating the NSF Comprehensive Program for Teacher Education, School Science and Mathematics 73, no.99 (Dec 1973): 759–767.https://doi.org/10.1111/j.1949-8594.1973.tb09166.xBikkar S. Randhawa, Lewis L. W. Fu Assessment and Effect of Some Classroom Environment Variables, Review of Educational Research 43, no.33 (Jun 2016): 303–321.https://doi.org/10.3102/00346543043003303Wayne W. Welch, Herbert J. Walberg A National Experiment in Curriculum Evaluation, American Educational Research Journal 9, no.33 (Jun 2016): 373–383.https://doi.org/10.3102/00028312009003373Gary J. Anderson Effects of Course Content and Teacher Sex on the Social Climate of Learning, American Educational Research Journal 8, no.44 (Jun 2016): 649–663.https://doi.org/10.3102/00028312008004649Wayne W. Welch, Herbert J Walberg Pretest and Sensitization Effects in Curriculum Evaluation, American Educational Research Journal 7, no.44 (Jun 2016): 605–614.https://doi.org/10.3102/00028312007004605Gary J. Anderson Effects of Classroom Social Climate on Individual Learning, American Educational Research Journal 7, no.22 (Jun 2016): 135–152.https://doi.org/10.3102/00028312007002135Herbert J. Walberg, Andrew Ahlgren Predictors of the Social Environment of Learning, American Educational Research Journal 7, no.22 (Jun 2016): 153–167.https://doi.org/10.3102/00028312007002153Herbert J. Walberg Predicting Class Learning An Approach to the Class as a Social System, American Educational Research Journal 6, no.44 (Jun 2016): 529–542.https://doi.org/10.3102/00028312006004529Herbert J. Walberg Class Size and the Social Environment of Learning, Human Relations 22, no.55 (Oct 1969): 465–475.https://doi.org/10.1177/001872676902200507Wayne W. Welch 4: Curriculum Evaluation, Review of Educational Research 39, no.44 (Jun 2016): 429–443.https://doi.org/10.3102/00346543039004429Herbert J. Walberg, Wayne W. Welch, Arthur I. Rothman Teacher heterosexuality and student learning, Psychology in the Schools 6, no.33 (Jul 1969): 258–266.https://doi.org/10.1002/1520-6807(196907)6:3<258::AID-PITS2310060307>3.0.CO;2-KRobert L. Baker 4: Curriculum Evaluation, Review of Educational Research 39, no.33 (Jun 2016): 339–358.https://doi.org/10.3102/00346543039003339Gary J. Anderson, Herbert J. Walberg, Wayne W. Welch Curriculum Effects on the Social Climate of Learning: A New Representation of Discriminant Functions, American Educational Research Journal 6, no.33 (Jun 2016): 315–328.https://doi.org/10.3102/00028312006003315Wayne W. Welch Correlates of courses satisfaction in high school physics, Journal of Research in Science Teaching 6, no.11 (Mar 1969): 54–58.https://doi.org/10.1002/tea.3660060112Wayne W. Welch, Herbert J. Walberg, Andrew Ahlgren The Selection of a National Random Sample of Teachers for Experimental Curriculum Evaluation, School Science and Mathematics 69, no.33 (Mar 2010): 210–216.https://doi.org/10.1111/j.1949-8594.1969.tb08443.xHERBERT J. WALBERG, GARY J. ANDERSON The Achievement-Creativity Dimension and Classroom Climate*, The Journal of Creative Behavior 2, no.44 (Dec 2011): 281–291.https://doi.org/10.1002/j.2162-6057.1968.tb00118.xWayne W. Welch, Robert G. Bridgham Physics Achievement Gains as a Function of Teaching Duration*, School Science and Mathematics 68, no.55 (Mar 2010): 449–454.https://doi.org/10.1111/j.1949-8594.1968.tb15413.x

Since it entered education in the mid-20th century, qualitative research has been a well-established methodology among educational researchers. Its strength lies in its nature, scope, and methodological flexibility to understand education's cultural, social, political and moral phenomena. Between the 1970s and 1980s, the relevance and superiority of quantitative vs. qualitative research in education were debated in academic forums. The paradigm debate was so intense that it reached political discussions. In 2002, the National Research Council (NRC) of the United States of America recommended the Federal Department of Education the exclusion of qualitative research from federal funds because it was not considered scientific research. A similar situation was observed in England and Australia. Since then, qualitative research has continued to develop as a model of educational research with the support of private funds. This paper revisits qualitative education research and examines its developments to reposition itself as a legitimate model of scholarly research in a political-scientific era.

The demand for scientifically-based educational research has fostered a new methodological orthodoxy exemplified by documents such as the National Research Council's Scientific Research in Education and Advancing Scientific Research in Education and American Educational Research Association's Standards for Reporting on Empirical Social Science Research in AERA Journals. This article criticizes the new orthodoxy as being a throwback to positivist reductionism and the “two dogmas” of educational research: the quantitative/ qualitative incompatibility thesis and fact/value dichotomy. It then criticizes the new orthodoxy for fostering a “third dogma” of educational research cut from the same cloth as the first two: the empirical science/humanities dualism. The article advances the view that no fundamental epistemological dividing line can be drawn between the empirical sciences and the humanities and that, accordingly, empirical research in education should not be cordoned off from the humanities, particularly their focus on values. It concludes with several observations about the problems and prospects for interdisciplinary research in education across the empirical science/humanities divide.

This report is the fourth of four that stem from a study of the Impact of Approved Induction

Contents: L.A. Shepard, Preface. J.L. Green, G. Camilli, P.B. Elmore, Introduction to the Handbook: What's Complementary About Complementary Methods. A. Skukauskaite, E. Grace, On Reading and Using the Volume: Notes to Students. Part I: Foundations. E. Bredo, Philosophies of Educational Research. G.J. Kelly, Epistemology and Educational Research. K.A. Strike, The Ethics of Educational Research. Part II: Introduction to Design & Analysis. C. Bazerman, Analyzing the Multidimensionality of Texts in Education. T. Barone, Arts-Based Educational Research. R.K. Yin, Case Study Methods. K.M. Borman, C. Clarke, B. Cotner, R. Lee, Cross-Case Analysis. A.C. Porter, Curriculum Assessment. K.H. Kim, P.M. Bentler, Data Modeling: Structural Equation Modeling. F. Erickson, Definition and Analysis of Data From Videotape: Some Research Procedures and Their Rationales. A.H. Schoenfeld, Design Experiments. H-Z. Ho, S.L. O'Farrell, S. Hong, S. You, Developmental Research: Theory, Method, Design, and Statistical Analysis. D. Bloome, C. Clark, Discourse-In-Use. H. Braun, Empirical Bayes. J.P. Shaffer, Estimation. K.M. Anderson-Levitt, Ethnography. J. Nespor, Finding Patters With Field Notes. R.J. Shavelson, N.M. Webb, Generalizability Theory. J.L. Rury, Historical Research in Education. A. Henry, Historical Studies: Groups/Institutions. M.E. Brenner, Interviewing in Educational Research. L. Crocker, Introduction to Measurement Theory. S. Embretson, X. Yang, Item Response Theory. D.M. Harrison, S.W. Raudenbush, Linear Regression and Hierarchical Linear Models. G.V. Glass, Meta-Analysis: The Quantitative Synthesis of Research Findings. C.A. Chinn, The Microgenetic Method: Current Work and Extensions to Classroom Research. M.L. Smith, Multiple Methodology in Education Research. F.M. Connelly, D.J. Clandinin, Narrative Inquiry. N.C. Burbules, B.R. Warnick, Philosophical Inquiry. M. Cochran-Smith, K. Donnell, Practitioner Inquiry: Blurring the Boundaries of Research and Practice. J.R. Levin, Probability and Hypothesis Testing. W.R. Shadish, J.K. Luellen, Quasi-Experimental Design. T.D. Cook, V. Sinha, Randomized Experiments in Educational Research. M. Eisenhart, Representing Qualitative Data. B. Thompson, Research Synthesis: Effect Sizes. H. Wainer, P.F. Velleman, Statistical Graphics: A Guidepost for Scientific Discovery. M. Berends, Survey Methods in Educational Research. J. Chromy, Survey Sampling. Part III: Programs of Research. C. Genishi, T. Glupczynski, Language and Literacy Research: Multiple Methods and Perspectives. L.M. McNeil, E.M. Coppola, Official and Unofficial Stories: Getting at the Impact of Policy on Educational Practice. M.E. Goertz, Policy Analysis: Studying Policy Implementation. S. Kushner, C. Adelman, Program Evaluation: A Democratic Practice. L.A. Rex, S.C. Steadman, M.K. Graciano, Researching the Complexity of Classroom Interaction. J.A. Banks, Researching Race, Culture, and Difference: Epistemological Challenges and Possibilities. J.G. Greeno, Theoretical and Practical Advances Through Research on Learning. S. Noffke, K. Zeichner, Programs of Research in Teacher Education.

Abstract: This paper begins by raising some questions about the purpose and scope of research in science education, followed by a commentary on the questions that have occupied the attention of most researchers in science education in recent years. The final section of the paper offers an opinion of some current strengths and weakness of science education as a field of research. The paper argues for a more generous interpretation of what constitutes research in science education and suggests that much remains to be done if the outcomes of such research are to influence policy and practice in science education.

This essay addresses the topic of the state of qualitative research in education by asserting that qualitative research in education is in quite a state. Drawing heavily on Denzin and Lincoln’s periodization of qualitative research as a guide, it outlines the various competing developments from within and outside that are vying to characterize the current moment and illustrates the difficulty of pinpointing the moment. Arguing for a conception of overlapping moments rather than a neat historical progression, the essay posits that the current period is simultaneously one of overt politicization, epistemological and paradigmatic proliferation, post‐posts (post‐postmodernism, post‐poststructuralism, post‐experimentation) and a new post (postcolonialism), as well as a new or renewed paradigm war. The conclusion drawn is that the current/next moment in qualitative research in education is one of methodological contestation, one that demands either complicity with or resistance to the government‐sanctioned resurgence of the hegemony of positivism.

Scientists are becoming increasingly aware of the need to approach their teaching with the same expectations for evidence that they use in their science. They are coming to recognize that, just as in science research, research on undergraduate science education has a literature as well as standards for practice. I (M.L.L.) am a recent convert to this way of thinking. I taught for 30 years using mostly a lecture style and believed myself quite revolutionary when I used projected images to make cell biological points in class. I now teach using the results of discipline-based education research (DBER) and do my best to use instructional strategies that are demonstrated to improve student learning.

This chapter reviews social justice research in science education, focusing on the developmental trajectory and theories of method that drive the research. I address the following questions in this review: What are some of the trends in social justice citations? What methodologies are employed in social justice research in science education? What are the epistemological, ontological, catalytic, and tactical assumptions embedded in the research methodologies? What are the implications for future research in social justice in science education? After an analysis of trends, I discuss early framing of social justice in science education research. Then, I review five studies, focusing on the authors’ social and theoretical positioning and on ways in which critical methodologies blend scholarship and activism. I conclude with several implications for future social justice research in science education.

The Elementary School Journal Volume 84, Number 4 S1984 by The University of Chicago. All rights reserved. 001 3-5984/84/8404-0004$01.00 Assumptions, even assumptions that seem logical and reasonable, can make it difficult or impossible for people to understand scientific concepts. Everyone makes assumptions about the way the world works, assumptions like "When the sky is cloudy and dark, it will probably rain" or "Bits of wood float in water." People often use those assumptions to explain how things work: "I can see myself in a mirror because light bounces off me to the mirror and off the mirror to my eyes." Such explanations or conceptions are often based on experience and common sense; however, experience and common sense can sometimes

Even though humanity faces grand challenges, including climate change, sustainability of the planet and its resources, and well-being of humans and other species, for the past 60 years science educators have been preoccupied with much the same priorities. Adherence to the tenets of crypto-positivism creates problems for research in the social sciences (e.g., over reliance on statistical analyses leads to oversimplified models that strip away context and are reductive). Hypotheses and associated statistical tests support causal models that rarely predict social conduct or blaze pathways for meaningful transformation. In contrast to the mainstream of research in science education, I advocate a multilogical methodology that embraces incommensurability, polysemia, subjectivity, and polyphonia as a means of preserving the integrity and potential of knowledge systems to generate and maintain disparate perspectives, outcomes, and implications for practice. In such a multilogical model, power discourses such as Western medicine carry no greater weight than complementary knowledge systems that may have been marginalized in a social world in which monosemia is dominant. I describe research methodologies that have the potential to transform science education and our ongoing research in urban science education. I show how our research evolved to include studies of science for literate citizenry – expanding foci to address birth through death and all settings in which learning occurs – not just schools. Our research aims to be transformative since it includes interventions developed to use what we learned from research to ameliorate intense emotions, improve learning, and enhance the well-being of participants. I explain how we incorporated Jin Shin Jyutsu, a complementary medical knowledge system, to ameliorate intense emotions, become mindful, and improve well-being of participants. I also address research on meditation and mindfulness and their potential to improve learning, emotional styles, and wellness. In a final section I address three of the most important questions raised by colleagues, including scholars from Asia, as I exhort science educators to address grand challenges that threaten the Earth and its social institutions – the alternatives are catastrophic.

Black feminist thought and qualitative research in education is guided by a particular understanding of the learning strategies informed by Black women’s historical experiences with race, gender, and class. Scholars of Black feminist thought remind us of a Black feminist pedagogy that fosters a mindset of intellectual inclusion. Black feminist thought challenges Western intellectual traditions of exclusivity and chauvinism. This article presents a synopsis of the nature and scope of Black feminist thought and qualitative research in education. Further, this article highlights the work of scholars who describe the importance of an Afrocentric methodological approach in the field of education because it offers scholars and practitioners a methodological opportunity to promote equality and multiple perspectives.