Game Theory based Peer Grading Mechanisms for MOOCs

A close look at the educational literature reveals both negative perceptions among peer assessment participants and evidence that peer assessment may not as effective as generally thought. An analysis of peer assessment using game theory is proposed and examined. This examination helps to explain the aforementioned lack of effectiveness and negative perceptions. The resulting model suggests interventions that may mitigate these negative perceptions and lead to an increase in peer assessment effectiveness.

With the rapid advancement of internet technology, online learning has emerged as a vital avenue for university students to acquire knowledge. Peer assessment, recognized as an effective pedagogical strategy, is extensively implemented in online learning environments. This paper aims to explore the factors influencing the online learning outcomes of university students based on peer assessment. Firstly, it elucidates the definition and principles of peer assessment, analyzing its implementation process and advantages within the online learning context. Secondly, it examines the impact of peer assessment on university students' online learning outcomes from four perspectives: students' educational backgrounds, motivation for learning, the quality of peer assessment, and the guidance and support provided by educators. Subsequently, a research design is established, employing surveys and interviews to empirically analyze the relevant factors affecting the online learning efficacy of students engaged in peer assessment. Finally, the results of the study are used to provide strategic support for faculty online teaching evaluation, which is then used to provide targeted recommendations for online education practices in colleges and universities.

In an online learning course like MOOC (Massive Open Online Course), peer grading is useful for scaling the grades of assignments to the large number of students. Peer grading is also attractive because it reduces the burden of teachers and gives rich feedback to learners. In Kyushu University, we have a demand for using peer grading in some classes that include a large number of students. However, the discussion about operating peer grading in our university has not been discussed sufficiently. We developed the online peer grading system to cope with the large number of students in the class. Students can upload their assignments, grade others, and receive feedback from other students via the system. We operated it in our university's class in order to verify whether peer grading works well and return accurate scores. In this paper, we verify it with an analysis of the data of user behavior, and also comparison between peer and teacher grading scores. We show questionnaire results. As a result, although there are still issues to improve, we find the validity in peer grading to some extent.

Peer grading affords a scalable and sustainable way of providing assessment and feedback to a massive student population, and has been used in massive open online courses (MOOCs) on the Coursera platform. However, currently there is little empirical evidence to support the credentials of peer grading as a learning assessment method in the MOOC context. To address this research need, this study examined 1825 peer grading assignments collected from a Coursera MOOC with the purpose of investigating the reliability and validity of peer grading as well as its perceived effects on students’ MOOC learning experience. The empirical findings proved that the aggregate ratings of student graders can provide peer grading scores that were fairly consistent and highly similar to the instructor grading scores. Student responses to a survey also show that the peer grading activity was well received as the majority of MOOC students believed it was fair, useful, beneficial, and would recommend it to be included in future MOOC offerings. Based on the empirical results, this study concludes with a set of principles for designing and implementing peer grading activities in the MOOC context.

Open online distance learning in higher education has quickly gained popularity, expanded, and evolved, with Massive Open Online Courses (MOOCs) as the most recent development. New web technologies allow for scalable ways to deliver video lecture content, implement social forums and track student progress in MOOCs. However, we remain limited in our ability to assess complex and open-ended student assignments. In this paper, we present a study on the quality of self- and peer assessments in three MOOCs. In general, the quality of self-assessments and peer assessments was low to moderate, suggesting that both self-assessment and peer assessment should be used as assessment for learning instead of assessment of learning. Based on low correlations with final exam grades and other assessment forms, we conclude that self-assessments might not be a valid way to assess students’ performance in MOOCs. Yet the weekly quizzes and peer assessment significantly explained differences in students’ final exam scores, with one of the weekly quizzes as the strongest predictor. Future research on MOOCs implies a reconceptualization of education variables, including the role of assessment of students’ achievements.

Effect sizes and research directions of peer assessments: From an integrated perspective of meta-analysis and co-citation network

BackgroundPeer assessment has played an important role in large‐scale online learning, as it helps promote the effectiveness of learners' online learning. However, with the emergence of numerical grades and textual feedback generated by peers, it is necessary to detect the reliability of the large amount of peer assessment data, and then develop an effective automated grading model to analyse the data and predict learners' learning results.ObjectivesThe present study aimed to propose an automated grading model with reliability detection.MethodsA total of 109,327 instances of peer assessment from a large‐scale teacher online learning program were tested in the experiments. The reliability detection approach included three steps: recurrent convolutional neural networks (RCNN) was used to detect grade consistency, bidirectional encoder representations from transformers (BERT) was used to detect text originality, and long short‐term memory (LSTM) was used to detect grade‐text consistency. Furthermore, the automated grading was designed with the BERT‐RCNN model.Results and ConclusionsThe effectiveness of the automated grading model with reliability detection was shown. For reliability detection, RCNN performed best in detecting grade consistency with an accuracy rate of 0.889, BERT performed best in detecting text originality with an improvement of 4.47% compared to the benchmark model, and LSTM performed best with an accuracy rate of 0.883. Moreover, the automated grading model with reliability detection achieved good performance, with an accuracy rate of 0.89. Compared to the absence of reliability detection, it increased by 12.1%.ImplicationsThe results strongly suggest that the automated grading model with reliability detection for large‐scale peer assessment is effective, with the following implications: (1) The introduction of reliability detection is necessary to help filter out low reliability data in peer assessment, thus promoting effective automated grading results. (2) This solution could assist assessors in adjusting the exclusion threshold of peer assessment reliability, providing a controllable automated grading tool to reducing manual workload with high quality. (3) This solution could shift educational institutions from labour‐intensive grading procedures to a more efficient educational assessment pattern, allowing for more investment in supporting instructors and learners to improve the quality of peer feedback.

Contributors

One of the major challenges facing Massive Open Online Courses (MOOCs) is assessing the learner performances beyond traditional automated assessment methods. This leads to a bottleneck problem due to the massiveness of course participants, especially in the context of problem solving. To tackle this issue, peer assessment has been proposed as an effective method. However, the validity of this process is still under discussion, suffers from a lack of credibility and has many weaknesses, particularly with regards to group formation. This paper develops a new method of peer assessment for MOOCs to improve the accuracy and exactitude of the learner grade. Our proposition is based on three main steps: the formation of learner groups, the assessment and synthesis of the results. First, the group definition process can use different elements of the learner model and enables to build heterogeneous groups. After, each learner is required to grade a small number of peer productions. Finally, a synthesis of the various grades is proposed using both data about the ability to assess of each learner and complexity of problems. To evaluate the proposed peer assessment process, we conducted an experimentation devoted to teaching Software Quality Assurance to beginners with computer science during the first university cycle.

On-line peer assessment and the role of the peer feedback: A study of high school computer course

Given examinees' performance (i. e., scores) on each problem, cognitive diagnosis models can discover the latent characteristics of examinees. Traditional cognitive diagnosis models require teachers to provide scores in time. Thus we can hardly apply traditional models in large-scale scenarios, such as Massive Open Online Courses (MOOC). Peer assessment refers to a teaching activity in which students evaluate each other's assignments. The scores given by students could replace the teacher's assessments to a certain extent. In this paper, we propose a novel cognitive diagnosis model named Peer-Assessment Cognitive Diagnosis Framework (PACDF). This model combines peer assessments with cognitive diagnosis, aiming at reduce the burden of teachers. Specifically, we propose a novel probabilistic graphic model at first. This model characterizes not only the relationships between real scores and scores given by peer assessment, but also the relationship between examinees' skill proficiency and problem mastery. Then we adopt Monte Carol Markov Chain (MCMC) sampling algorithm to estimate the parameters of the model. Lastly, we use the model to predict examinees' performance. The experimental results show that PACDF could quantitatively explain and analyze skill proficiencies of examinees, thus perform better in predicting examinees' performances.

Peer Assessment is an essential activity in massive open online courses (MOOCs) to provide students with evaluation and feedback in a timely manner. However, the lack of instructor involvement and the diverse student population have raised the concern regarding its credibility, and there has been insufficient validity evidence to support the interpretation and refinement of current peer assessment approach. The present paper proposed a comprehensive framework to examine the validity evidence of peer assessment scores in MOOCs by analyzing the interplay among peer assessment ratings, instructor ratings, and various peer engagement behaviors in MOOCs. The study results can extend our understanding of peer assessment approach and inform its future usage and refinement in MOOCs.

Objective: economic and legal description, formal justification and construction of models of the balanced economic, legal and social macrosystems in order to determine effective strategic decisions under uncertainty.Methods: the theoretical and methodological basis of the study is the synthesis of the provisions and principles of game and management theories, economic and mathematical modeling and the game-theoretical models of decision-making in complex systems developed on their basis, as well as theoretical and methodological approaches of system analysis, economics, sociology, and law.Results: the balance of economic, legal and social systems is determined by the target orientation to the growth of the population’s well-being. The existing mechanisms for achieving this balance do not take into account the interests of all participants of the systems. The paper presents a structural analysis of the interaction and mutual influence of economic, legal and social macrosystems (a complex metasystem); the necessity of transforming the existing approaches to their regulation is shown. The problem of ensuring the systems’ balancing is largely related to the conflict of interests of the parties. Therefore, traditional methods of modeling decision-making processes do not reflect the real socio-economic processes. This problem was solved through game-theory models that contribute to the implementation of the constitutionally established norms which define Russia as a social state. When making strategic decisions in the social sphere, it is proposed to use the concept of the Golden rule of morality as the main economic doctrine, instead of the currently used neoliberal approach, based, among other things, on the concept of Nash equilibrium. The equilibrium models of the complex metasystem by Nash and Berge were formalized, constructed and compared.Scientific novelty: in contrast to the Nash equilibrium (principles of rationality) - the traditional approach used in the construction of equilibrium models, - the paper offers a theoretical justification of the development and construction of equilibrium models and balance sheet equilibrium of economic, legal and social macrosystems by Berge, as well as a formalization of effective strategic decisions and their respective risks.A distinctive feature of the constructed macromodels is the use of the philosophical and moral concept of the Golden rule (the mathematical basis is the balance by Berge) as an economic doctrine for strategic decision-making under uncertainty, and the construction of the so-called guaranteed optimal solutions by Pareto (effective), based, among other things, on the concept of the pair “social guarantee - life risk”. If, in real conditions, one of the macrosystems is economic and is a “generator” of uncertainties, a guaranteed solution is formalized, optimal by Slater (weakly effective).Practical significance: the developed methodology for building the system equilibrium of economic, legal and social macro-systems enables to make decisions socially significant for the population that adequately take into account the peculiarities of the modern socio-economic macro-systems, such as nonstationarity, dysfunctionality, incomplete information, inconsistency, conflict, alternatives, multicriterial character, and risk; and to develop assisting intellectual systems of supporting decision-making based on mathematical models built using the game-theory approach.

As the number of participants in online distance learning courses increases, peer assessment is becoming a popular strategy for evaluating open assignments and for breaking the social isolation surrounding distance education. Yet, the quality and characteristics of peer assessment in massive online courses has received little attention. Hence, this study was set to examine peer feedback quality and grading accuracy in a project-based course. The study applied a sequential exploratory mixed methods design. It included 339 participants who studied the same engineering course, but in three different modes: on-campus (n = 77), small private online course (n = 110), and massive open online course (MOOC) (n = 152). Content analysis of feedback comments identified four categories: reinforcement, statement, verification and elaboration, arranged in an ascending scale of cognitive ability. The findings indicated that the MOOC participants provided more feedback comments and volunteered to assess more projects than their counterparts did. However, the on-campus students provided higher quality feedback and their peer grading was better correlated with the grades assigned by the teaching assistants.

Mathematical modeling is both a skill and an art and must be practiced in order to maintain and enhance the ability to use those skills. Though the topics covered in this book are the typical topics of most mathematical modeling courses, this book is best used for individuals or groups who have already taken an introductory mathematical modeling course. Advanced Mathematical Modeling with Technology will be of interest to instructors and students offering courses focused on discrete modeling or modeling for decision making. Each chapter begins with a problem to motivate the reader. The problem tells "what" the issue is or problem that needs to be solved. In each chapter, the authors apply the principles of mathematical modeling to that problem and present the steps in obtaining a model. The key focus is the mathematical model and the technology is presented as a method to solve that model or perform sensitivity analysis. We have selected , where applicable to the content because of their wide accessibility. The authors utilize technology to build, compute, or implement the model and then analyze the it. Features: MAPLE©, Excel©, and R© to support the mathematical modeling process. Excel templates, macros, and programs are available upon request from authors. Maple templates and example solution are also available. Includes coverage of mathematical programming. The power and limitations of simulations is covered. Introduces multi-attribute decision making (MADM) and game theory for solving problems. The book provides an overview to the decision maker of the wide range of applications of quantitative approaches to aid in the decision making process, and present a framework for decision making. Table of Contents 1. Perfect Partners: Mathematical Modeling and Technology 2. Review of Modeling with Discrete Dynamical Systems and Modeling Systems of DDS 3. Modeling with Differential Equations 4. Modeling System of Ordinary Differential Equation 5. Regression and Advanced Regression Methods and Models 6. Linear, Integer and Mixed Integer Programming 7. Nonlinear Optimization Methods 8. Multivariable Optimization 9. Simulation Models 10. Modeling Decision Making with Multi-Attribute Decision Modeling with Technology 11. Modeling with Game Theory 12. Appendix Using R Index Biographies Dr. William P. Fox is currently a visiting professor of Computational Operations Research at the College of William and Mary. He is an emeritus professor in the Department of Defense Analysis at the Naval Postgraduate School and teaches a three-course sequence in mathematical modeling for decision making. He received his Ph.D. in Industrial Engineering from Clemson University. He has taught at the United States Military Academy for twelve years until retiring and at Francis Marion University where he was the chair of mathematics for eight years. He has many publications and scholarly activities including twenty plus books and one hundred and fifty journal articles. Colonel (R) Robert E. Burks, Jr., Ph.D. is an Associate Professor in the Defense Analysis Department of the Naval Postgraduate School (NPS) and the Director of the NPS’ Wargaming Center. He holds a Ph.D. in Operations Research form the Air Force Institute of Technology. He is a retired logistics Army Colonel with more than thirty years of military experience in leadership, advanced analytics, decision modeling, and logistics operations who served as an Army Operations Research analyst at the Naval Postgraduate School, TRADOC Analysis Center, United States Military Academy, and the United States Army Recruiting Command.