Refining Evaluation Functions for Game 2048 by Extended Temporal Difference Learning

Learning complex game functions is still a difficult task. We apply temporal difference learning (TDL), a well-known variant of the reinforcement learning approach, in combination with n-tuple networks to the game Connect-4. Our agent is trained just by self-play. It is able, for the first time, to consistently beat the optimal-playing Minimax agent (in game situations where a win is possible). The n-tuple network induces a mighty feature space: It is not necessary to design certain features, but the agent learns to select the right ones. We believe that the n-tuple network is an important ingredient for the overall success and identify several aspects that are relevant for achieving high-quality results. The architecture is sufficiently general to be applied to similar reinforcement learning tasks as well.

Accuracy of evaluation functions is one of the critical factors in computer game players. Evaluation functions are usually constructed manually as a weighted linear combination of evaluation features that characterize game positions. Selecting evaluation features and tuning their weights require deep knowledge of the game and largely alleviates such efforts. In this paper, we propose a new fast and scalable method to automatically generate game position features based on game records to be used in evaluation functions. Our method treats two-class problems which is widely applicable to many types of games. Evaluation features are built as conjunctions of the simplest features representing positions. We select these features based on two measures: frequency and conditional mutual information. To evaluate the proposed method, we applied it to 200,000 Othello positions. The proposed selection method is found to be effective, showing much better results than when simple features are used. The naive Bayesian classifier using automatically generated features showed the accuracy close to 80% in win/lose classification. We also show that this generation method can be parallelized easily and can treat large scale problems by converting these selection algorithms into incremental selection algorithms

Introduction It has been shown that a form of temporal difference (TD) learning for predicting the value of the membrane potential of a neuron, at a fixed delay after the neuron received a presynaptic spike, results in a learning rule that is very similar to hebbian spike-timing-dependent plasticity (STDP) [1]. Since this result was obtained using a relatively complex neural model (two-compartmental, with Hodgkin-Huxley-like currents) and a simple setup (a single presynaptic spike followed by a single current pulse), we investigated whether it holds for simpler neural models and for more general situations. This is relevant both for the theoretical understanding of this phenomenon and for verifying that it holds in common simulations of spiking neural networks.

Spellcaster is a turn-based strategy board game for two players, involves players casting spells by making sequences of gestures, aiming to defeat their opponent through different kinds of spells within a turn-based system. The research succeeded in the implementation of Spellcaster as a 2-player game system, with an AI opponent who can make reasonable decisions. The digitised game is developed using the LibGDX game engine because of its cross-platform features and high editability; and implemented in the Java language, using java's object-oriented features to improve programming efficiency. The game's AI opponent is implemented using the Mini-Max ideas, with an evaluation function that focuses on the gesture selection of the current turn.

With the proliferation of computing technologies and an ongoing trend of introducing digital and blended learning aspects into higher education, innovative approaches to teaching complex topics like artificial intelligence (AI) have emerged. Of particular interest is the use of game-based learning approaches. According to problem-based learning theory, providing students with an interactive problem and encouraging them to independently find solutions promotes deeper understanding and skill acquisition. Thus, game-based approaches offer an engaging way for students to explore challenging concepts. However, despite the growing use of game-based methods in fields like economics, their application in computer science - especially in teaching game AI - remains under-explored.Understanding AI is increasingly critical for game development, as modern games emulate human-like behaviour in areas such as decision-making, character routines, and adaptive strategies. While many mechanisms and approaches of agent-level decision-making and planning are well understood, their application in video games poses unique challenges, such as accommodating unpredictable player interactions and ensuring performance efficiency without degrading the player experience. While strategy games like StarCraft or multiplayer online battle arenas like DotA 2 have been of interest as proving grounds for advanced AI training methods, their use in education has been limited due to their high complexity and associated learning curve.This work proposes the development of a novel interactive application to fill this niche. Taking inspiration from city building and management games, the application simulates the campus of the University of Applied Sciences Mittweida, where students are given control of agents acting as archetypal roles of students. The agents' goal is the acquisition of knowledge, an abstract resource gained by participating in courses, requiring the agent to navigate the campus. Students interact with the system through an API that provides information on the state of the simulation and allows issuing commands to specific agents. For example, agents who continuously acquire knowledge over a prolonged period do so at decreasing efficiency. To remedy this, a student implements a routine checking the learning efficiency of all agents, commanding "tired" agents to take a break. Alternatively, the student could train a machine learning algorithm to do the same task, albeit more adaptive. Additionally, the application enables dynamic changes to the environment at runtime, such as adding or removing courses or buildings, simulating player-driven alterations to the game world. By designing decision-making algorithms for these agents, students gain hands-on experience with fundamental AI concepts, i.e. decision trees, bridging the gap between theoretical knowledge and practical application.To evaluate the effectiveness of the application, a comparative study with undergraduate students is planned. Over the course of two semesters, two groups of students will be taught the basics of game AI - one using traditional teaching methods (primarily lectures), the other using a game-based method incorporating the new application. The learning progress of both groups will be monitored using assignments, with students being given a sample project and tasked to develop a game AI solution, i.e. for a non-player character in a first-person shooter.

An approach to accelerating the learning process of the actor-critic learning algorithm for reinforcement learning is presented. The algorithm was derived from principles based on the prediction of average rewards and temporal difference (TD) learning with averaged and discounted rewards. The derived algorithm was applied to neural networks, demonstrating their effective operation in nonlinear control problems. The motivation of the proposed algorithm was to elaborate how a learning scheme, implemented by artificial neural networks (ANNs), can speed up learning processes based on an arrangement akin to the Dyna-Q learning, where a simulative model of the controlled plant is established for virtual learning between two control cycles. Instead of modeling the complicated plant, the approach just introduced a simple predictor of rewards for virtual learning in simulation mode. Two TD learning methods based discounted and averaged rewards respectively, are used alternatively in the control and simulation mode to facilitate the derived algorithm. The proposed Alternative Learning Critic (ALC) algorithm consists of two sub-systems: one is Evaluation Predictor (EP), which performs an approximation of a long-term evaluation function, and the other is an immediate action selector, which is composed of two ANNs: Action Controller (AC) and Reinforcement Predictor (RP). The proposed learning scheme is then applied to control a pendulum system for tracking a desired trajectory to demonstrate its applausive performance and robustness. Through reinforcement signals from the environment, the system takes an appropriate action to a plant with unknown dynamics so the actual output of the plant can track the desired one concisely within a short learning cycles. Further, ALC is used as the compensator of a PI controller, which is actually only working well on a linear system, to control that pendulum system. The results show the affined system, trained ALC and the PI controller can manipulate together on a nonlinear system with unknown dynamics.

The computer chess is an important research direction of artificial intelligence. Amazon chess game is an extremely complicated strategy game and the search algorithm relies heavily on the evaluation function. The design of evaluation function faces great challenges. This process requires a comprehensive consideration of the characteristics of the board and build up a series of complex formulas, which requires a lot of human experience to support. The author tries to apply the reinforcement learning method to the design process of Game of the Amazons evaluation function. Using the self-game generated data sets and reward signals to stimulates network, it could learn how to evaluate the situation of chess board without human experience. Then this neural network can be used as an evaluation function. Finally, through battling with the game of Amazon chess game using traditional evaluation function, it is verified that the evaluation function design method proposed in this paper can reduce the human experience requirement in the design process of Game of the Amazons evaluation function and effectively evaluate the situation.

Soft sensing methods were widely studied due to their attractive properties. The soft sensing models based on supervised learning neural networks were well researched in the last decade. This paper proposes a soft sensing method based on the temporal difference (TD) learning. TD methods are more preferable to deal with multi-step prediction problems that involve temporal sequences of a dynamical process. The soft sensor is implemented with an Elman neural network, a multilayer network with local feedback, which is trained by the TD algorithm. Finally, the TD-based soft sensor is applied to the Kappa number prediction in the batch kraft pulping process

Effects of a social regulation-based online learning framework on students’ learning achievements and behaviors in mathematics

Recently, we can easily obtain a fast computer and a digital video camera as their prices go down. In this paper, we describe a method and a system for analyzing performance of tennis players of a given singles tennis game. We assume that a movie of the game has been shot by a digital video camera which is fixed at a place such that the camera could film whole area of the tennis court and the two players, and that we have the sequence of the images in the computer. Using the positions of the players and the ball detected by techniques of image processing and partially by human assistance, we analyzed the game by defining an evaluation function. We made a prototype of a tennis-player-evaluation system by implementing the method. The system appropriately evaluated performances of players of some sample tennis games, although the evaluation function cannot analyze the player's tactics enough.

Autonomous agents are occupying more roles in our world than ever. They are present as AI in games, decide on which ads users see on the internet, and are even considered in more impactful environments such as finances, health, and perhaps traffic. As our expectations grow, their responsibilities become increasingly complex. To meet these demands, agents must reason over the current and potential future states of the environment. Reasoning over the state is not limited to understanding the latest observation but extends to inference over the hidden part of the environment. A natural solution is to maintain a probability distribution, or belief, over the state. Similarly, regarding future states, intelligent behavior considers the likelihood of multiple outcomes far into the future, also called planning. Both aspects of decision-making require full knowledge of the behavior, or dynamics, of the environment. These dynamics are used in planning to simulate future interactions. Similarly, imagined simulations of the system allow for reasoning about its current state. For example, a self-driving car may want to reason about the behavior of pedestrians for its decision-making (or indeed to estimate the likelihood of one being hidden from view!). Typically the dynamics are not fully known and, thus, these techniques are not directly applicable in practice. When the dynamics are not available, then the best thing we can do is to provide a prior over the unknown quantities instead. For example, even if there is no fully accurate model of pedestrians, we may provide a distribution over possible behaviors. This approach, called Bayesian reinforcement learning, allows us to provide the agent with expert knowledge. With this setup, it is now possible to update our understanding of the unknown quantities of the system when new data becomes available. The result is an inference problem where we maintain a belief over both the state and the dynamics of the environment. This Bayesian reinforcement learning approach was best formalized in the literature with the Bayes-adaptive models. Unfortunately, although elegant in theory, the approach had some limitations and saw little adoption in practice. In particular, no planner had been developed to allow for efficient action selection. Second, the type of prior knowledge that the proposed models were able to capture restricted their usage to small problems. Variants with more complex representations required strong priors and the structure of the dynamics would be assumed known. This thesis advances Bayesian partially observable reinforcement learning to non-trivial domains with several contributions. In particular, it discusses a holistic definition of the Bayesian inference problem, improved planning algorithms, and scalable model representations plus approaches for their approximation. First, we visualize the inference problem over state and dynamics with a graphical model and exploit its structure to define a novel posterior derivation. Not only does this derivation unify previous work into a single recipe, but also opens the door to other (machine learning) approaches. The resulting framework is formalized as the general Bayes-adaptive Markov decision process'' (GBA-POMDP). The second contribution is a family of efficient planning algorithms that were the first technique that made it possible to do decision-making in the Bayes-adaptive models. These planners are specializations of Monte-Carlo tree search, which are sample-based methods that approximate the value of future actions through simulations. In particular, we combine additional sampling approximations with inference simplifications to make reasoning over the complex GBA-POMDP possible. Third, we discuss two sophisticated models that move away from the tabular representations that were assumed in previous work. We show how a Bayes-net representation allows us to model and learn structure in the dynamics of the system. This includes the usage of an intricate and targeted sampling scheme to avoid the collapse of the posterior approximation. Lastly, we show the practical use of the GBA-POMDP withBayes-Adaptive Deep Dropout Reinforcement learning (BADDr)'', which employs neural networks to model the dynamics of the environment. BADDr, with the expressiveness and scalability of neural networks, showcases how Bayesian reinforcement learning can be both principled and practical.--Author's abstract

This study aims to determine the effect of formative assessment and learning approach to the mathematics learning outcome after controlling the numerical aptitude. It was a quasi-experiment with a sample of 186 students obtained by using multistage random sampling technique with 2x2 factorial designs. The data were analyzed by ANCOVA. After controlling the numerical aptitude, the results are: the mathematics learning outcome of the students who followed a contextual approach was better than the ones who followed a conventional learning approach, the mathematics learning outcome of the students who were given a performance assessment was better than the ones who were given a conventional assessment, the interaction between the learning approach and formative assessment affected the students learning outcome for mathematics, the students who followed a contextual learning approach were more suitable to be given a performance assessment, whereas the ones who followed a conventional learning approach were more appropriate to be given a conventional assessment. Based on the research findings, junior high school teachers are suggested to improve their students learning outcome for mathematics. Then, teachers need to use a learning approach and formative assessment accurately and correctly.

This study aims to determine the effect of formative assessment and learning approach to the mathematics learning outcome after controlling the numerical aptitude. It was a quasi-experiment with a sample of 186 students obtained by using multistage random sampling technique with 2x2 factorial designs. The data were analyzed by ANCOVA. After controlling the numerical aptitude, the results are: the mathematics learning outcome of the students who followed a contextual approach was better than the ones who followed a conventional learning approach, the mathematics learning outcome of the students who were given a performance assessment was better than the ones who were given a conventional assessment, the interaction between the learning approach and formative assessment affected the students learning outcome for mathematics, the students who followed a contextual learning approach were more suitable to be given a performance assessment, whereas the ones who followed a conventional learning approach were more appropriate to be given a conventional assessment. Based on the research findings, junior high school teachers are suggested to improve their students learning outcome for mathematics. Then, teachers need to use a learning approach and formative assessment accurately and correctly.

Although dating apps are the preferred means of meeting sexual and romantic partners, users frequently experience disappointment, highlighting the importance of understanding self-presentation strategies and selection processes to mitigate negative experiences. This study examines self-presentation strategies in Tinder profile pictures, aiming to identify typological patterns, characterize common profile-building strategies, and analyze differences by age, gender, and sexual orientation. We employed a mixed dimensional approach-both categorical and numerical-to characterize and categorize 1000 Tinder profile pictures. A descriptive category set was developed to analyze key picture elements, and a not-safe-for-work nudity index was computed using an open-source neural network. We then applied K-means clustering to identify patterns in the data. The main outcome measures included the clustering distribution of profile picture types and their associations with demographic variables (Standardized Pearson Residuals). Our analysis identified nine prototypical Tinder profile image categories, confirming the presence of stereotypical patterns in self-presentation. Additionally, we found statistically significant associations between profile clustering and user demographics, particularly age, gender, and sexual orientation. Stereotyped self-disclosure in dating apps may hinder partner selection by reinforcing social biases related to age, gender, and sexual orientation, with potential consequences for sexual and couple therapy. Key strengths include the use of a large and diverse dataset, robust cluster validation techniques, and a novel approach to analyzing self-presentation on dating apps. Limitations, however, include potential biases due to the categorical nature of the data, difficulties in capturing individual nuances in self-presentation, and the inability to account for Tinder algorithm influences on image use. Clustering techniques provide an empirical framework for identifying stereotypical self-presentation patterns and demographic differences, which could be extended to analyzing written descriptions and partner selection mechanisms.

Ant-TD: Ant colony optimization plus temporal difference reinforcement learning for multi-label feature selection