Monitoring Affective Trajectories During Complex Learning

The role that affective states play in learning was investigated from the perspective of a constructivist learning framework. We observed six different affect states (frustration, boredom, flow, confusion, eureka and neutral) that potentially occur during the process of learning introductory computer literacy with AutoTutor, an intelligent tutoring system with tutorial dialogue in natural language. Observational analyses revealed significant relationships between learning and the affective states of boredom, flow and confusion. The positive correlation between confusion and learning is consistent with a model that assumes that cognitive disequilibrium is one precursor to deep learning. The findings that learning correlates negatively with boredom and positively with flow are consistent with predictions from Csikszentmihalyi's analysis of flow experiences.

In an attempt to discover the facial action units for affective states that occur during complex learning, this study adopted an emote-aloud procedure in which participants were recorded as they verbalised their affective states while interacting with an intelligent tutoring system (AutoTutor). Participants’ facial expressions were coded by two expert raters using Ekman's Facial Action Coding System and analysed using association rule mining techniques. The two expert raters received an overall kappa that ranged between .76 and .84. The association rule mining analysis uncovered facial actions associated with confusion, frustration, and boredom. We discuss these rules and the prospects of enhancing AutoTutor with non-intrusive affect-sensitive capabilities.

Simple SummaryEmotions or affective states recognition in farm animals is an underexplored research domain. Despite significant advances in animal welfare research, animal affective state computing through the development and application of devices and platforms that can not only recognize but interpret and process the emotions, are in a nascent stage. The analysis and measurement of unique behavioural, physical, and biological characteristics offered by biometric sensor technologies and the affiliated complex and large data sets, opens the pathway for novel and realistic identification of individual animals amongst a herd or a flock. By capitalizing on the immense potential of biometric sensors, artificial intelligence enabled big data methods offer substantial advancement of animal welfare standards and meet the urgent needs of caretakers to respond effectively to maintain the wellbeing of their animals.Farm animals, numbering over 70 billion worldwide, are increasingly managed in large-scale, intensive farms. With both public awareness and scientific evidence growing that farm animals experience suffering, as well as affective states such as fear, frustration and distress, there is an urgent need to develop efficient and accurate methods for monitoring their welfare. At present, there are not scientifically validated ‘benchmarks’ for quantifying transient emotional (affective) states in farm animals, and no established measures of good welfare, only indicators of poor welfare, such as injury, pain and fear. Conventional approaches to monitoring livestock welfare are time-consuming, interrupt farming processes and involve subjective judgments. Biometric sensor data enabled by artificial intelligence is an emerging smart solution to unobtrusively monitoring livestock, but its potential for quantifying affective states and ground-breaking solutions in their application are yet to be realized. This review provides innovative methods for collecting big data on farm animal emotions, which can be used to train artificial intelligence models to classify, quantify and predict affective states in individual pigs and cows. Extending this to the group level, social network analysis can be applied to model emotional dynamics and contagion among animals. Finally, ‘digital twins’ of animals capable of simulating and predicting their affective states and behaviour in real time are a near-term possibility.

Intelligent tutoring systems (ITSs) are computer programs that provide instruction adapted to the needs of individual students. Dialog systems are computer programs that communicate with human users by using natural language. This paper presents a systematic literature review to address ITSs that incorporate dialog systems and have been implemented in the last twenty years. The review found 33 ITSs and focused on answering the following five research questions. a) What ITSs with natural language dialogue have been developed? b) What is the main purpose of the tutoring dialogue in each system? c) What are the pedagogical features of the teaching process performed by the ITSs with natural language dialogue? d) What natural language understanding approach does each system employ to understand students’ utterances? e) What evidence exists related to the evaluation of ITSs with natural language dialogue? The results of this review reveal that most ITSs are directed toward science, technology, engineering, and mathematics (STEM) domains at the university level, and the majority of the selected ITSs implement the expectations and misconceptions tailored approach. Furthermore, most ITSs use dialog to help students learn how to solve a problem by applying rules, laws, etc. (the apply level in Bloom’s taxonomy). With regard to the instructional approach, the selected ITSs help students write correct explanations or answers for deep questions; assist students in problem solving; or support a reflective dialogue motivated by either previously provided content or the result of a simulation. Additionally, we found empirical evaluations for 90.91% of the selected ITSs that measure the learning gains and/or assess the impacts of different tutoring strategies.

A modern Intelligent Tutoring System (ITS) should be sentient of learner's cognitive and affective states, as learner's performance could be affected by motivational and emotional factors. It is important to design a low-cost and unobtrusive computational method for ITS to determine learner's states automatically. The automated learner's data sensing is useful in the development of personalized e-learning that can adapt learning content according to individual's emotion. Although many past findings relate stressor such as task demand to learner's emotion, however there is a lack of existing studies that examine the impact of stressor on learners' cognitive states and their mouse and keystroke dynamics. This exploratory research, conducted on 160 undergraduate students, found that the impacts of direct learning instruction and external stress stimuli, such as timer display, on learner's cognitive and affective states are significant. The correlations between direct instruction, external stimuli, learners' cognitive and affective states, as well as their mouse and keystroke behaviours are also significant.

This paper investigates how frequent conversation patterns from a mixed-initiative dialogue with an intelligent tutoring system, AutoTutor, can significantly predict users' affective states (e.g. confusion, eureka, frustration). This study adopted an emote-aloud procedure in which participants were recorded as they verbalized their affective states while interacting with AutoTutor. The tutor-tutee interaction was coded on scales of conversational directness (the amount of information provided by the tutor to the learner, with a theoretical ordering of assertion > prompt for particular information > hint), feedback (positive, neutral, negative), and content coverage scores for each student contribution obtained from the tutor's log files. Correlation and regression analyses confirmed the hypothesis that dialogue features could significantly predict the affective states of confusion, eureka, and frustration. Standard classification techniques were used to assess the reliability of the automatic detection of learners' affect from the conversation features. We discuss the prospects of extending AutoTutor into an affect-sensing intelligent tutoring system.

We explored the reliability of detecting a learner's affect from conversational features extracted from interactions with AutoTutor, an intelligent tutoring system (ITS) that helps students learn by holding a conversation in natural language. Training data were collected in a learning session with AutoTutor, after which the affective states of the learner were rated by the learner, a peer, and two trained judges. Inter-rater reliability scores indicated that the classifications of the trained judges were more reliable than the novice judges. Seven data sets that temporally integrated the affective judgments with the dialogue features of each learner were constructed. The first four datasets corresponded to the judgments of the learner, a peer, and two trained judges, while the remaining three data sets combined judgments of two or more raters. Multiple regression analyses confirmed the hypothesis that dialogue features could significantly predict the affective states of boredom, confusion, flow, and frustration. Machine learning experiments indicated that standard classifiers were moderately successful in discriminating the affective states of boredom, confusion, flow, frustration, and neutral, yielding a peak accuracy of 42% with neutral (chance = 20%) and 54% without neutral (chance = 25%). Individual detections of boredom, confusion, flow, and frustration, when contrasted with neutral affect, had maximum accuracies of 69, 68, 71, and 78%, respectively (chance = 50%). The classifiers that operated on the emotion judgments of the trained judges and combined models outperformed those based on judgments of the novices (i.e., the self and peer). Follow-up classification analyses that assessed the degree to which machine-generated affect labels correlated with affect judgments provided by humans revealed that human-machine agreement was on par with novice judges (self and peer) but quantitatively lower than trained judges. We discuss the prospects of extending AutoTutor into an affect-sensing ITS.

Giving agents the ability to sense, recognise and appropriately respond to the human emotions is one of the main methods to make the agents more believable. In intelligent tutoring systems, learners’ affective states have been incorporated in providing adaptive feedback. In game industry, however, emotions of players have not been paid enough attention. Games normally do not take into account of players’ emotions when they play games. Non-player characters do not respond to players’ emotional status. We argue that adapting to players’ emotions can make the non-player characters more believable and the game more enjoyable.

Many previous studies have highlighted the influence of learners’ affective states on learning with tutoring systems. However, the associations between learning and learners’ meta-affective capability are still unclear. The goal of this paper is to analyse meta-affective capability and its influence on learning outcomes as well as the dynamics of affect over time. Two criteria, awareness and self-regulation, were employed to define meta-affective capability. An exploratory study (n = 54) was conducted in which students at the secondary level were asked to interact with an intelligent tutoring system for mathematics and to self-report their affect during their interactions with the system. Pre-post learning outcomes were also measured. A post-hoc comparison of learning gains was made between more meta-affectively capable and less meta-affectively capable students. The results provide some empirical evidence to support the hypothesis that having meta-affective capability is positively associated with learning. Students not demonstrating meta-affective capability seemed to transition frequently from boredom to frustration (p = .0284) and from concentration to neutral (p = 0.0017). However, only a small percentage of the sample were classified as having meta-affective capability, indicating that it is important to scaffold students who are not meta-affectively capable.

Conversational Intelligent Tutoring Systems (C-ITS) are capable of providing personalized instruction to students in different domains, using dialogues in natural language to interact with the user. In this paper we describe our recent experience in adapting an algebraic/arithmetic word problem-solving Intelligent Tutoring System by replacing the original button-based user interface by a conversational one primarily based on the use of natural language. The proposed method led to an average intent-recognition accuracy of 0.97, supporting the use of this kind of tools to seamlessly migrate existing interfaces to a more convenient form of interaction based on natural language.KeywordsConversational agentsIntelligent Tutoring Systems (ITS)Natural Language Understanding (NLU)

The link between affect and student learning has been the subject of increasing attention. Because of the possible impacts of affective state on learning, it is a goal of many intelligent tutoring systems to attempt to control student emotional states through affective interventions. While much work has gone into improving the quality of these interventions, we are only beginning to understand the complexities of the relationships between affect, learning, and feedback. This paper investigates the consequences associated with providing affective feedback. It represents a first step toward the long-term objective of designing intelligent tutoring systems that can utilize this information for analysis of the risks and benefits of affective intervention. It reports on the results of two studies that were conducted with students interacting with affect-informed virtual agents. The studies reveal that emotion-specific risk/reward information is associated with particular affective states and suggests that future systems might leverage this information to make determinations about affective interventions.

In this paper we explore findings from three experiments that use minimally invasive sensors with a web based geometry tutor to create a user model. Minimally invasive sensor technology is mature enough to equip classrooms of up to 25 students with four sensors at the same time while using a computer based intelligent tutoring system. The sensors, which are on each student’s chair, mouse, monitor, and wrist, provide data about posture, movement, grip tension, arousal, and facially expressed mental states. This data may provide adaptive feedback to an intelligent tutoring system based on an individual student’s affective states. The experiments show that when sensor data supplements a user model based on tutor logs, the model reflects a larger percentage of the students’ self-concept than a user model based on the tutor logs alone. The models are further expanded to classify four ranges of emotional self-concept including frustration, interest, confidence, and excitement with over 78% accuracy. The emotional predictions are a first step for intelligent tutor systems to create sensor based personalized feedback for each student in a classroom environment. Bringing sensors to our children’s schools addresses real problems of students’ relationship to mathematics as they are learning the subject.KeywordsSkin ConductanceSensor FeatureIntelligent Tutoring SystemLearning CompanionFacial Expression Recognition SystemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we investigate the correspondence between student affect and behavioural engagement in a web-based tutoring platform throughout the school year and learning outcomes at the end of the year on a high-stakes mathematics exam in a manner that is both longitudinal and fine-grained. Affect and behaviour detectors are used to estimate student affective states and behaviour based on post-hoc analysis of tutor log-data. For every student action in the tutor, the detectors give us an estimated probability that the student is in a state of boredom, engaged concentration, confusion, or frustration, and estimates of the probability that the student is exhibiting off-task or gaming behaviours. We used data from the ASSISTments math tutoring system and found that boredom during problem solving is negatively correlated with performance, as expected; however, boredom is positively correlated with performance when exhibited during scaffolded tutoring. A similar pattern is unexpectedly seen for confusion. Engaged concentration and, surprisingly, frustration are both associated with positive learning outcomes. In a second analysis, we build a unified model that predicts student standardized examination scores from a combination of student affect, disengaged behaviour, and performance within the learning system. This model achieves high overall correlation to standardized exam score, showing that these types of features can effectively infer longer-term learning outcomes.

Intelligent Tutoring Systems usually take into account only the cog-nitive aspects of the student: they may suggest the right actions to perform, cor-rect mistakes, and provide explanations. However, besides cognition, educa-tional researchers increasingly recognize the importance of factors such as self-confidence and interest that contribute to learner intrinsic motivation. We be-lieve that the student affective goals can be taken into account by implementing a model of politeness into the tutoring system. This paper aims at providing an overall account of politeness in tutoring interactions (in particular, natural lan-guage dialogs), and describes the way in which politeness has been imple-mented in an intelligent tutoring system based on an animated pedagogical agent. The work is part of a larger project building a socially intelligent peda-gogical agent able to monitor learner performance and provide socially sensi-tive coaching and feedback at appropriate times. The project builds on the ex-perience gained in realizing several other pedagogical agents.

Online learning with Intelligent Tutoring System (ITS) is becoming very popular where the system models the student's learning behavior and presents to the student the learning material (content, questions‐answers, assignments) accordingly. In today's distributed computing environment, the tutoring system can take advantage of networking to utilize the model for a student for students from other similar groups. In the present paper we present a methodology where using Case Based Reasoning (CBR), ITS provides student modeling for online learning in a distributed environment with the help of agents.Student modeling is a key component in any Intelligent Tutoring System (ITS). In today's distributed computing environment, the tutoring system can take advantage of networking to utilize the model for a student for students from other similar group. In the present paper we present a methodology where using Case Based Reasoning (CBR), the ITS provides student modeling for online learning in a distributed environment with the help of agents.The Chapter describes the approach, the architecture, and the agent characteristics for the student modeling in the ITS. This concept can be deployed to develop ITS where the tutor can author and the students can learn locally whereas the ITS can model the students' learning globally in a distributed environment. The advantage of such an approach is that both the learning material (domain knowledge) and students' model can be globally distributed thus enhancing the efficiency of ITS with reducing the bandwidth requirement and complexity of the system.