Automatic evaluation of students' answers using syntactically enhanced LSA

Implementing Latent Semantic Analysis in Learning Environments with Conversational Agents and Tutorial Dialog Arthur C. Graesser (a-graesser@memphis.edu) Department of Psychology, 202 Psychology Building, University of Memphis, Memphis, TN 38152 USA Xiangen Hu (xhu@memphis.edu) Department of Psychology, 202 Psychology Building, University of Memphis, Memphis, TN 38152 USA Brent A. Olde (baolde@memphis.edu) Department of Psychology, 202 Psychology Building, University of Memphis, Memphis, TN 38152 USA Matthew Ventura (mventura45@hotmail.com) Department of Psychology, 202 Psychology Building, University of Memphis, Memphis, TN 38152 USA Andrew Olney (aolney@hotmail.com) Department of Psychology, 202 Psychology Building, University of Memphis, Memphis, TN 38152 USA Max Louwerse (mlouwers@memphis.edu) Department of Psychology, 202 Psychology Building, University of Memphis, Memphis, TN 38152 USA Donald R. Franceschetti (dfrncsch@memphis.edu) Department of Physics, University of Memphis, CAMPUS BOX 523390, Memphis, TN 38152 USA Natalie Person (person@rhodes.edu) Department of Psychology, Rhodes College, 2000 N. Parkway, Memphis, TN 38112 USA We have been developing learning environments with animated conversational agents. The agents manage a mixed-initiative dialog between the learner and the computer system either by a direct conversational interaction or by serving as a navigational guide on a web site. Two of the systems simulate human tutors by (a) presenting difficult questions that require deep reasoning, (b) attempting to comprehend the learner’s typed input, (c) formulating dialog acts that are sensitive to the learner’s contributions, and (d) speaking to the student with the animated agent. AutoTutor teaches computer literacy whereas Why/AutoTutor teaches conceptual physics (Graesser, VanLehn, Rose, Jordan, & Harter, 2001). The Human Use Regulatory Affairs Advisor (HURAA) teaches officers in the military about the ethical use of human subjects on a web site with a search facility that accesses documents through questions posed in natural language. All three systems have used Latent Semantic Analysis (LSA) as its primary representation of world knowledge. LSA is a statistical technique that compresses a large corpus texts into a space of 100-500 dimensions (Landauer, Foltz, & Laham, 1998). The K-dimensional space is used when evaluating the similarity between any two bags of words, with values ranging from 0 to 1. From the standpoint of AutoTutor and Why/AutoTutor, one bag of words is the set of assertions that a student expresses within a dialog turn; the other bag of words is the content of the curriculum script for a particular topic. From the standpoint of HURAA, one bag of words is the learner’s query in natural language and the other is a paragraph in the document space. LSA has generally been successful in evaluating the quality of student explanations, in evaluating the quality of student assertions in tutorial dialog, and in the retrieval of documents from natural language queries. Successes and failures of LSA are identified in these three learning environments. Acknowledgements This research was supported by the Office of Naval Research (N61339-01-C1006), the Institute for Defense Analyses (AK-2-1801), the National Science Foundation (REC 0106965), and the DoD Multidisciplinary University Research Initiative (MURI) administered by ONR under grant N00014-00-1-0600. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of ONR, IDA, DoD or NSF. References Graesser, A.C., VanLehn, K., Rose, C., Jordan, P., & Harter, D. (2001). Intelligent tutoring systems with conversational dialogue. AI Magazine, 22, 39-51. Landauer, T.K., Foltz, P.W., Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259-284.

We present in this paper an approach to assessing student paraphrases in the intelligent tutoring system iSTART. The approach is based on measuring the semantic similarity between a student paraphrase and a reference text, called the textbase. The semantic similarity is estimated using knowledge-based word relatedness measures. The relatedness measures rely on knowledge encoded in Word-Net, a lexical database of English. We also experiment with weighting words based on their importance. The word importance information was derived from an analysis of word distributions in 2,225,726 documents from Wikipedia. Performance is reported for 12 different models which resulted from combining 3 different relatedness measures, 2 word sense disambiguation methods, and 2 word-weighting schemes. Furthermore, comparisons are made to other approaches such as Latent Semantic Analysis and the Entailer.

Cyberlearning is challenged by the lack of personal and assessment-driven learning, and students are often puzzled by the lack of instructor guidance and feedback, the huge volume and diversity of the learning materials, and thelack of the ability to zoom in from the general concepts to the more specific ones, or the opposite. Intelligent tutoring systems are needed to improve the cyberlearning quality. One of the major difficulties is knowledge representation. The current industry standard is to use Web Ontology Language (OWL) forrepresenting knowledge structure. But OWL only supports one first-class relation, is-a, between the concepts, and different knowledge areas usually need different custom relations to describe the relations among the concepts. For example part-of and time dependency are important relations torepresent most engineering knowledge bodies. OWL has to use object properties to emulate such custom relations, leading to awkward knowledge representation hard for domain experts to code, validate and use such knowledge bases. This research uses Pace University's extension to OWL, named Knowledge Graph (KG), to support knowledge representation with custom relations. The instructors can use Pace University extended Protege IDE to declare and apply custom relations in a single document. The instructor teaching experience is also coded in the KG to better support custom learning order by students with different backgrounds. The prototype of a knowledge-driven tutoring system was designed and implemented to illustrate how the KG supports integrated assessments, using assessment results to custom student learning order or material, and let the students freely navigate in the knowledge space from general to specific or the opposite, and following various custom relations. A web technology tutorial is used to validate the design and effectiveness of this approach.

One of the 9-11 commission's recommendations on a different way of organizing intelligence activities of the United States was to unify the effort in information sharing across the Intelligence Community. Challenges include the need to deal with information that is geographically distributed and held in compartmented repositories having restricted access. A demonstrated ‘need to know' is required before the data can be shared, and that assumes that one knows it exists and where to ask for it. Each intelligence agency has its own data security practices that restrict out-right sharing of the data within the Intelligence Community at large. Commercial off-the-shelf solutions exist for securely sharing highly sensitive ‘need to know' data between cooperating agencies via digitally cosigned contracts, even using the Internet. Once data has been exchanged it still needs to be protected with the same ‘need to know' restrictions by the receiving agency. In the post 9-11 world the question is how do we securely share information in a manner that protects the data but enables its value to be discovered by others having a ‘need to know'. This poster session proposes a demonstration of a secure data sharing technique that allows sharing of sensitive documents to influence collections of documents available to first responders and others; without exposing the contents of the sensitive documents. The underlying technology that makes this demonstration possible is latent semantic indexing (LSI). LSI is a robust dimensionality-reduction technique for the processing of textual data. The technique can be applied to collections of documents independent of subject matter or language. Given a collection of documents, LSI indexing can be employed to create a vector space in which both the documents and their constituent terms can be represented. Sensitive documents are employed as part of the training data. The relationship information implicit in the sensitive documents is smoothly blended with the relationship information implicit in the non-sensitive documents. This has the effect of slightly perturbing the representation vectors for tens to hundreds of thousands of term vectors in the resulting LSI space. The sensitive documents can be completely protected in this process – there is no way, even in principle, for the text of sensitive documents to be reconstructed. However, the subtle changes in the term representation vectors can yield dramatic improvements in analysis activities carried out using the non-sensitive documents. Non-sensitive documents are made available to first responders and others who need to be made aware of threats and vulnerabilities. The influence of sensitive documents on the non-sensitive documents Background Use of Sensitive Information to Aid in Analysis of Non-sensitive Data 653 will produce a re-ordering of non-sensitive documents as the result of a first responder query. For example, first responder queries of similar semantic context to the sensitive training documents will return non-sensitive documents highly influenced by the sensitive training documents. In a real implementation of the system, first responders would be able to infer the implications of the sensitive training documents through the non-sensitive searchable documents. The richness of both document sets (sensitive and non-sensitive) has implications for performance of the system. This technique directly addresses the recent GAO report on recommendations for improving the sharing of information between the federal government and the private sector on incidents, threats, and vulnerabilities.

Text representation, which is a fundamental and necessary process for text-based intelligent information processing, includes the tasks of determining the index terms for documents and producing the numeric vectors corresponding to the documents. In this paper, multi-word, which is regarded as containing more contextual semantics than individual word and possessing the favorable statistical characteristics, is proposed as an alternative index terms in vector space model for text representation with theoretical support. We investigate the traditional indexing methods as TF*IDF (term frequency inverse document frequency) and LSI (latent semantic indexing) for comparative study. The performances of TF*IDF, LSI and multi-word are examined on the tasks of text classification, which includes information retrieval (IR) and text categorization (TC), in Chinese and English document collection respectively. We also attempt to tune the rescaling factor of LSI and observe its effectiveness in text classification. The experimental results demonstrate that TF*IDF and multi-word are comparable when they are used for IR and TC and LSI is the poorest one of them. Moreover, the rescaling factor of LSI has an insignificant influence on its effectiveness on text classification for both Chinese and English text classification.

Latent semantic analysis (LSA) is a model of knowledge representation for words. It works by applying dimension reduction to local co-occurrence data from a large collection of documents after performing singular value decomposition on it. When the reduction is applied, the system forms condensed representations for the words that incorporate higher order associations. The higher order associations are primarily responsible for any semantic similarity between words in LSA. In this article, a memory model is described that creates semantic representations for words that are similar in form to those created by LSA. However, instead of applying dimension reduction, the model builds the representations by using a retrieval mechanism from a well-known account of episodic memory.

A text classification model based on Latent Semantic Analysis and Improved Hyper-sphere Support Vector Machine, is proposed in order to improve the accuracy and efficiency of text classification. Latent Semantic Analysis is used in this model for feature extraction, eliminating the text representation errors caused by synonyms and polysemes, and reducing the dimension of text vector. At the same time, a new decision-making approach based on concentration, is designed for improving the classification of texts in overlapping regions. Experimental results show that the model will give good classification results when the number of classes is small. And the improved algorithm is effective and feasible.

The recent emphasis on intelligent systems has increased the focus on categorization techniques as it is an important step in information retrieval and natural language processing. The text categorization is largely achieved using machine learning techniques. In most of the approaches one-hot encoding or pre-trained word embedding such as word2vec or glove vectors are used. This study explores the feature vectors based encoding using Latent Semantic Analysis (LSA) technique along with the Convolutional Neural Network (CNN) being used as a classifier. It was found that applying LSA followed by CNN for text classification offers better accuracy than the conventional methods of CNN with other approaches. This research, thus, highlights the importance of Latent Semantic Analysis technique coupled with convolutional neural networks for text classification.

Dimensionality reduction is an essential task for many large-scale information processing problems such as classifying document sets, searching over Web data sets, etc. It can be used to improve both the efficiency and the effectiveness of classifiers. In this paper, a comparative study is conducted of five Dimension Reduction Techniques in the context of the Arabic text classification problem using an in house Arabic dataset. We evaluated and compared Stemming, Light-Stemming, Document Frequency (DF), TFIDF and Latent Semantic Indexing (LSI)methods to reduce the feature space into an input space of much lower dimension for the neural network classifier. The results showed that the proposed model was able to achieve high categorization effectiveness as measured by Macro-Average F1 measure. Experiments on Arabic datasets indicate that the DF, TFIDF and LSI techniques are favorable in terms of its effectiveness and efficiency when compared with the two other methods.

Text often expresses the writer's emotional state or evokes emotions in the reader. The nature of emotional phenomena like reading and writing can be interpreted in different ways and represented with different computational models. Affective computing (AC) researchers often use a categorical model in which text data are associated with emotional labels. We introduce a new way of using normative databases as a way of processing text with a dimensional model and compare it with different categorical approaches. The approach is evaluated using four data sets of texts reflecting different emotional phenomena. An emotional thesaurus and a bag‐of‐words model are used to generate vectors for each pseudo‐document, then for the categorical models three dimensionality reduction techniques are evaluated: Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (PLSA), and Non‐negative Matrix Factorization (NMF). For the dimensional model a normative database is used to produce three‐dimensional vectors (valence, arousal, dominance) for each pseudo‐document. This three‐dimensional model can be used to generate psychologically driven visualizations. Both models can be used for affect detection based on distances amongst categories and pseudo‐documents. Experiments show that the categorical model using NMF and the dimensional model tend to perform best.

This research addresses the widening gap between research in intelligent tutoring systems and practical use of this technology by the educational community. In order to insure that intelligent tutoring systems (ITSs) are effective, teachers must be involved in their design and evaluation. We have followed a user participatory design process to build a set of ITS knowledge acquisition tools tailored for usability by teachers. The system facilitates rapid prototyping and testing of curriculum and multiple tutoring strategies. Teachers use the system to create, modify, and test the system's domain content and tutoring strategy knowledge bases. The design includes novel methodologies for strategy representation and overlay student modeling, and incorporates considerations from instructional design theory. Tools have been designed to provide the user with visual models of the concepts and structures of the underlying framework. In close collaboration with a veteran high school teacher, we have used the interface to design a tutor for statics (part of a high school physics course). In this paper we describe the system (called KAFITS), report on our experience involving educators in ITS development, discuss issues of ITS knowledge representation and acquisition, and compare the system with related research in generic tutoring systems and knowledge acquisition.

The standard application of Latent Semantic Indexing (LSI), a well-known technique for information retrieval, requires the computation of a partial Singular Value Decomposition (SVD) of the term-document matrix. This computation becomes infeasible for large document collections, since it is very demanding both in terms of arithmetic operations and in memory requirements. This paper discusses two divide and conquer strategies, with the goal of alleviating these difficulties. Both strategies divide the document collection in subsets, perform relevance analysis on each subset, and conquer the analysis results to form the query response. Since each sub-problem resulting from the recursive division process has a smaller size, the processing of large scale document collections requires much fewer resources. In addition, the computation is highly parallel and can be easily adapted to a parallel computing environment. To reduce the computational cost, we perform the analysis on the subsets by using the Lanczos vectors instead of singular vectors as in the standard LSI method. This technique is far more efficient than the computation of the truncated SVD, while its accuracy is comparable. Experimental results confirm that the proposed divide and conquer strategies are effective for information retrieval problems.

Automatic summarization is a topic of common concern in computational linguistics and information science, since a computer system of text summarization is considered to be an effective means of processing information resources. A method of text summarization based on latent semantic indexing (LSI), which uses semantic indexing to calculate the sentence similarity, is proposed in this article. It improves the accuracy of sentence similarity calculations and subject delineation, and helps the abstracts generated to cover the documents comprehensively as well as reducing redundancies. The effectiveness of the method is proved by the experimental results. Compared with the traditional keyword-based vector space model method of automatic text summarization, the quality of the abstracts generated was significantly improved.

In this paper we present a hybrid text categorization method based on Rough Sets theory. A central problem in good text classification for information filtering and retrieval (IF/IR) is the high dimensionality of the data. It may contain many unnecessary and irrelevant features. To cope with this problem, we propose a hybrid technique using Latent Semantic Indexing (LSI) and Rough Sets theory (RS) to alleviate this situation. Given corpora of documents and a training set of examples of classified documents, the technique locates a minimal set of co-ordinate keywords to distinguish between classes of documents, reducing the dimensionality of the keyword vectors. This simplifies the creation of knowledge-based IF/IR systems, speeds up their operation, and allows easy editing of the rule bases employed. Besides, we generate several knowledge base instead of one knowledge base for the classification of new object, hoping that the combination of answers of the multiple knowledge bases result in better performance. Multiple knowledge bases can be formulated precisely and in a unified way within the framework of RS. This paper describes the proposed technique, discusses the integration of a keyword acquisition algorithm, Latent Semantic indexing (LSI) with Rough Set-based rule generate algorithm, and provides experimental results. The test results show the hybrid method is better than the previous rough set-based approach.

High-dimensional semantic space accounts of priming