Experimenting with Latent Semantic Analysis and Latent Dirichlet Allocation on Automated Essay Grading

This paper explained a research that was conducted to find the best methodology for automatic essay grading. The Winnowing method was used in this paper with specific application for Japanese language exam. With Japanese language input, romanization was implemented to change the input that were in hiragana, katakana, or kanji into romaji. After the system was developed, it still needed to find the best parameters with the highest accuracy or closer even similar to the results from human rater. From testing the short essay Japanese language exam answers with 100% of score from human rater, several testing show that the best parameter for this system is with n=2, w=2, and p=2 and the average accuracy of total score gathered for all students is 86.86%. Since the average accuracy is above 80%, it could be concluded that winnowing algorithm is suitable for Japanese language examination term.

Topic modeling is the statistical model for discovering hidden topics or keywords in a collection of documents. Topic modeling is also considered a probabilistic model for learning, analyzing, and discovering topics from the document collection. The most popular techniques for topic modeling are latent semantic analysis (LSA), probabilistic latent semantic analysis (pLSA), latent Dirichlet allocation (LDA), and the recent deep learning-based lda2vec. LDA is most commonly used in extractive multi-document summarization to determine whether the extracted sentence reflects the concept of the input document. In this paper, we will try to explore various multi-document summarization techniques that use LDA as a topic modeling method for improving final summary coverage and to reduce redundancy. Finally, we compared LDA and LSA using the Genism toolkit, and our experiment results show that LDA outperforms LSA if we increase the number of features considered for sentence selection.KeywordsTopic modelingText summarizationLDALSA

Funding agencies such as the U.S. National Science Foundation (NSF), U.S. National Institutes of Health (NIH), and the Transportation Research Board (TRB) of The National Academies make their online grant databases publicly available which document a variety of information on grants that have been funded over the past few decades. In this paper, based on a quantitative analysis of the TRB’s Research In Progress (RIP) online database, we explore the feasibility of automatically estimating the appropriate funding level, given the textual description of a transportation research project. We use statistical Text Mining (TM) and Machine Learning (ML) technologies to build this model using the 14,000 or more records of the TRB’s RIP research grants big data. Several Natural Language Processing (NLP) based text representation models such as the Latent Dirichlet Allocation (LDA), Latent Semantic Indexing (LSI) and the Doc2Vec Machine Learning (ML) approach are used to vectorize the project descriptions and generate semantic vectors. Each of these representations is then used to train supervised regression models such as Random Forest (RF) regression. Out of the three latent feature generation models, we found LDA gives the least Mean Absolute Error (MAE) using 300 feature dimensions and RF regression model. However, based on the correlation coefficients, it was found that it is not very feasible to accurately predict the funding level directly from the unstructured project abstract, given the large variations in source agencies, subject areas, and funding levels. By using separate prediction models for different types of funding agencies, funding levels were better correlated with the project abstract.

Standardized testing such as the SAT often requires students to write essays and hires a large number of graders to evaluate these essays which can be time and cost consuming.Using natural language processing tools such as Global Vectors for word representation (GloVe), and various types of neural networks designed for picture classification, we developed an automatic grading system that is more time-and cost-efficient compared to human graders.We applied our application to a set of manually graded essays provided by a previous competition on Kaggle in 2012 on automated essay grading and conducted a qualitative evaluation of the approach.The result shows that the program is able to correctly score most of the essay and give an evaluation close to that of a human grader on the rest.The system proves itself to be effective in evaluating various essay prompts and capable of real-life application such as assisting another grader or even used as a standalone grader.

A questionnaire text mining model is built based on latent semantic analysis (LSA). Take open questionnaire as the research object and improve Latent Dirichlet Allocation (LDA) model according to the text characteristics, a BM25-LDA model is built. By comparing the LDA model and BM25-LDA model on the open questionnaire text, we can verify that the performance of BM25-LDA mode is better than that of original LDA model.

This paper aims to investigate the effectiveness of automatic grading systems, with a focus on their uses within Computer Science. Automatic grading systems have seen a rise in popularity in recent years with publications concerning automatic grading systems usually linked to a specific system. This paper will discuss the factors that need to be considered when using automatic grading, regardless of which system is being used, and will make recommendations for each factor. This discussion is based on the authors' experience of using an automatic grading system in a CS1 environment. From the research conducted, many elements should be considered when using these systems. These include how the code will be tested, the need for plagiarism checks and how marks are awarded. The findings of this study suggest there is a lack of defined standards when using these systems. This analysis of the considerations provides valuable insight into how these systems should be used and what the standards should be built on.

The intelligent task of semantically assigning a paper to a reviewer with respect to his knowledge domain remains a challenging task in academic conferences. From literature, a number of automated reviewer assignment systems have been presented which are based on distributional semantic models such as Term Frequency-Inverse Document Frequency (TF-IDF), Latent Semantic Indexing (LSI) and Latent Dirichlet Allocation (LDA) have been used to capture semantics. Thus, this study presents the comparative study of the three models based on their derived suitability scores between a paper meant for review and a reviewer’s representation papers. From the experimental results obtained, it shows that TF-IDF outperformed the accuracy level of the other two models by a substantial margin.

We present Topic View, an application for visually comparing and exploring multiple models of text corpora. Topic View uses multiple linked views to visually analyze both the conceptual content and the document relationships in models generated using different algorithms. To illustrate Topic View, we apply it to models created using two standard approaches: Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA). Conceptual content is compared through the combination of (i) a bipartite graph matching LSA concepts with LDA topics based on the cosine similarities of model factors and (ii) a table containing the terms for each LSA concept and LDA topic listed in decreasing order of importance. Document relationships are examined through the combination of (i) side-by-side document similarity graphs, (ii) a table listing the weights for each document's contribution to each concept/topic, and (iii) a full text reader for documents selected in either of the graphs or the table. We demonstrate the utility of Topic View's visual approach to model assessment by comparing LSA and LDA models of two example corpora.

To support program comprehension, software artifacts can be labeled--for example within software visualization tools--with a set of representative words, hereby referred to as labels. Such labels can be obtained using various approaches, including Information Retrieval (IR) methods or other simple heuristics. They provide a bird-eye's view of the source code, allowing developers to look over software components fast and make more informed decisions on which parts of the source code they need to analyze in detail. However, few empirical studies have been conducted to verify whether the extracted labels make sense to software developers. This paper investigates (i) to what extent various IR techniques and other simple heuristics overlap with (and differ from) labeling performed by humans; (ii) what kinds of source code terms do humans use when labeling software artifacts; and (iii) what factors--in particular what characteristics of the artifacts to be labeled--influence the performance of automatic labeling techniques. We conducted two experiments in which we asked a group of students (38 in total) to label 20 classes from two Java software systems, JHotDraw and eXVantage. Then, we analyzed to what extent the words identified with an automated technique--including Vector Space Models, Latent Semantic Indexing (LSI), latent Dirichlet allocation (LDA), as well as customized heuristics extracting words from specific source code elements--overlap with those identified by humans. Results indicate that, in most cases, simpler automatic labeling techniques--based on the use of words extracted from class and method names as well as from class comments--better reflect human-based labeling. Indeed, clustering-based approaches (LSI and LDA) are more worthwhile to be used for source code artifacts having a high verbosity, as well as for artifacts requiring more effort to be manually labeled. The obtained results help to define guidelines on how to build effective automatic labeling techniques, and provide some insights on the actual usefulness of automatic labeling techniques during program comprehension tasks.

Stack Overflow has undoubtedly become a big trend among the programmers and researchers. The problems or doubts faced by any programmer are put on this forum making it available to many knowledgeable people and hence giving out possibly correct solutions. This paper focuses on Tag based Answer Recommendation System on Stack Overflow. This model follows Topic Modeling methodology wherein two techniques were attempted, i.e., LSA(Latent semantic analysis) and LDA(Latent Dirichlet Allocation). These methods deal with various User Queries to find out the most similar questions from the database and hence, deliver the most valid answers related to the question as output. The results from both the Topic Modeling techniques were analyzed with the help of Cosine Similarity as metrics and hence, we found out that LDA performed better than LSA, thereby giving better and more accurate results.

Multiple-choice (MC) forms have become the dominant, fast, reliable and the easiest tools of assessment in learning environments such as schools. Present automatic grading systems for such forms, though fast and accurate are either rated as expensive due to materials and equipments they require for their operation or can only operate on a particular set of forms or papers. Part of the main reasons this is so is due to the inability of such systems to generate a template from an unknown form or another form produced by a similar system. As a novel step to creating a more dynamic and efficient automatic MC test grading system, we present in this paper a template generation algorithm capable of generating a template from any type of bubble MC test form. The empty sheet is first scanned into digital form and then normalized to a given size. To consistently remove non-relevant objects such as lines from the image, a unique combination of Hough transform (HT) and Region of interest (ROI) is proposed. Finally, the true answer regions are segmented using region processing by calculating their sizes and positions, this result in the final template. The preliminary results showed that the algorithm is dynamic with overall high precision accuracy even at low resolutions.

Reliable and pedagogically sound automated feedback and grading systems are highly coveted by educators. Automatic grading systems are useful for ensuring equity of grading of student submissions to assignments and providing timely feedback on the work. Many of these systems test submissions to assignments based on test cases and the outputs that they achieve, while others use unit tests to check the submissions. The approach presented in this paper checks submissions based on test cases but also analyses what the students actually wrote in their code. Assignment questions are constructed based around the concepts that the student are currently learning in lectures, and the patterns searched for in their submissions are based on these concepts. In this paper we show how to implement this approach effectively. We analyse the use of an automatic grading system within first year Computer Science programming modules and show that the system is straightforward to use and suited for novice programmers, while providing automatic grading and feedback. Evaluation received from students, demonstrators and lecturers show the system is extremely beneficial. The evaluation shows that such systems allow demonstrators more time to assist students during labs. Lecturers can also provide instant feedback to students while keeping track of their progress and identifying where the gaps in students’ knowledge are.

Markit© is an Automated Essay Grading (AEG) system capable of running on typical desktop PC platforms. Its performance compares favourably with human graders and with commercially available systems. A distinct advantage of Markit over existing commercial systems is that it requires only one model answer against which the student essays are compared. In this paper we report on a trial of Markit with second year law students’ essays.

Topmodpy: A Simple Python Script for Topic Modeling

People sourcing for a particular job role in any corporate venture is a painstaking, progress-impeding task, especially given the changing job market trends, quality of candidate profile and the sheer number of applicants. This paper elucidates an improvised topic modelling approach, where the semantic analysis stage is prepended with text summarization, for the procedure of people sourcing for job roles using topic modelling and machine learning techniques. The distinction between choosing the perfect candidate for the job and choosing a good candidate who is adept in certain domains but not relevant for the job and getting the candidate up to speed by providing on job training, and the downtime involved in the process is certainly a deciding factor in hiring a potential employee. The following paper aims to alleviate this issue by describing the algorithm which identifies the most suitable candidate in the applicants’ pool through a novel, robust approach which uses topic modelling techniques like Latent Semantic Indexing (LSA) and Latent Dirichlet Allocation (LDA). It is empirically found that the precision of profile matching task was enhanced by using text summarization (through TextRank model) for both LSA and LDA by 21.4% and 50% respectively, and LSA outperforming LDA with regard to precision, in both with and without text summarization cases by 23.8% and 51.57%, respectively.