Automated Essay Scoring System using Multi-Model Machine Learning

The demand of scoring natural language responses has created a need for new computational tools that can be applied to automatically grade student essays. Systems for automatic essay assessment have been commercially available since 1990's. However, the progress in the field was obstructed by a lack of qualitative information regarding the effectiveness of such systems. Most of the research in automatic essay grading has been associated with English writing due to its widespread use and the availability of more learner collection and language processing software for the language. In addition, there is large number of commercial software for grading programming assignments automatically. In this work, we investigate document semantic similarity based on Latent Semantic Analysis (LSA) and on Latent Dirichlet Allocation (LDA). We use an open-source Python software, Gensim, to develop and implement an essay grading system able to compare an essay to an answer-key and assign to it a grade based on semantic similarity between the two. We test our tool on variable-size essays and conduct experimental tests to compare the results obtained from human grader (professor) and those obtained from the automatic grading system. Results show high correlation between the professor grades and the grades assigned by both modeling techniques. However, LSA-based modeling showed more promising results than the LDA-based method.

Applications in the World Wide Web aggregate vast amounts of information from different data sources. The aggregation process is often implemented with Extract, Transform and Load (ETL) processes. Usually ETL processes require information for aggregation available in structured formats, e. g. XML or JSON. In many cases the information is provided in natural language text which makes the application of ETL processes impractical. Due to the fact that information is provided in natural language, Information Extraction (IE) systems have been evolved. They make use of Natural Language Processing (NLP) tools to derive meaning from natural language text. State-of-the-art NLP tools apply Machine Learning methods. These NLP tools perform on newspapers with good quality, but they drop accuracy in other domains. However, to improve the quality for IE systems in specific domains often NLP tools are trained on domain specific text which is a time consuming process. This paper introduces an approach using a Continuous Integration pipeline for organizing and monitoring the annotation process on domain specific corpora.

We present SimpleApprenant, a platform aiming to improve French L2 learners’ knowledge of Multi Word Expressions (MWEs). SimpleApprenant integrates an MWE database annotated with the Common European Framework of Reference for languages (CEFR) level and several Natural Language Processing (NLP) tools: a spelling checker, a parser, and a set of transformation rules. NLP tools and resources are used to build training and writing exercises to improve MWE knowledge and writing skills of French L2 learners. We present the user scenarios, the platform’s architecture, as well as the preliminary evaluation of its NLP tools.

Objective: The prevalence of moderate to severe asymptomatic carotid stenosis (ie, atherosclerotic narrowing of the extracranial carotid arteries) is generally approximately 6% and 2%, respectively. Most prior studies of carotid stenosis risk factors have been small. This study describes the development and validation of a natural language processing (NLP) tool to identify carotid stenosis and uses it to identify significant risk factors, presence, and severity of carotid stenosis. Methods: We created an NLP tool to extract the ratio of peak systolic velocity of the internal carotid artery to the common carotid artery (ICA/CCA ratio) in veterans receiving carotid duplex ultrasound examinations in the Veteran's Health Administration from 2001 to 2020. Among those who had at least one valid ICA/CCA ratio, we identified carotid stenosis severity (<50%, 50%-69%, ≥70%) based on the ICA/CCA ratio (<2, ≥2 to <4, ≥4) and assessed the association between presence and severity of carotid stenosis and clinical and demographic characteristics, including age, sex, self-identified race and ethnicity, smoking status, body mass index, systolic and diastolic blood pressures, indicator variables for pre-existing hypertension, coronary heart disease, and type 2 diabetes, and selected laboratory measures (ie, hemoglobin A1c, low-density lipoprotein cholesterol, high-density lipoprotein cholesterol, triglyceride, and creatinine). Results: The harmonic F1 score of the NLP tool was 0.907 for the right value, 0.882 for the left value, and 0.920 for the maximum value. Among the 290,517 veterans in the cohort, the median age was 68.2 years. Black patients had 16% decreased risk of more severe carotid stenosis (odds ratio: 0.84, 95% confidence interval: 0.81-0.87, P < .001). All patient-level risk factors except high-density lipoprotein cholesterol were significantly associated with carotid stenosis severity. Conclusions: The NLP tool performed well, and the study performed with our NLP-created cohort largely validates the risk factors identified by previous smaller studies, demonstrating the utility of big data and NLP in carotid stenosis research.

A tweet is an authentic use of Natural Language where the user has to deliver the message in 140 characters or less. According to previous researchers, this restriction increases the possible ambiguity of a tweet making it difficult for traditional Natural Language Processing (NLP) tools to analyze it. This research enhances the machine learning based Stanford CoreNLP Part-of-Speech (POS) tagger with the Twitter model to extract essential keywords from a tweet. The system was enhanced using two rule-based parsers and a corpus. The research was conducted using tweets of customer service requests sent to a telecommunication company. A domain specific corpus was compiled after analyzing the tweets. The POS tagger extracted the keywords while the parsers removed any possible noise and extracted any other keywords missed by the POS tagger. The evaluation of the system was done using the Turing Test. The proposed system was tested and compared against the Stanford CoreNLP. The testing was conducted using 6 test cases, each consisting of a human keyword generator and a supervisor. In order to ensure the impartiality and intellectual diversity, the response generators and supervisors were representatives of 6 different fields. As a result of the enhancements, the Turing Test score of the system increased from 50.00% to 83.33%. The accuracy of the system could be further improved by using a complete domain specific corpus. Since the approach used theoretical linguistic features of a sentence, the same method could be employed for other NLP tools.

The US Food and Drug Administration Adverse Event Reporting System (FAERS) contains over 24 million individual case safety reports (ICSRs). In this research project, we evaluated a natural language processing (NLP) tool’s ability to extract four demographic variables (gender, weight, ethnicity, race) from ICSR narratives. Specificity of the NLP algorithm was over 94% for all demographics, while sensitivity varied between the demographics: 98.6% (gender), 45.5% (weight), 100% (ethnicity), and 85.3% (race). Among ICSRs missing weight, ethnicity, and race in the structured field, few cases had this information in the narrative (&amp;gt;95% missing); consequently, the positive predictive value (PPV) for these three demographics had wide 95% confidence intervals. After NLP implementation, the total number of ICSRs missing gender was reduced by 33% (i.e., NLP identified 472 thousand reports having a gender value in the narrative that was not in the structured field), while the total number of ICSRs missing weight, ethnicity, or race was reduced by less than 4%. This study demonstrated that the implementation of an NLP tool can provide meaningful improvements in the availability of gender information for pharmacovigilance activities conducted with FAERS data. In contrast, NLP tools targeting the extraction of weight, ethnicity, or race from free-text fields have minimal impact largely because the information was infrequently provided by the reporter. Further gains in completeness of these fields must originate from increases in provision of demographic information from the reporter rather than informatic solutions.

Evaluation of Natural Language Processing (NLP) systems to annotate drug product labeling with MedDRA terminology

The aim of this study was to test the ability of a commercially available natural language processing (NLP) tool to accurately extract examination quality-related and large polyp information from colonoscopy reports with varying report formats. Colonoscopy quality reporting often requires manual data abstraction. NLP is another option for extracting information; however, limited data exist on its ability to accurately extract examination quality and polyp findings from unstructured text in colonoscopy reports with different reporting formats. NLP strategies were developed using 500 colonoscopy reports from Kaiser Permanente Northern California and then tested using 300 separate colonoscopy reports that underwent manual chart review. Using findings from manual review as the reference standard, we evaluated the NLP tool's sensitivity, specificity, positive predictive value (PPV), and accuracy for identifying colonoscopy examination indication, cecal intubation, bowel preparation adequacy, and polyps ≥10 mm. The NLP tool was highly accurate in identifying examination quality-related variables from colonoscopy reports. Compared with manual review, sensitivity for screening indication was 100% (95% confidence interval: 95.3%-100%), PPV was 90.6% (82.3%-95.8%), and accuracy was 98.2% (97.0%-99.4%). For cecal intubation, sensitivity was 99.6% (98.0%-100%), PPV was 100% (98.5%-100%), and accuracy was 99.8% (99.5%-100%). For bowel preparation adequacy, sensitivity was 100% (98.5%-100%), PPV was 100% (98.5%-100%), and accuracy was 100% (100%-100%). For polyp(s) ≥10 mm, sensitivity was 90.5% (69.6%-98.8%), PPV was 100% (82.4%-100%), and accuracy was 95.2% (88.8%-100%). NLP yielded a high degree of accuracy for identifying examination quality-related and large polyp information from diverse types of colonoscopy reports.

Purpose This study explores the potential of Natural Language Processing (NLP) tools, specifically ChatGPT-4o and Data Analyst, in supporting health commissions with disability assessments. Methods Nine realistic patient scenarios were created to reflect typical cases in disability evaluations, encompassing conditions like stroke, cerebral palsy, traumatic injuries, and hereditary disorders, each involving varied motor and functional impairments. These scenarios were input into ChatGPT-4o and Data Analyst. Their outputs were evaluated using a 5-point Likert scale on alignment with expert guidelines, completeness, and lack of false information. Interrater reliability was established before scoring. Results Both models produced generally high-quality narrative reports, ChatGPT-4o was rated “very good” and Data Analyst “good,” with no statistically significant difference in overall scores. However, ChatGPT-4o failed to calculate correct disability percentages in 55.6%, and Data Analyst failed in 88.9% of scenarios. Conclusions While NLP tools can assist in generating structured disability reports, they currently lack the precision needed for calculating disability percentages reliably. Expert oversight remains essential for decision-making in disability assessments.

Markit© is an Automated Essay Grading (AEG) system capable of running on typical desktop PC platforms. Its performance compares favourably with human graders and with commercially available systems. A distinct advantage of Markit over existing commercial systems is that it requires only one model answer against which the student essays are compared. In this paper we report on a trial of Markit with second year law students’ essays.

Biographical writing is one of the earliest and most extensive forms of Arabic literature. Some scholars tend to assume that classical Arabic biographies, widely known as Tarāǧim, arose in conjunction with the study of the reliability of the Hadith transmitters (the reciters of the Prophet Mohammad's sayings) which lead to a proliferation of biographical material collected and used to assess the transmitter's trustworthiness . However, a scrutiny of the well-known classical Arabic biographical dictionaries such as Siyaru 'A`lāmi an-Nubalā' `The Lives of the Noble Figures' for Adh-Dhahabī shows that they extend their entries to other classes of persons important to the development of particular fields such as Islamic jurisprudents, rulers, poets, philosophers or physicians. The main contribution of Arabic biographical dictionaries is the cumulative value of the thousands of life histories which construct a picture of the Islamic society in different eras. An Arabic biographical dictionary, therefore, is predominantly used by scholars to look up an eminent person's achievements and historical background. In this project, however, we explore Arabic biographies as a prosopography, rather than a biography in the strict sense. We introduce a novel method for a better understanding of Arabic biographical dictionaries by creating a network of relations among different persons. We utilise Natural Language Processing (NLP) tools to create a topological network from the unstructured data of 45,500 biographical entries collected from different dictionaries. We aim to illustrate how network analysis leveraged by NLP tools can provide scholars with innovative methods for discovering complex constellation of relations between prominent and non-prominent figures spanning over several eras and from different fields of knowledge. We also use graph visualisation as a means to effectively communicate and explore such complex constellations. Each network visualisation is purposefully designed to be as simple and robust as possible to offer scholars a way to move relatively fluidly between the large scale of biographical entries and to easily interpret the minute ties between persons of different walks of life. We make both our data and code publicly available for researchers to replicate the experiment. It can be found at:https://github.com/sadanyh/Relational-Network-for-Arabic-Tarajem

Historical data poses unique challenges to natural language processing (NLP) and information retrieval (IR) tools, including digitization errors, lack of annotated data, and diachronic-specific issues. However, the increasing recognition of the value in historical documents has promoted efforts to semantically enrich and optimize their analysis. This article contributes to this endeavour by enriching the Corpus de Textos Antigos through NLP tools and techniques to enhance its usability and support research. The corpus undergoes linguistic annotation, including part-of-speech tagging, lemma annotation and named entity recognition (NER). Subsequently, the article delves into the tasks of entity disambiguation and entity linking, which involve identifying and disambiguating named entities by referring to a knowledge base (KB). Addressing the challenges posed by factors such as text state, epoch and the chosen KB, the article presents insights into related work, annotation results and the linguistic interest of a medieval annotated corpus for named entities. It concludes by discussing the challenges and providing avenues for future research in this domain.

Introduction:Nowadays in Panama, there is a lot of patient information stored in textual form which cannot be manipulated to manage adequate knowledge. There are multiple resources created to represent knowledge, including specialized glossaries, ontologies, among others. The ontologies are an important part within the scope of the recovery and organization of the information and the semantic web. Also in recent works they are used in applications of natural language processing (NLP), as a knowledge base.Aim:This research was conducted with the aim of creating a methodology that allows from a text written in NL, extract the necessary elements using NLP tools and with them create a knowledge base represented by one domain ontology and extract knowledge to help medical specialists.Material and Methods:In this study we carried out a methodology that allows the extraction of knowledge of patient clinical records, general medicine and palliative care, in order to show relevant knowledge elements to specialists. The methodology was validated with a data corpus of approximately 200 patient records.Conclusion:We have created a knowledge representation methodology, combining NLP techniques and tools and the automatic instantiation of an ontology, which can serve as a software agent for other applications or used to visualize the patient’s clinical information. The study was validated using the traditional metrics of information retrieval systems precision, recall, F-measure obtaining excellent results, and can be used as a software agent or methodology for the development of information extraction software systems in the medical domain.

The fashion department at SMK Negeri 1 Warureja faces significant challenges in understanding and applying Artificial Intelligence (AI)-based Natural Language Processing (NLP) among its students and teachers. Despite the potential benefits of NLP technology for the fashion industry, the limited knowledge and skills concerning NLP tools at SMK Negeri 1 Warureja impede effective utilization. This Community Service Program (CSP) is strategically designed to address these gaps by providing targeted training on NLP tools such as ChatGPT, Chatbot, and Brand24.com within the fashion context. This CSP activity applies the technique of training an initial evaluation through a pre-test assessed participants' baseline understanding prior to the training. During the practical tutorial phase, participants received hands-on training with NLP tools, including practical guidance to ensure real-world application of these concepts. Evaluation results indicated a significant improvement in participants' understanding and skills. The post-test average of 80.47 points showed a significant improvement of 32.19 points over the pre-test average of 48.28 points.

To evaluate the diagnostic performance of a natural language processing (NLP) model in detecting incidental lung nodules (ILNs) in unstructured chest computed tomography (CT) reports. All unstructured consecutive reports of chest CT scans performed at a tertiary hospital between 2020 and 2021 were retrospectively reviewed (n = 21,542) to train the NLP tool. Internal validation was performed using reference readings by two radiologists of both CT scans and reports, using a different external cohort of 300 chest CT scans. Second, external validation was performed in a cohort of all random unstructured chest CT reports from 57 different hospitals conducted in May 2022. A review by the same thoracic radiologists was used as the gold standard. The sensitivity, specificity, and accuracy were calculated. Of 21,542 CT reports, 484 mentioned at least one ILN (mean age, 71 ± 17.6 [standard deviation] years; women, 52%) and were included in the training set. In the internal validation (n = 300), the NLP tool detected ILN with a sensitivity of 100.0% (95% CI, 97.6 to 100.0), a specificity of 95.9% (95% CI, 91.3 to 98.5), and an accuracy of 98.0% (95% CI, 95.7 to 99.3). In the external validation (n = 977), the NLP tool yielded a sensitivity of 98.4% (95% CI, 94.5 to 99.8), a specificity of 98.6% (95% CI, 97.5 to 99.3), and an accuracy of 98.6% (95% CI, 97.6 to 99.2). Twelve months after the initial reports, 8 (8.60%) patients had a final diagnosis of lung cancer, among which 2 (2.15%) would have been lost to follow-up without the NLP tool. NLP can be used to identify ILNs in unstructured reports with high accuracy, allowing a timely recall of patients and a potential diagnosis of early-stage lung cancer that might have been lost to follow-up.