Considerations when using an Automatic Grading System within Computer Science Modules

The demand of scoring natural language responses has created a need for new computational tools that can be applied to automatically grade student essays. Systems for automatic essay assessment have been commercially available since 1990's. However, the progress in the field was obstructed by a lack of qualitative information regarding the effectiveness of such systems. Most of the research in automatic essay grading has been associated with English writing due to its widespread use and the availability of more learner collection and language processing software for the language. In addition, there is large number of commercial software for grading programming assignments automatically. In this work, we investigate document semantic similarity based on Latent Semantic Analysis (LSA) and on Latent Dirichlet Allocation (LDA). We use an open-source Python software, Gensim, to develop and implement an essay grading system able to compare an essay to an answer-key and assign to it a grade based on semantic similarity between the two. We test our tool on variable-size essays and conduct experimental tests to compare the results obtained from human grader (professor) and those obtained from the automatic grading system. Results show high correlation between the professor grades and the grades assigned by both modeling techniques. However, LSA-based modeling showed more promising results than the LDA-based method.

Reliable and pedagogically sound automated feedback and grading systems are highly coveted by educators. Automatic grading systems are useful for ensuring equity of grading of student submissions to assignments and providing timely feedback on the work. Many of these systems test submissions to assignments based on test cases and the outputs that they achieve, while others use unit tests to check the submissions. The approach presented in this paper checks submissions based on test cases but also analyses what the students actually wrote in their code. Assignment questions are constructed based around the concepts that the student are currently learning in lectures, and the patterns searched for in their submissions are based on these concepts. In this paper we show how to implement this approach effectively. We analyse the use of an automatic grading system within first year Computer Science programming modules and show that the system is straightforward to use and suited for novice programmers, while providing automatic grading and feedback. Evaluation received from students, demonstrators and lecturers show the system is extremely beneficial. The evaluation shows that such systems allow demonstrators more time to assist students during labs. Lecturers can also provide instant feedback to students while keeping track of their progress and identifying where the gaps in students’ knowledge are.

Optical mark recognition machines are used for performing automatic grading of the exam papers that have multiple choice answers. They use some mathematical operations to achieve recognizing the answers marked by the ones who take the exam. In this study, an automatic grading system developed by the use of Hough transform and a filtering system is proposed. The system introduced brings a new perspective for grading the multiple choice exam papers. It focuses on adapting the energy based circular Hough transform for identifying the marked answer bubbles. The procedure is also combined with a data filtering method known as casual median filter. The filtering system, which targets for detecting the outliers and removing them, is commonly used by the robotics and mechatronics researchers for cleaning the unwanted data. The whole system is verified by testing more than 2500 exam answer sheets of the Technical English course offered to the second year Mechanical Engineering students of the Bulent Ecevit University located in Zonguldak, Turkey. The system performance is also tested by observing the results obtained in three different case studies designed and conducted for different goals.

Multiple-choice (MC) forms have become the dominant, fast, reliable and the easiest tools of assessment in learning environments such as schools. Present automatic grading systems for such forms, though fast and accurate are either rated as expensive due to materials and equipments they require for their operation or can only operate on a particular set of forms or papers. Part of the main reasons this is so is due to the inability of such systems to generate a template from an unknown form or another form produced by a similar system. As a novel step to creating a more dynamic and efficient automatic MC test grading system, we present in this paper a template generation algorithm capable of generating a template from any type of bubble MC test form. The empty sheet is first scanned into digital form and then normalized to a given size. To consistently remove non-relevant objects such as lines from the image, a unique combination of Hough transform (HT) and Region of interest (ROI) is proposed. Finally, the true answer regions are segmented using region processing by calculating their sizes and positions, this result in the final template. The preliminary results showed that the algorithm is dynamic with overall high precision accuracy even at low resolutions.

Cataract is the leading cause of blindness worldwide. Two automatic grading systems are presented in this paper for nuclear cataract and cortical cataract diagnosis respectively. Model-based approach was applied to detect anatomical structure in slit-lamp images. Features were extracted based on the lens structure and severity of nuclear cataract was predicted using Support Vector Machines (SVM) regression. For cortical cataract, the opacity was detected using region growing. The seeds were selected by local thresholding and edge detection in radial direction. Cortical cataract was graded based on the area of cortical opacity. Both of the systems were tested by clinical data and results show that the automatic systems can provide objective grading of cataracts.

This paper presents several improvements to standardize the creation of test questions and answers and digitize all of these documents. The authors successfully researched and developed an automatic grading system for the subject of General Computer Science. The automatic grading system will select the criteria that must be assessed on the student's test, match them with the digitized criteria in the answers, and grade them according to the corresponding ratio. This solution improves efficiency and consistency and ensures accuracy and objectivity in student assessment. Experimental results show that the proposed method can entirely and effectively replace the traditional manual grading method, improving the quality of teaching and learning for General Informatics.

The quality of malt barley is a key requirement in the brewery industry. Manual inspection of barley quality is time-consuming, expensive, and prone to human bias, which can ultimately affect the final product. Therefore, an automatic and accurate grading system for barley grain is essential to ensure the consistent quality of beer. This paper presents an automatic barley grain grading system using a deep learning approach based on the texture features of the grains. Specifically, we employed a convolutional neural network (CNN) to classify barley grains into predefined qualitycategories. Unlike the conventional approach of using raw images of barley grains, our method uses texture features as inputs to the CNN model. To train and test the system, we collected images of barley grains in Ethiopia and categorized them into four quality classes (Grade 1, Grade 2, Grade 3, and Under-grade) according to the Ethiopian Standards Authority specifications. Experimental results demonstrate that using texture features instead of raw images significantly improves accuracy in determining the quality of barley grains, achieving an average testing accuracy of 98.61% and a training accuracy of 99.81%. Additionally, we compared the performance of the proposed system with three other methods—ResNet-50, DenseNet-121, and VGGNet-19. The proposed system outperformed these methods, demonstrating superior accuracy. • A barley grading dataset was created using Ethiopian Standard Agency criteria. • A model was built by integrating segmentation, feature extraction, and CNN methods. • CNN training was modified by adding segmentation and texture feature extraction. • Performance was improved over ResNet-50, DenseNet-121, and VGGNet-19.

This paper explained a research that was conducted to find the best methodology for automatic essay grading. The Winnowing method was used in this paper with specific application for Japanese language exam. With Japanese language input, romanization was implemented to change the input that were in hiragana, katakana, or kanji into romaji. After the system was developed, it still needed to find the best parameters with the highest accuracy or closer even similar to the results from human rater. From testing the short essay Japanese language exam answers with 100% of score from human rater, several testing show that the best parameter for this system is with n=2, w=2, and p=2 and the average accuracy of total score gathered for all students is 86.86%. Since the average accuracy is above 80%, it could be concluded that winnowing algorithm is suitable for Japanese language examination term.

e13607 Background: The Nottingham histologic grade (NHG) is a strong prognostic factor in early-stage breast cancer. It consists of nuclear pleomorphism, tubular formation, and mitotic count. We recently developed an artificial intelligence (AI) based automatic grading system. Methods: In this study, we have retrospectively evaluated 179,651 hematoxylin and eosin-stained patches extracted from 402 digitized biopsies of 338 patients with confirmed invasive ductal carcinoma diagnosis. The data was collected at Acıbadem University Hospital between 2017 and 2021. The digitized slides were manually labeled by seven pathologists before being used to train the deep learning models (DL). The pre-trained (on ImageNet) DL architectures which are EfficientNet backbone U-Net, YOLOv5, DenseNet161, and a modified VGG-11 have been fine-tuned with this study’s dataset for the tubule segmentation, nuclei detection, mitosis classification, and the nuclear pleomorphism classification tasks, respectively. Data augmentation for boosting the accuracy was done. Semantic segmentation for tubular formation, object detection for detection of nuclei, and image classification for the mitotic count and nuclear pleomorphism was also performed. Results: The AI-based algorithms have obtained reproducible histologic scores with mean F1 scores, sensitivities, and specificities as presented in Table. Conclusions: The AI-based automatic grading system is accurate in the evaluation of the components of NHG. This system will speed up the pathology workflow in the clinic, provide decision support to pathologists and mitigate the sensitivity associated with the traditional grading process. [Table: see text]

BackgroundEpidural spinal cord compression (ESCC) assessment is a critical step in surgical intervention for lung cancer spinal metastasis. However, clinicians can only assess compression grades through manual observation, which is subjective and inefficient. The metastatic cancer features on magnetic resonance imaging (MRI) slices are complex and difficult to identify, and there are no publicly labeled datasets currently. The objective of this study was to develop an automatic and efficient system for assessing spinal cord compression grading based on multi-tissue and feature segmentation.MethodsWe constructed a dataset containing 577 lung cancer spinal metastasis MRI slices with multi-tissue semantic segmentation masks and ESCC grading. We then proposed a progressive approach for grading ESCC via segmentation and transfer learning (TL) classification to improve clinical assessment accuracy and efficiency. First, we used an improved encoder-decoder segmentation network with a pyramid pooling module (PPM) on the bottleneck and cross-attention modules on the skip connections to segment the compression-related tissues, such as the spine, spinal cord, and bone tumors. Secondly, we proposed a dual attention network (DAN) with position and channel attention modules, and pre-trained it on ImageNet1000 to improve network performance. Finally, we used the multi-tissue features of the previous segmentation as training data and fine-tuned the DAN via TL to achieve accurate compression grading.ResultsThe segmentation results achieved an average pixel accuracy of 0.819 and an average intersection over union of 0.711. The classification accuracy reached 0.95, and the classification precision was 0.89. These results demonstrate the superiority of the proposed few-shot feature recognition and classification method in multi-target recognition. Compared to the traditional segmentation and classification models, our model exhibits enhanced performance and effectiveness in grading spinal compression.ConclusionsThe spinal cord compression grading in lung cancer spinal metastasis is an important surgical indication. The system developed herein has been verified with an accuracy of 0.95 in compression grading, which will be helpful for orthopedic surgeons to determine reasonable treatment plans for patients.

Superb microvascular flow signals in joints are important indicators for evaluating inflammation in arthritis diagnosis. Super Microvascular Imaging (SMI), a musculoskeletal ultrasound technique, captures microvascular signals with enhanced resolution, enabling improved quantitative analysis of joint superb microvascular flow. However, existing musculoskeletal ultrasound imaging predominantly relies on static observations for analyzing these signals, which are heavily influenced by subjective factors, thereby limiting diagnostic accuracy for arthritis. This study introduces a novel quantitative and automated grading method utilizing dynamic analysis through an optical flow model. Real-time dynamic quantification of superb microvascular flow signals is achieved via motion estimation and skeleton extraction based on the optical flow model. The Kappa consistency test evaluates the agreement between the automated grading system and physician assessments, with differences between the two methods analyzed. A total of 47 patient samples were included, comprising 20 males and 27 females (p = 0.307 > 0.05, χ2=1.042). The agreement between the automated grading system and physician assessments reached 70.2%, with a Kappa value of 0.627 (p < 0.001), indicating good consistency. Nonetheless, the system displayed a tendency to high-grade cases of moderate inflammation. The proposed quantitative and automated grading method for superb microvascular flow, based on dynamic analysis through an optical flow model, improves the objectivity and consistency of superb microvascular flow grading and demonstrates significant clinical potential. The method shows strong anti-interference performance in noisy signal environments, representing a promising advancement for non-invasive arthritis diagnosis.

Spreadsheet is one of the tools that can be used to learn data analysis. Data analysis in spreadsheet can be done using formula. Spreadsheet tools can also be used for exams. For the assessment, there is a problem when the number of answers that need to be checked is large, that is it takes a long time to check all the answers. For this reason, an automatic grading system (autograder) that can evaluate formula in spreadsheet is needed. The method used in developing the autograder system is matching the answer key formula with the student's answer formula. The autograder system assesses the answer by calculating the similarity of the student's answer formula with the answer key formula. This paper explains how to build an autograder system that can evaluate the formula. At the end, an autograder system has been built successfully. It has been tested with 43 testcases and all of them are passed.

Agriculture plays a crucial role in the economic growth of a country as it is one of the main means of subsistence. Recently, technological methods have been designed for the identification of plants and detection of their diseases in order to meet the new challenges facing farmers and their learning needs. This chapter provides an overview of various methods and techniques for feature extraction, segmentation and the classification of patterns of captured leaves in order to identify plant leaf diseases and the estimation of their severity. This chapter analyzes various automatic grading systems and parameters used in estimating the severity of different plant diseases and discusses a variety those plant diseases. The chapter also discusses the use of a radial basis function (RBF) kernel-based support vector machine (SVM) learning algorithm for the detection of these diseases, which include rust; tikka; powdery and downy mildew; late blight and early blight in groundnut, apple, potato and tomato plants. An analysis is made of the factors that stress plants; for example, water, pests and soil in green house plants. Leaf blast, brown spot and sheath rot detected in rice plants are discussed using a cluster validation algorithm. The chapter also deals with the assessment of severity of foliar diseases found in soybean plants (using segmentation), rice blast disease (using an SVM) and single disease severity level of plasmopara viticola in grape leaves (using a k-means clustering algorithm). The existing gaps in the technology for precision agriculture are discussed, and the requirements and demands to be met by agriculturists are presented. The whole impetus of this chapter is to motivate researchers to focus and develop efficient machine learning and classification techniques for leaf identification and disease detection to meet the new challenges in the field of agriculture.

A lightweight vision based method for pre storage potato sorting in an automatic grading system

Prostate cancer is generally graded by pathologists based on hematoxylin and eosin (H and E) stained slides. Because of the large size of the tumor areas in radical prostatectomies (RP), this task can be tedious and error prone with known high interobserver variability. Recent advancements in deep learning have enabled development of automated systems that may assist pathologists in prostate diagnostics. As prostate cancer originates from glandular tissue, an important prerequisite for development of such algorithms is the possibility to automatically differentiate between glandular tissue and other tissues. In this paper, we propose a method for automatically segmenting epithelial tissue in digitally scanned prostatectomy slides based on deep learning. We collected 30 single-center whole mount tissue sections, with reported Gleason growth patterns ranging from 3 to 5, from 27 patients that underwent RP. Two different network architectures, U-Net and regular fully convolutional networks with varying depths, were trained using a set of sparsely annotated slides. We evaluated the trained networks on exhaustively annotated regions from a separate test set. The test set contained both healthy and cancerous epithelium with different Gleason growth patterns. The results show the effectiveness of our approach given a pixel-based AUC score of 0.97. Our method contains no prior assumptions on glandular morphology, does not directly rely on the presence of lumina, and all features are learned by the network itself. The generated segmentation can be used to highlight regions of interest for pathologists and to improve cancer annotations to further enhance an automatic cancer grading system.