Analysis of an automatic grading system within first year Computer Science programming modules

The demand of scoring natural language responses has created a need for new computational tools that can be applied to automatically grade student essays. Systems for automatic essay assessment have been commercially available since 1990's. However, the progress in the field was obstructed by a lack of qualitative information regarding the effectiveness of such systems. Most of the research in automatic essay grading has been associated with English writing due to its widespread use and the availability of more learner collection and language processing software for the language. In addition, there is large number of commercial software for grading programming assignments automatically. In this work, we investigate document semantic similarity based on Latent Semantic Analysis (LSA) and on Latent Dirichlet Allocation (LDA). We use an open-source Python software, Gensim, to develop and implement an essay grading system able to compare an essay to an answer-key and assign to it a grade based on semantic similarity between the two. We test our tool on variable-size essays and conduct experimental tests to compare the results obtained from human grader (professor) and those obtained from the automatic grading system. Results show high correlation between the professor grades and the grades assigned by both modeling techniques. However, LSA-based modeling showed more promising results than the LDA-based method.

This paper aims to investigate the effectiveness of automatic grading systems, with a focus on their uses within Computer Science. Automatic grading systems have seen a rise in popularity in recent years with publications concerning automatic grading systems usually linked to a specific system. This paper will discuss the factors that need to be considered when using automatic grading, regardless of which system is being used, and will make recommendations for each factor. This discussion is based on the authors' experience of using an automatic grading system in a CS1 environment. From the research conducted, many elements should be considered when using these systems. These include how the code will be tested, the need for plagiarism checks and how marks are awarded. The findings of this study suggest there is a lack of defined standards when using these systems. This analysis of the considerations provides valuable insight into how these systems should be used and what the standards should be built on.

Multiple-choice (MC) forms have become the dominant, fast, reliable and the easiest tools of assessment in learning environments such as schools. Present automatic grading systems for such forms, though fast and accurate are either rated as expensive due to materials and equipments they require for their operation or can only operate on a particular set of forms or papers. Part of the main reasons this is so is due to the inability of such systems to generate a template from an unknown form or another form produced by a similar system. As a novel step to creating a more dynamic and efficient automatic MC test grading system, we present in this paper a template generation algorithm capable of generating a template from any type of bubble MC test form. The empty sheet is first scanned into digital form and then normalized to a given size. To consistently remove non-relevant objects such as lines from the image, a unique combination of Hough transform (HT) and Region of interest (ROI) is proposed. Finally, the true answer regions are segmented using region processing by calculating their sizes and positions, this result in the final template. The preliminary results showed that the algorithm is dynamic with overall high precision accuracy even at low resolutions.

Optical mark recognition machines are used for performing automatic grading of the exam papers that have multiple choice answers. They use some mathematical operations to achieve recognizing the answers marked by the ones who take the exam. In this study, an automatic grading system developed by the use of Hough transform and a filtering system is proposed. The system introduced brings a new perspective for grading the multiple choice exam papers. It focuses on adapting the energy based circular Hough transform for identifying the marked answer bubbles. The procedure is also combined with a data filtering method known as casual median filter. The filtering system, which targets for detecting the outliers and removing them, is commonly used by the robotics and mechatronics researchers for cleaning the unwanted data. The whole system is verified by testing more than 2500 exam answer sheets of the Technical English course offered to the second year Mechanical Engineering students of the Bulent Ecevit University located in Zonguldak, Turkey. The system performance is also tested by observing the results obtained in three different case studies designed and conducted for different goals.

The quality of malt barley is a key requirement in the brewery industry. Manual inspection of barley quality is time-consuming, expensive, and prone to human bias, which can ultimately affect the final product. Therefore, an automatic and accurate grading system for barley grain is essential to ensure the consistent quality of beer. This paper presents an automatic barley grain grading system using a deep learning approach based on the texture features of the grains. Specifically, we employed a convolutional neural network (CNN) to classify barley grains into predefined qualitycategories. Unlike the conventional approach of using raw images of barley grains, our method uses texture features as inputs to the CNN model. To train and test the system, we collected images of barley grains in Ethiopia and categorized them into four quality classes (Grade 1, Grade 2, Grade 3, and Under-grade) according to the Ethiopian Standards Authority specifications. Experimental results demonstrate that using texture features instead of raw images significantly improves accuracy in determining the quality of barley grains, achieving an average testing accuracy of 98.61% and a training accuracy of 99.81%. Additionally, we compared the performance of the proposed system with three other methods—ResNet-50, DenseNet-121, and VGGNet-19. The proposed system outperformed these methods, demonstrating superior accuracy. • A barley grading dataset was created using Ethiopian Standard Agency criteria. • A model was built by integrating segmentation, feature extraction, and CNN methods. • CNN training was modified by adding segmentation and texture feature extraction. • Performance was improved over ResNet-50, DenseNet-121, and VGGNet-19.

Cataract is the leading cause of blindness worldwide. Two automatic grading systems are presented in this paper for nuclear cataract and cortical cataract diagnosis respectively. Model-based approach was applied to detect anatomical structure in slit-lamp images. Features were extracted based on the lens structure and severity of nuclear cataract was predicted using Support Vector Machines (SVM) regression. For cortical cataract, the opacity was detected using region growing. The seeds were selected by local thresholding and edge detection in radial direction. Cortical cataract was graded based on the area of cortical opacity. Both of the systems were tested by clinical data and results show that the automatic systems can provide objective grading of cataracts.

This paper explained a research that was conducted to find the best methodology for automatic essay grading. The Winnowing method was used in this paper with specific application for Japanese language exam. With Japanese language input, romanization was implemented to change the input that were in hiragana, katakana, or kanji into romaji. After the system was developed, it still needed to find the best parameters with the highest accuracy or closer even similar to the results from human rater. From testing the short essay Japanese language exam answers with 100% of score from human rater, several testing show that the best parameter for this system is with n=2, w=2, and p=2 and the average accuracy of total score gathered for all students is 86.86%. Since the average accuracy is above 80%, it could be concluded that winnowing algorithm is suitable for Japanese language examination term.

Diabetic maculopathy is the major cause of irreversible vision loss due to retinopathy and is found in 10% of the world diabetic population. Compulsory mass screening will help to identify the maculopathy at early stage and reduce the risk of severe vision loss. In this paper, we present a computer based system for automatic detection and grading of diabetic maculopathy severity level without manual intervention. The optic disc is detected automatically and its location and diameter is used to detect fovea and to mark the macular region respectively. Next, hard exudates are detected using clustering and mathematical morphological techniques. Based on the location of exudates in marked macular region the severity level of maculopathy is classified into mild, moderate and severe. The method achieves a sensitivity of 95.6% and specificity of 96.15% with 148 retinal images for detecting maculopathy stages in fundus images as comparable to that of human expert.

PurposeIn order to assess the severity and the progression of a unilateral peripheral facial palsy the Sunnybrook Facial Grading System (SFGS) is a well-established grading system due to its clinical relevance, sensitivity, and robust measuring method. However, training is required in order to achieve a high inter-rater reliability. This study investigated the automated grading of facial palsy patients based on the SFGS using a convolutional neural network. MethodsA total of 116 patients with a unilateral peripheral facial palsy and 9 healthy subjects were recorded performing the Sunnybrook poses. A separate model was trained for each of the 13 elements of the SFGS and then used to calculate the Sunnybrook subscores and composite score. The performance of the automated grading system was compared to three clinicians experienced in the grading of a facial palsy. ResultsThe inter-rater reliability of the convolutional neural network was within the range of human observers, with an average intra-class correlation coefficient of 0.87 for the composite Sunnybrook score, 0.45 for the resting symmetry subscore, 0.89 for the symmetry of voluntary movement subscore, and 0.77 for the synkinesis subscore. ConclusionsThis study showed the potential of the automated SFGS to be implemented in a clinical setting. The automated grading system adhered to the original SFGS, which makes the implementation and interpretation of the automated grading more straightforward. The automated system can be implemented in numerous settings such as online consults in an e-Health environment, since the model used 2D images captured from a video recording.

BackgroundEpidural spinal cord compression (ESCC) assessment is a critical step in surgical intervention for lung cancer spinal metastasis. However, clinicians can only assess compression grades through manual observation, which is subjective and inefficient. The metastatic cancer features on magnetic resonance imaging (MRI) slices are complex and difficult to identify, and there are no publicly labeled datasets currently. The objective of this study was to develop an automatic and efficient system for assessing spinal cord compression grading based on multi-tissue and feature segmentation.MethodsWe constructed a dataset containing 577 lung cancer spinal metastasis MRI slices with multi-tissue semantic segmentation masks and ESCC grading. We then proposed a progressive approach for grading ESCC via segmentation and transfer learning (TL) classification to improve clinical assessment accuracy and efficiency. First, we used an improved encoder-decoder segmentation network with a pyramid pooling module (PPM) on the bottleneck and cross-attention modules on the skip connections to segment the compression-related tissues, such as the spine, spinal cord, and bone tumors. Secondly, we proposed a dual attention network (DAN) with position and channel attention modules, and pre-trained it on ImageNet1000 to improve network performance. Finally, we used the multi-tissue features of the previous segmentation as training data and fine-tuned the DAN via TL to achieve accurate compression grading.ResultsThe segmentation results achieved an average pixel accuracy of 0.819 and an average intersection over union of 0.711. The classification accuracy reached 0.95, and the classification precision was 0.89. These results demonstrate the superiority of the proposed few-shot feature recognition and classification method in multi-target recognition. Compared to the traditional segmentation and classification models, our model exhibits enhanced performance and effectiveness in grading spinal compression.ConclusionsThe spinal cord compression grading in lung cancer spinal metastasis is an important surgical indication. The system developed herein has been verified with an accuracy of 0.95 in compression grading, which will be helpful for orthopedic surgeons to determine reasonable treatment plans for patients.

The integration of AI in educational assessments has revolutionized the evaluation process, offering unprecedented scalability, efficiency, and objectivity. The complexity and opacity of AI models, particularly deep learning systems, have raised significant concerns regarding transparency, fairness, and trust. This chapter explores the role of XAI in automated grading systems, emphasizing its importance in addressing these concerns. By combining techniques from computer vision and NLP, AI-based grading systems can accurately evaluate both handwritten and text-based student responses. The chapter investigates the application of rule-based and hybrid approaches to enhance model interpretability, ensuring that grading decisions are both reliable and transparent. The significance of addressing ethical considerations, such as bias mitigation and data privacy, was also discussed, highlighting the need for AI systems that align with institutional standards and educational values. Case studies illustrating the effectiveness of XAI techniques in academic assessments are presented, demonstrating their capacity to build trust with both educators and students. The chapter concludes with a forward-looking perspective on the future of AI in education, proposing strategies for further enhancing transparency and accountability in automated grading systems.

e13607 Background: The Nottingham histologic grade (NHG) is a strong prognostic factor in early-stage breast cancer. It consists of nuclear pleomorphism, tubular formation, and mitotic count. We recently developed an artificial intelligence (AI) based automatic grading system. Methods: In this study, we have retrospectively evaluated 179,651 hematoxylin and eosin-stained patches extracted from 402 digitized biopsies of 338 patients with confirmed invasive ductal carcinoma diagnosis. The data was collected at Acıbadem University Hospital between 2017 and 2021. The digitized slides were manually labeled by seven pathologists before being used to train the deep learning models (DL). The pre-trained (on ImageNet) DL architectures which are EfficientNet backbone U-Net, YOLOv5, DenseNet161, and a modified VGG-11 have been fine-tuned with this study’s dataset for the tubule segmentation, nuclei detection, mitosis classification, and the nuclear pleomorphism classification tasks, respectively. Data augmentation for boosting the accuracy was done. Semantic segmentation for tubular formation, object detection for detection of nuclei, and image classification for the mitotic count and nuclear pleomorphism was also performed. Results: The AI-based algorithms have obtained reproducible histologic scores with mean F1 scores, sensitivities, and specificities as presented in Table. Conclusions: The AI-based automatic grading system is accurate in the evaluation of the components of NHG. This system will speed up the pathology workflow in the clinic, provide decision support to pathologists and mitigate the sensitivity associated with the traditional grading process. [Table: see text]

This paper presents several improvements to standardize the creation of test questions and answers and digitize all of these documents. The authors successfully researched and developed an automatic grading system for the subject of General Computer Science. The automatic grading system will select the criteria that must be assessed on the student's test, match them with the digitized criteria in the answers, and grade them according to the corresponding ratio. This solution improves efficiency and consistency and ensures accuracy and objectivity in student assessment. Experimental results show that the proposed method can entirely and effectively replace the traditional manual grading method, improving the quality of teaching and learning for General Informatics.

Vision-Guided Robotic System for Automatic Fish Quality Grading and Packaging

Simple SummaryAn established dataset of histopathology images obtained by biopsy and reviewed by two pathologists is used to create a two-stage oral squamous cell carcinoma diagnostic AI-based system. In the first stage, automated multiclass grading of OSCC is performed to improve the objectivity and reproducibility of histopathological examination. Furthermore, in the second stage, semantic segmentation of OSCC on epithelial and stromal tissue is performed in order to assist the clinician in discovering new informative features. Proposed AI-system based on deep convolutional neural networks and preprocessing methods achieved satisfactory results in terms of multiclass grading and segmenting. This research is the first step in analysing the tumor microenvironment, i.e., tumor-stroma ratio and segmentation of the microenvironment cells.Oral squamous cell carcinoma is most frequent histological neoplasm of head and neck cancers, and although it is localized in a region that is accessible to see and can be detected very early, this usually does not occur. The standard procedure for the diagnosis of oral cancer is based on histopathological examination, however, the main problem in this kind of procedure is tumor heterogeneity where a subjective component of the examination could directly impact patient-specific treatment intervention. For this reason, artificial intelligence (AI) algorithms are widely used as computational aid in the diagnosis for classification and segmentation of tumors, in order to reduce inter- and intra-observer variability. In this research, a two-stage AI-based system for automatic multiclass grading (the first stage) and segmentation of the epithelial and stromal tissue (the second stage) from oral histopathological images is proposed in order to assist the clinician in oral squamous cell carcinoma diagnosis. The integration of Xception and SWT resulted in the highest classification value of 0.963 (σ = 0.042) AUCmacro and 0.966 (σ = 0.027) AUCmicro while using DeepLabv3+ along with Xception_65 as backbone and data preprocessing, semantic segmentation prediction resulted in 0.878 (σ = 0.027) mIOU and 0.955 (σ = 0.014) F1 score. Obtained results reveal that the proposed AI-based system has great potential in the diagnosis of OSCC.