A novel approach to a dynamic template generation algorithm for multiple-choice forms

Optical mark recognition machines are used for performing automatic grading of the exam papers that have multiple choice answers. They use some mathematical operations to achieve recognizing the answers marked by the ones who take the exam. In this study, an automatic grading system developed by the use of Hough transform and a filtering system is proposed. The system introduced brings a new perspective for grading the multiple choice exam papers. It focuses on adapting the energy based circular Hough transform for identifying the marked answer bubbles. The procedure is also combined with a data filtering method known as casual median filter. The filtering system, which targets for detecting the outliers and removing them, is commonly used by the robotics and mechatronics researchers for cleaning the unwanted data. The whole system is verified by testing more than 2500 exam answer sheets of the Technical English course offered to the second year Mechanical Engineering students of the Bulent Ecevit University located in Zonguldak, Turkey. The system performance is also tested by observing the results obtained in three different case studies designed and conducted for different goals.

The demand of scoring natural language responses has created a need for new computational tools that can be applied to automatically grade student essays. Systems for automatic essay assessment have been commercially available since 1990's. However, the progress in the field was obstructed by a lack of qualitative information regarding the effectiveness of such systems. Most of the research in automatic essay grading has been associated with English writing due to its widespread use and the availability of more learner collection and language processing software for the language. In addition, there is large number of commercial software for grading programming assignments automatically. In this work, we investigate document semantic similarity based on Latent Semantic Analysis (LSA) and on Latent Dirichlet Allocation (LDA). We use an open-source Python software, Gensim, to develop and implement an essay grading system able to compare an essay to an answer-key and assign to it a grade based on semantic similarity between the two. We test our tool on variable-size essays and conduct experimental tests to compare the results obtained from human grader (professor) and those obtained from the automatic grading system. Results show high correlation between the professor grades and the grades assigned by both modeling techniques. However, LSA-based modeling showed more promising results than the LDA-based method.

This contribution will report on, and evaluate, the use of multiple-choice questions, in both continuous assessment and terminal examination modes, on a first year module in electrical engineering at Dublin Institute of Technology. The author’s experiences are that multiple-choice questions tend to be student friendly (e.g. given a choice in a terminal examination, students will opt to attempt the questions with multiple choice parts). The author has used the assessment method over three academic years, and some assessment data is reported and analysed in the contribution. The pedagogical approach is explicitly pragmatic.

This paper aims to investigate the effectiveness of automatic grading systems, with a focus on their uses within Computer Science. Automatic grading systems have seen a rise in popularity in recent years with publications concerning automatic grading systems usually linked to a specific system. This paper will discuss the factors that need to be considered when using automatic grading, regardless of which system is being used, and will make recommendations for each factor. This discussion is based on the authors' experience of using an automatic grading system in a CS1 environment. From the research conducted, many elements should be considered when using these systems. These include how the code will be tested, the need for plagiarism checks and how marks are awarded. The findings of this study suggest there is a lack of defined standards when using these systems. This analysis of the considerations provides valuable insight into how these systems should be used and what the standards should be built on.

Reliable and pedagogically sound automated feedback and grading systems are highly coveted by educators. Automatic grading systems are useful for ensuring equity of grading of student submissions to assignments and providing timely feedback on the work. Many of these systems test submissions to assignments based on test cases and the outputs that they achieve, while others use unit tests to check the submissions. The approach presented in this paper checks submissions based on test cases but also analyses what the students actually wrote in their code. Assignment questions are constructed based around the concepts that the student are currently learning in lectures, and the patterns searched for in their submissions are based on these concepts. In this paper we show how to implement this approach effectively. We analyse the use of an automatic grading system within first year Computer Science programming modules and show that the system is straightforward to use and suited for novice programmers, while providing automatic grading and feedback. Evaluation received from students, demonstrators and lecturers show the system is extremely beneficial. The evaluation shows that such systems allow demonstrators more time to assist students during labs. Lecturers can also provide instant feedback to students while keeping track of their progress and identifying where the gaps in students’ knowledge are.

Tests of educational achievement typically present items in the multiple-choice format. Some achievement test items may be so saturated with aptitude (Willingham, 1980) as to be insensitive to skills acquired through education. Multiple-choice tests are ill-suited for assessing productive thinking and problem-solving skills, skills that often constitute important objectives of education. Viewed as incentives for learning, multiple-choice tests may impede student progress toward these objectives. There is need for accelerated research to develop alternatives to multiple-choice achievement tests, with content selected to match the specified educational objectives.

The quality of malt barley is a key requirement in the brewery industry. Manual inspection of barley quality is time-consuming, expensive, and prone to human bias, which can ultimately affect the final product. Therefore, an automatic and accurate grading system for barley grain is essential to ensure the consistent quality of beer. This paper presents an automatic barley grain grading system using a deep learning approach based on the texture features of the grains. Specifically, we employed a convolutional neural network (CNN) to classify barley grains into predefined qualitycategories. Unlike the conventional approach of using raw images of barley grains, our method uses texture features as inputs to the CNN model. To train and test the system, we collected images of barley grains in Ethiopia and categorized them into four quality classes (Grade 1, Grade 2, Grade 3, and Under-grade) according to the Ethiopian Standards Authority specifications. Experimental results demonstrate that using texture features instead of raw images significantly improves accuracy in determining the quality of barley grains, achieving an average testing accuracy of 98.61% and a training accuracy of 99.81%. Additionally, we compared the performance of the proposed system with three other methods—ResNet-50, DenseNet-121, and VGGNet-19. The proposed system outperformed these methods, demonstrating superior accuracy. • A barley grading dataset was created using Ethiopian Standard Agency criteria. • A model was built by integrating segmentation, feature extraction, and CNN methods. • CNN training was modified by adding segmentation and texture feature extraction. • Performance was improved over ResNet-50, DenseNet-121, and VGGNet-19.

Diabetic maculopathy is the major cause of irreversible vision loss due to retinopathy and is found in 10% of the world diabetic population. Compulsory mass screening will help to identify the maculopathy at early stage and reduce the risk of severe vision loss. In this paper, we present a computer based system for automatic detection and grading of diabetic maculopathy severity level without manual intervention. The optic disc is detected automatically and its location and diameter is used to detect fovea and to mark the macular region respectively. Next, hard exudates are detected using clustering and mathematical morphological techniques. Based on the location of exudates in marked macular region the severity level of maculopathy is classified into mild, moderate and severe. The method achieves a sensitivity of 95.6% and specificity of 96.15% with 148 retinal images for detecting maculopathy stages in fundus images as comparable to that of human expert.

PurposeIn order to assess the severity and the progression of a unilateral peripheral facial palsy the Sunnybrook Facial Grading System (SFGS) is a well-established grading system due to its clinical relevance, sensitivity, and robust measuring method. However, training is required in order to achieve a high inter-rater reliability. This study investigated the automated grading of facial palsy patients based on the SFGS using a convolutional neural network. MethodsA total of 116 patients with a unilateral peripheral facial palsy and 9 healthy subjects were recorded performing the Sunnybrook poses. A separate model was trained for each of the 13 elements of the SFGS and then used to calculate the Sunnybrook subscores and composite score. The performance of the automated grading system was compared to three clinicians experienced in the grading of a facial palsy. ResultsThe inter-rater reliability of the convolutional neural network was within the range of human observers, with an average intra-class correlation coefficient of 0.87 for the composite Sunnybrook score, 0.45 for the resting symmetry subscore, 0.89 for the symmetry of voluntary movement subscore, and 0.77 for the synkinesis subscore. ConclusionsThis study showed the potential of the automated SFGS to be implemented in a clinical setting. The automated grading system adhered to the original SFGS, which makes the implementation and interpretation of the automated grading more straightforward. The automated system can be implemented in numerous settings such as online consults in an e-Health environment, since the model used 2D images captured from a video recording.

BackgroundWritten assessments fall into two classes: constructed-response or open-ended questions, such as the essay and a number of variants of the short-answer question, and selected-response or closed-ended questions; typically in the form of multiple-choice. It is widely believed that constructed response written questions test higher order cognitive processes in a manner that multiple-choice questions cannot, and consequently have higher validity.DiscussionAn extensive review of the literature suggests that in summative assessment neither premise is evidence-based. Well-structured open-ended and multiple-choice questions appear equivalent in their ability to assess higher cognitive functions, and performance in multiple-choice assessments may correlate more highly than the open-ended format with competence demonstrated in clinical practice following graduation. Studies of construct validity suggest that both formats measure essentially the same dimension, at least in mathematics, the physical sciences, biology and medicine. The persistence of the open-ended format in summative assessment may be due to the intuitive appeal of the belief that synthesising an answer to an open-ended question must be both more cognitively taxing and similar to actual experience than is selecting a correct response. I suggest that cognitive-constructivist learning theory would predict that a well-constructed context-rich multiple-choice item represents a complex problem-solving exercise which activates a sequence of cognitive processes which closely parallel those required in clinical practice, hence explaining the high validity of the multiple-choice format.SummaryThe evidence does not support the proposition that the open-ended assessment format is superior to the multiple-choice format, at least in exit-level summative assessment, in terms of either its ability to test higher-order cognitive functioning or its validity. This is explicable using a theory of mental models, which might predict that the multiple-choice format will have higher validity, a statement for which some empiric support exists. Given the superior reliability and cost-effectiveness of the multiple-choice format consideration should be given to phasing out open-ended format questions in summative assessment. Whether the same applies to non-exit-level assessment and formative assessment is a question which remains to be answered; particularly in terms of the educational effect of testing, an area which deserves intensive study.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-014-0249-2) contains supplementary material, which is available to authorized users.

The integration of AI in educational assessments has revolutionized the evaluation process, offering unprecedented scalability, efficiency, and objectivity. The complexity and opacity of AI models, particularly deep learning systems, have raised significant concerns regarding transparency, fairness, and trust. This chapter explores the role of XAI in automated grading systems, emphasizing its importance in addressing these concerns. By combining techniques from computer vision and NLP, AI-based grading systems can accurately evaluate both handwritten and text-based student responses. The chapter investigates the application of rule-based and hybrid approaches to enhance model interpretability, ensuring that grading decisions are both reliable and transparent. The significance of addressing ethical considerations, such as bias mitigation and data privacy, was also discussed, highlighting the need for AI systems that align with institutional standards and educational values. Case studies illustrating the effectiveness of XAI techniques in academic assessments are presented, demonstrating their capacity to build trust with both educators and students. The chapter concludes with a forward-looking perspective on the future of AI in education, proposing strategies for further enhancing transparency and accountability in automated grading systems.

Vision-Guided Robotic System for Automatic Fish Quality Grading and Packaging

Simple SummaryAn established dataset of histopathology images obtained by biopsy and reviewed by two pathologists is used to create a two-stage oral squamous cell carcinoma diagnostic AI-based system. In the first stage, automated multiclass grading of OSCC is performed to improve the objectivity and reproducibility of histopathological examination. Furthermore, in the second stage, semantic segmentation of OSCC on epithelial and stromal tissue is performed in order to assist the clinician in discovering new informative features. Proposed AI-system based on deep convolutional neural networks and preprocessing methods achieved satisfactory results in terms of multiclass grading and segmenting. This research is the first step in analysing the tumor microenvironment, i.e., tumor-stroma ratio and segmentation of the microenvironment cells.Oral squamous cell carcinoma is most frequent histological neoplasm of head and neck cancers, and although it is localized in a region that is accessible to see and can be detected very early, this usually does not occur. The standard procedure for the diagnosis of oral cancer is based on histopathological examination, however, the main problem in this kind of procedure is tumor heterogeneity where a subjective component of the examination could directly impact patient-specific treatment intervention. For this reason, artificial intelligence (AI) algorithms are widely used as computational aid in the diagnosis for classification and segmentation of tumors, in order to reduce inter- and intra-observer variability. In this research, a two-stage AI-based system for automatic multiclass grading (the first stage) and segmentation of the epithelial and stromal tissue (the second stage) from oral histopathological images is proposed in order to assist the clinician in oral squamous cell carcinoma diagnosis. The integration of Xception and SWT resulted in the highest classification value of 0.963 (σ = 0.042) AUCmacro and 0.966 (σ = 0.027) AUCmicro while using DeepLabv3+ along with Xception_65 as backbone and data preprocessing, semantic segmentation prediction resulted in 0.878 (σ = 0.027) mIOU and 0.955 (σ = 0.014) F1 score. Obtained results reveal that the proposed AI-based system has great potential in the diagnosis of OSCC.

The citrus industry plays a pivotal role in the global agricultural sector, contributing significantly to the economy. Approximately 161 million tonnes of citrus fruits were produced worldwide. One third of the total production is used in processing industry. The price of citrus fruits is highly dependent on its quality parameters such as size, citrus color index, maturity index, diseases etc. Therefore, grading and sorting of citrus fruits based on the quality parameters is demand of the market to maximize the profit. Manual grading and sorting are labour-intensive, time-consuming, and often prone to human error. Automatic grading and sorting of citrus fruits can resolve the above-stated issues. In this paper, we utilize the pattern recognition and image processing potential of deep learning techniques and propose a deep learning based automated system to revolutionize the citrus fruits grading process. The system utilizes Convolutional Neural Networks (CNNs) to extract high-level features from lemon, orange images, allowing for accurate classification based on quality parameters such as size, color, and visual defects. We curated a dataset comprising diverse lemon, orange samples from Kaggle to evaluate the performance of the developed model. Additionally, our system incorporates real-time processing capabilities, ensuring a swift and continuous flow of fruits through the sorting process. Thus, it offers a robust and scalable solution for the citrus industry to enhance profit in the global citrus market.

Handing in a paper or exercise and merely receiving “bad” or “incorrect” as feedback is not very helpful when the goal is to improve. Unfortunately, this is currently the kind of feedback given by Automatic Short Answer Grading (ASAG) systems. One of the reasons for this is a lack of content-focused elaborated feedback datasets. To encourage research on explainable and understandable feedback systems, we present the Short Answer Feedback dataset (SAF). Similar to other ASAG datasets, SAF contains learner responses and reference answers to German and English questions. However, instead of only assigning a label or score to the learners’ answers, SAF also contains elaborated feedback explaining the given score. Thus, SAF enables supervised training of models that grade answers and explain where and why mistakes were made. This paper discusses the need for enhanced feedback models in real-world pedagogical scenarios, describes the dataset annotation process, gives a comprehensive analysis of SAF, and provides T5-based baselines for future comparison.