Mixed Method Development of Evaluation Metrics

The paper addresses the limitations of traditional evaluation metrics for Question Answering (QA) systems that primarily focus on syntax and n-gram similarity. We propose a novel model-based evaluation metric, MQA-metric, and create a human-judgment-based dataset, squad-qametric and marco-qametric, to validate our approach. The research aims to solve several key problems: the objectivity in dataset labeling, the effectiveness of metrics when there is no syntax similarity, the impact of answer length on metric performance, and the influence of real answer quality on metric results. To tackle these challenges, we designed an interface for dataset labeling and conducted extensive experiments with human reviewers. Our analysis shows that the MQA-metric outperforms traditional metrics like BLEU, ROUGE and METEOR. Unlike existing metrics, MQA-metric leverages semantic comprehension through large language models (LLMs), enabling it to capture contextual nuances and synonymous expressions more effectively. This approach sets a standard for evaluating QA systems by prioritizing semantic accuracy over surface-level similarities. The proposed metric correlates better with human judgment, making it a more reliable tool for evaluating QA systems. Our contributions include the development of a robust evaluation workflow, creation of high-quality datasets, and an extensive comparison with existing evaluation methods. The results indicate that our model-based approach provides a significant improvement in assessing the quality of QA systems, which is crucial for their practical application and trustworthiness.

Clustering algorithms are essential in data analysis, but evaluating their performance is challenging when the true labels are not available, especially for non-convex clusters. Traditional performance evaluation metrics struggle to identify clustering quality, often assigning higher scores for linearly separated clusters than the true clusters. We propose an original approach to distance computation that accounts for the data structure, thus improving the clustering quality evaluation for non-convex clusters without affecting other shapes of clusters. We also showcase the applicability of this method through a modified version of K-Means using the proposed method that is capable of correctly separating non-convex clusters. The validation included the analysis of performance and time complexity of 3 traditional clustering quality evaluation metrics and the K-Means clustering algorithm against their augmented versions with the proposed approach. This analysis conducted on 7 benchmark synthetic datasets and 6 real datasets with various numbers of examples and features of diverse characteristics and joint complexities: simple convex clusters, overlapped and imbalanced clusters, and non-convex clusters. Through these analyses, we show the ineffectiveness of traditional methods and that the proposed approach overcomes the weaknesses of traditional methods.

Robust assessment of artificial intelligence (AI) models in medical imaging is paramount for reliable clinical integration. This international collaborative review paper provides an overview of key evaluation metrics across diverse tasks, including classification, regression, survival analysis, detection, and segmentation, as well as specialized metrics for calibration, foundation models, large language models, and synthetic images. Challenges of comparing models statistically and translating metric scores to clinical practice are also discussed. For each section, the paper outlines fundamental metrics, identifies common pitfalls and misapplications, and offers recommendations for more robust evaluations. Key recommendations often involve utilizing multiple, complementary metrics tailored to the specific task and dataset properties, transparent reporting of methodology, and critically, considering the clinical utility and real-world implications of model performance. Ultimately, effective evaluation requires a comprehensive, context-aware approach that goes beyond statistical metrics to ensure model trust and clinical relevance. The authors hope this review will serve as a practical reference for researchers aiming to implement robust and clinically meaningful AI evaluations in medical imaging. • This review outlines the key metrics for evaluating medical imaging AI. • Common pitfalls and misapplications are critically examined, with corresponding recommendations provided for each. • Appropriate metric selection depends on the specific AI task. • Foundation and generative models require broader evaluation methods, beyond traditional evaluation metrics. • A multi-metric, context-aware evaluation is essential for reliability.

At present, the manual survey based traditional building damage detection highly depends on the prior experience and subjective judgment of the professional inspector. To realize the intelligent detection of building crack, this study proposes a crack detection method using a modified YOLOv4 based on the improvement of focal loss and adding attention modules. Taking consideration of both uneven positive crack and negative noncrack samples as well as different training coefficients in the loss function, the YOLOv4 model was deeply trained. To solve the inapplicability of traditional evaluation metrics for complex regional cracks, a refined evaluation metric is proposed. The proposed method achieved efficient and accurate detection of complex building cracks as large as 69.18%.

Recommender systems have been important helpful software system techniques and tools giving desirable recommendation to items that are valuable to end-users. In past era, the measure of services, users, and online data have expanded very quickly, so it recognizes the Big data that is the reason there is examination issue for recommender systems. As a result, efficiency and scalability are issues in existing service recommender systems when the analyzing or processing Big Data. Also, the same rankings and ratings are provided by existing service recommender systems to various users without counting his/her distinctive preferences and in this manner, doesn't meet user's customized essentials. In this paper, it's proposed to utilize service recommendation techniques, which goes for showing a customized service recommendation set and recommender systems are recommending the foremost valuable services to end-users. To improve efficiency and scalability issue in massive data surroundings, it is implemented in hadoop framework which uses map reduce parallel processing model.

A multi-institute automated segmentation evaluation on a standard dataset: Findings from the international workshop on osteoarthritis imaging segmentation challenge

The standard system-based evaluation paradigm has focused on assessing the performance of retrieval systems in serving the best results for a single query. Real users, however, often begin an interaction with a search engine with a sufficiently under-specified query that they will need to reformulate be-fore they find what they are looking for. In this work we con-sider the problem of evaluating retrieval systems over test collections of multi-query sessions. We propose two families of measures: a model-free family that makes no assumption about the user’s behavior over a session, and a model-based family with a simple model of user interactions over the ses-sion. In both cases we generalize traditional evaluation met-rics such as average precision to multi-query session evalua-tion. We demonstrate the behavior of the proposed metrics by using the new TREC 2010 Session track collection and simulations over the TREC-9 Query track collection.

In this paper, an efficient deep convolutional neural network (CNN) architecture is proposed based on depthwise temporal convolution along with a robust end-to-end scheme to automatically detect and classify arrhythmia from denoised electrocardiogram (ECG) signal, which is termed as `DeepArrNet'. Firstly, considering the variational pattern of wavelet denoised ECG data, a realistic augmentation scheme is designed that offers a reduction in class imbalance as well as increased data variations. A structural unit, namely PTP (Pontwise-Temporal-Pointwise Convolution) unit, is designed with its variants where depthwise temporal convolutions with varying kernel sizes are incorporated along with prior and post pointwise convolution. Afterward, a deep neural network architecture is constructed based on the proposed structural unit where series of such structural units are stacked together while increasing the kernel sizes for depthwise temporal convolutions in successive units along with the residual linkage between units through feature addition. Moreover, multiple depthwise temporal convolutions are introduced with varying kernel sizes in each structural unit to make the process more efficient while strided convolutions are utilized in the residual linkage between subsequent units to compensate the increased computational complexity. This architecture provides the opportunity to explore the temporal features in between convolutional layers more optimally from different perspectives utilizing diversified temporal kernels. Extensive experimentations are carried out on two publicly available datasets to validate the proposed scheme that results in outstanding performances in all traditional evaluation metrics outperforming other state-of-the-art approaches.

In many databases, biocuration primarily involves literature curation, which usually involves retrieving relevant articles, extracting information that will translate into annotations and identifying new incoming literature. As the volume of biological literature increases, the use of text mining to assist in biocuration becomes increasingly relevant. A number of groups have developed tools for text mining from a computer science/linguistics perspective, and there are many initiatives to curate some aspect of biology from the literature. Some biocuration efforts already make use of a text mining tool, but there have not been many broad-based systematic efforts to study which aspects of a text mining tool contribute to its usefulness for a curation task. Here, we report on an effort to bring together text mining tool developers and database biocurators to test the utility and usability of tools. Six text mining systems presenting diverse biocuration tasks participated in a formal evaluation, and appropriate biocurators were recruited for testing. The performance results from this evaluation indicate that some of the systems were able to improve efficiency of curation by speeding up the curation task significantly (∼1.7- to 2.5-fold) over manual curation. In addition, some of the systems were able to improve annotation accuracy when compared with the performance on the manually curated set. In terms of inter-annotator agreement, the factors that contributed to significant differences for some of the systems included the expertise of the biocurator on the given curation task, the inherent difficulty of the curation and attention to annotation guidelines. After the task, annotators were asked to complete a survey to help identify strengths and weaknesses of the various systems. The analysis of this survey highlights how important task completion is to the biocurators’ overall experience of a system, regardless of the system’s high score on design, learnability and usability. In addition, strategies to refine the annotation guidelines and systems documentation, to adapt the tools to the needs and query types the end user might have and to evaluate performance in terms of efficiency, user interface, result export and traditional evaluation metrics have been analyzed during this task. This analysis will help to plan for a more intense study in BioCreative IV.

The performance of emotive text classification using affective hierarchical schemes (e.g., WordNet-Affect) is often evaluated using the same traditional measures used to evaluate the performance of when a finite set of isolated classes are used. However, applying such measures means the full characteristics and structure of the emotive hierarchical scheme are not considered. Thus, the overall performance of emotive text classification using emotion hierarchical schemes is often inaccurately reported and may lead to ineffective information retrieval and decision making. This paper provides a comparative investigation into how methods used in hierarchical classification problems in other domains, which extend traditional evaluation metrics to consider the characteristics of the hierarchical classification scheme, can be applied and subsequently improve the classification of emotive texts. This study investigates the classification performance of three widely used classifiers, Naive Bayes, J48 Decision Tree, and SVM, following the application of the aforementioned methods. The results demonstrated that all the methods improved the emotion classification. However, the most notable improvement was recorded when a depth-based method was applied to both the testing and validation data, where the precision, recall, and F1-score were significantly improved by around 70 percentage points for each classifier.

Validation of ICON model configurations optimized over a limited domain is essential before accepting new semi-empirical parameters that influence the behavior of subgrid-scale schemes. Because such parameters can modify the dynamics of a numerical weather prediction (NWP) model in highly nonlinear ways, we analyze one season of forecasts (December 2022, January and February 2023) generated with the NWP ICON-LAM through the lens of machine learning–based diagnostics as a complement to traditional evaluation metrics. The goal is to extract physically interpretable information on the model behavior induced by the optimized parameters. This work represents the first part of a wider study exploring machine learning tools for model validation, focusing on two specific approaches: Empirical Orthogonal Functions (EOFs), which are widely used in meteorology and climate science, and autoencoders, which are increasingly adopted for their nonlinear feature extraction capability. In this first part, EOF analysis is used as the primary tool to decompose weather fields from observed reanalysis and forecast datasets. Hourly 2-m temperature forecasts for winter 2022–2023 from multiple regional ICON configurations are compared against downscaled ERA5 data and in situ observations from ground station. EOF analyses revealed that the optimized configurations demonstrate a high skill in predicting surface temperature. From the signal error decomposition, the fourth EOF mode is effective particularly during night-time hours, and contributes to enhancing the performance of ICON. Analyses based on autoencoders will be presented in a companion paper (Part II).

A high‐quality simulation model should help its users to easily and appropriately calibrate their trust in the model. Traditional evaluation metrics such as validation and robustness are necessary but insufficient for this task. Trust calibration depends on factors like the model's transparency, applicability to intended use, usability, reputation, and consideration of potential bias. This article proposes a framework for designing and evaluating system dynamics models by considering factors that contribute to the proper calibration of user trust. This framework takes inspiration from trusted artificial intelligence, broadening our traditional concept of model quality and explicitly focusing on what users need to consider a model trustworthy and to understand the model's relevance to its intended purpose. The trusted simulation framework can improve our integration of model quality activities throughout the modeling process, leading to more impactful and better‐targeted model design, development, and evaluation.

In this paper we introduce a method to automatically generate facial sketches from digital color photographs. First, the image is submitted to a sub-band homomorphic filtering for illumination compensation. Then, the result is fed into a multi-scale edge detection process. The homomorphic filter parameters and the optimal scales for the edge detector are acquired by means of a genetic algorithm optimization. An analysis comparing the results of the proposed method and those ones of a plain Canny edge detector is presented. The analysis has been performed considering manually labeled ground truth facial sketches and using three traditional evaluation metrics: Prattpsilas figure of merit (FoM), peak signal to noise ratio (PSNR) and structural similarity (SSIM). Among all metrics, the Prattpsilas FoM metric was the one with best stability. Parameter optimization using FoM rendered promising results to the proposed method. A subjective evaluation is also provided, which emphasizes the strengths of the proposed method.

In this research, we introduce a groundbreaking methodology that leverages deep learning techniques to revolutionize the process of generating descriptive captions and hashtags for images, effectively bridging the gap between computer vision and natural language understanding. Traditional approaches to image caption generation have often relied on rudimentary techniques such as handcrafted features and rule-based systems, which inherently struggle to capture the intricate semantics of images and adapt to diverse datasets. Recognizing these limitations, our novel framework integrates state-of-the-art convolutional neural networks (CNNs) for precise image feature extraction and recurrent neural networks (RNNs), specifically employing the ResNet-50 architecture, for seamless sequence generation. Furthermore, in addition to traditional evaluation metrics such as BLEU scores and human assessments, our system, DeepLens, introduces a groundbreaking feature: automatic hashtag generation. By meticulously analyzing both the content and context of images, DeepLens autonomously generates hashtags that enrich social media content sharing and engagement, presenting a novel paradigm in the realm of image captioning. In addition to its advancements in captioning accuracy and user experience enhancement, DeepLens offers scalability and adaptability to various domains. Its robust architecture allows for seamless integration with different datasets and environments, making it versatile for a wide range of applications. Through this comprehensive approach, we aim to not only enhance the accuracy and relevance of generated captions but also elevate the overall user experience in navigating and interacting with visual content across various platforms and applications.

Fluid Dynamics is a key scientific field to multitudes of engineering applications. Experimental work in this field requires careful set-up and expensive image-capturing equipment, particularly when considering the finer details of complex phenomena. In this work, we study the application of super-resolution Generative Adversarial Networks (GANs) to achieve high-resolution results by upscaling lower-resolution experimental images.We train GANs proposed for natural images on a bubbly flow experimental Fluid Dynamics dataset and compare common super-resolution evaluation metrics to domain expert assessments of the upscaled images. We find that these models achieve promising results, as evaluated by experts, and transfer learning from natural images translates to better performance overall. Attention mechanisms are found to be particularly useful in recreating sharper details. On the other hand, traditional super-resolution evaluation metrics are found to align poorly with expert perception of quality, signaling the need for better systematic evaluation methodologies in this domain.