MoOnEv: Modular Ontology Evaluation and Validation tool

Many ontologies have been developed recently, and evaluation is important for researchers or users when searching ontologies needed in their work. Ontology evaluation also is important phase of ontology development, refinement or evolution process. Automatically-developed ontologies as a result of ontology learning also need from evaluation. So, ontology evaluation is important both for selecting ontologies, meeting some requirements, and as a part of ontology life cycle. In this paper we propose deep analysis of ontology evaluation and discuss how multilingualism affects ontology evaluation. We classify ontology evaluation approaches, methods and metrics according to several dimensions. Our analysis and conclusions will be helpful for development of good ontologies and for finding or selection of needed ontologies.

Ontology helps semantic systems to establish a model of the world in order to derive machine processed knowledge. It represents formally the concepts, objects, properties and establishes the relationship between them. By providing the structure and semantics for data representation ontology has opened an avenue for appropriate representation of the world for specific and explicit sharing of knowledge without ambiguities. The use of ontology in various applications and the availability of ontology on the web have increased. Ontology should be evaluated from various perspectives to determine the quality of ontology before use or reuse. Ontology evaluation is one of the key phases in ontology development. The ontology should be accurate and provide the accurate results to a user. Unless the quality of the ontology is assured the accuracy of the results cannot be guaranteed. Metrics is used to measure the quality of ontology from different dimensions. This paper explores various metrics used for ontology evaluation and investigates the underlying mechanisms.

Biomedical knowledge graphs (KGs), which can help with the understanding of complex biological systems and pathologies, have begun to play a critical role in medical practice and research. However, challenges remain in their embedding and use due to their complex nature and the specific demands of their construction. Existing studies often suffer from problems such as sparse and noisy datasets, insufficient modeling methods and non-uniform evaluation metrics. In this work, we established a comprehensive KG system for the biomedical field in an attempt to bridge the gap. Here, we introduced PharmKG, a multi-relational, attributed biomedical KG, composed of more than 500000 individual interconnections between genes, drugs and diseases, with 29 relation types over a vocabulary of ~8000 disambiguated entities. Each entity in PharmKG is attached with heterogeneous, domain-specific information obtained from multi-omics data, i.e. gene expression, chemical structure and disease word embedding, while preserving the semantic and biomedical features. For baselines, we offered nine state-of-the-art KG embedding (KGE) approaches and a new biological, intuitive, graph neural network-based KGE method that uses a combination of both global network structure and heterogeneous domain features. Based on the proposed benchmark, we conducted extensive experiments to assess these KGE models using multiple evaluation metrics. Finally, we discussed our observations across various downstream biological tasks and provide insights and guidelines for how to use a KG in biomedicine. We hope that the unprecedented quality and diversity of PharmKG will lead to advances in biomedical KG construction, embedding and application.

BackgroundDe novo assembly of non-model organism’s transcriptomes has recently been on the rise in concert with the number of de novo transcriptome assembly software programs. There is a knowledge gap as to what assembler software or k-mer strategy is best for construction of an optimal de novo assembly. Additionally, there is a lack of consensus on which evaluation metrics should be used to assess the quality of de novo transcriptome assemblies.ResultSix different assembly strategies were evaluated from four different assemblers. The Trinity assembly was used in its default 25 single k-mer value while Bridger, Oases, and SOAPdenovo-Trans were performed with multiple k-mer strategies. Bridger, Oases, and SOAPdenovo-Trans used a small multiple k-mer (SMK) strategy consisting of the k-mer lengths of 21, 25, 27, 29, 31, and 33. Additionally, Oases and SOAPdenovo-Trans were performed using a large multiple k-mer (LMK) strategy consisting of k-mer lengths of 25, 35, 45, 55, 65, 75, and 85. Eleven metrics were used to evaluate each assembly strategy including three genome related evaluation metrics (contig number, N50 length, Contigs >1 kb, reads) and eight transcriptome evaluation metrics (mapped back to transcripts (RMBT), number of full length transcripts, number of open reading frames, Detonate RSEM-EVAL score, and percent alignment to the southern platyfish, Amazon molly, BUSCO and CEGMA databases). The assembly strategy that performed the best, that is it was within the top three of each evaluation metric, was the Bridger assembly (10 of 11) followed by the Oases SMK assembly (8 of 11), the Oases LMK assembly (6 of 11), the Trinity assembly (4 of 11), the SOAP LMK assembly (4 of 11), and the SOAP SMK assembly (3 of 11).ConclusionThis study provides an in-depth multi k-mer strategy investigation concluding that the assembler itself had a greater impact than k-mer size regardless of the strategy employed. Additionally, the comprehensive performance transcriptome evaluation metrics utilized in this study identified the need for choosing metrics centered on user defined research goals. Based on the evaluation metrics performed, the Bridger assembly was able to construct the best assembly of the testis transcriptome in Fundulus heteroclitus.

Story visualization is an underexplored task that falls at the intersection of many important research directions in both computer vision and natural language processing. In this task, given a series of natural language captions which compose a story, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform text-to-image synthesis models on this task. However, there is room for improvement of generated images in terms of visual quality, coherence and relevance. We present a number of improvements to prior modeling approaches, including (1) the addition of a dual learning framework that utilizes video captioning to reinforce the semantic alignment between the story and generated images, (2) a copy-transform mechanism for sequentially-consistent story visualization, and (3) MART-based transformers to model complex interactions between frames. We present ablation studies to demonstrate the effect of each of these techniques on the generative power of the model for both individual images as well as the entire narrative. Furthermore, due to the complexity and generative nature of the task, standard evaluation metrics do not accurately reflect performance. Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images. We also present correlation experiments of our proposed automated metrics with human evaluations.

Machine-readable semantic knowledge in the form of taxonomies (i.e., a collection of is-a edges) has proved to be beneficial in an array of NLP tasks including inference, textual entailment, question answering and information extraction. Such widespread utility of taxonomies has led to multiple large-scale manual efforts towards taxonomy induction such as WordNet and Cyc. However, manual construction of taxonomies is time-intensive, and usually, requires substantial annotation efforts by domain experts. Furthermore, the resulting taxonomies suffer from low coverage and are unavailable for specific domains or languages. Therefore, in recent years, there has been a growing body of work, which aims to induce taxonomies automatically, either from semi-structured knowledge resources or unstructured text. In this thesis, we focus on the task of automated taxonomy induction under a variety of different settings. We first focus on taxonomy induction from the largest semi-structured knowledge resource, i.e., Wikipedia. More specifically, we introduce a set of novel heuristics aimed towards inducing a large-scale taxonomy from the English Wikipedia categories network. We also propose a novel comprehensive path-based evaluation framework for taxonomies. Taxonomy induced using our approach significantly outperforms the state of the art across both edge-based as well as path-based evaluation metrics. Moreover, our experiments also demonstrate that high accuracy in edge-based evaluation metrics does not always translate to high-accuracy in path-based evaluation metrics. Subsequently, we propose a novel approach, which leverages the interlanguage links of Wikipedia to induce taxonomies in other Wikipedia languages. Compared to the state of the art, our approach is simpler, more principled, and results in taxonomies that are significantly more accurate across both edge-based and path-based evaluation metrics. In the second part of this thesis, we focus on taxonomy induction from unstructured text. We propose a novel approach towards taxonomy induction from an input vocabulary of seed terms. Unlike previous approaches, which extract singular hypernym edges for terms, our approach utilizes a novel probabilistic framework to extract long-range hypernym subsequences. Taxonomy induction from the extracted subsequences is cast as an instance of the minimum-cost flow optimization problem on a carefully-designed flow network. Through experiments, we demonstrate that our approach outperforms the state of the art across four languages. We also show that our approach is robust to the presence of noise in the input vocabulary. Our approach facilitates the relaxation of many simplifying assumptions, which were employed by previous taxonomy induction approaches. As a result, our work serves to automate the process of taxonomy induction from unstructured text in the true sense. Finally, we introduce a task of discovering and generalizing lexicalized templates from the titles of Wikipedia entities. The experimental results on this task demonstrate that taxonomies, which perform better on our proposed path-based metrics, result in a more accurate set of generalizations for a given set of entities. In summary, this thesis proposes new approaches towards automated taxonomy induction. It improves upon the state of the art in a variety of different settings. It also relaxes many simplifying assumptions that limited the applicability of prior approaches.

Objective and fair scholar evaluation promotes academic engagement and ensures proper allocation of research resources. However, current evaluation methods face challenges such as insufficient consideration of disciplinary differences, over-reliance on publication output, neglect of low-cited papers, and limited discriminative power. This paper proposes an improved evaluation metric—the Academic Equilibrium Value index (AEV-index)—based on academic equilibrium value. By incorporating weights for publication and citation tiers and integrating average publication and citation levels within disciplines, the AEV-index aims to offer a more comprehensive and equitable assessment of scholarly performance. Using Scopus data from computer science conference papers (2012-2021) across the subfields of "Network and Information Security", "Computer Graphics and Multimedia", and "Artificial Intelligence", we weighted publications and citations to evaluate researchers' academic levels. The research findings indicate that: (i) The AEV-index enhances existing scholar evaluation metrics from multiple perspectives, providing a relatively reasonable and broadly applicable comprehensive evaluation index. (ii) The AEV-index is significantly correlated with traditional scholar evaluation indicators such as total publications, total citations, and the h-index, while also moderating disciplinary differences, thereby facilitating cross-disciplinary comparisons of academic influence. (iii) Additionally, the AEV-index mitigates the negative impact of low publication output, recognizes the value of low-cited but high-quality papers, and offers improved discriminative power, thereby addressing several limitations of existing evaluation metrics.

Cloud computing has emerged as the spine of cutting-edge digital infrastructure, necessitating efficient workload prediction to optimise useful resource allocation, reduce prices, and enhance machine performance. Traditional workload forecasting techniques, including statistical models, frequently fail to seize the complexity and dynamic nature of cloud environments. Machine learning (ML) and deep master (DL) strategies, especially long-term short-term memory (LSTM) networks, provide great improvements by using historical workload statistics to expect future demands with better accuracy. This research proposes a smart workload prediction machine integrating ML/DL models with real-time statistics processing to beautify cloud scalability and performance. The take a look at explores diverse prediction procedures, evaluates their overall performance, and proposes a hybrid version combining statistical and deep mastering techniques. Key evaluation metrics including trunk middle square errors (MSE) and square errors (RMSE). The findings show that ML-pushed workload prediction complements resource control, decreasing beneath-provisioning and over-provisioning risks, in the end, main to greater sustainable cloud computing environments.

Self-Admitted Technical Debt (SATD), cases where developers intentionally acknowledge suboptimal solutions in code through comments, poses a significant challenge to software maintainability. Left unresolved, SATD can degrade code quality, increase maintenance costs, and hinder long-term software sustainability. Manually identifying and resolving SATD is time-consuming and error-prone, often competing with feature development and release deadlines. Automating SATD repayment is therefore essential to alleviate developer burden. While Large Language Models (LLMs) have shown promise in tasks like code generation and program repair, their potential in automated SATD repayment remains underexplored. In this paper, we identify three key technical challenges in training and evaluating LLMs for SATD repayment: (1) dataset representativeness and scalability, (2) removal of irrelevant SATD repayment samples, and (3) limitations of existing evaluation metrics, i.e., BLEU and CrystalBLEU. To address the first two dataset-related challenges, we adopt a language-independent SATD tracing tool and design a 10-step filtering pipeline to extract SATD repayments from repository commit histories, leveraging LLM-as-judge for relevance filtering. This results in two new large-scale SATD repayment datasets: 58,722 items for Python and 97,347 items for Java. To improve evaluation, we introduce two diff-based metrics, BLEU-diff and CrystalBLEU-diff, which measure code changes rather than whole code, thereby mitigating the impact of SATD-irrelevant source code. Additionally, we propose a new metric, Line-Level Exact Match on Diff (LEMOD), which is both interpretable and informative, providing fine-grained insights into SATD repayment quality. Using our new benchmarks and evaluation metrics, we evaluate two types of automated SATD repayment methods: fine-tuning smaller models (a few hundred million parameters) and prompt engineering with large-scale models, including four SOTA open-source LLMs and GPT-4o-mini. Our results reveal that while fine-tuned smaller models achieve Exact Match (EM) scores comparable to prompt-based approaches, they underperform on BLEU-based metrics and LEMOD. The best EM performance is achieved by Gemma-2-9B, correctly addressing 10.1% of Python SATDs and 8.1% of Java SATDs with a simple prompt. The only previous study in this domain focused solely on Java, achieving an EM score of 2.3%. When using BLEU-diff, CrystalBLEU-diff, and LEMOD metrics, Llama-3.1-70B-Instruct and GPT-4o-mini deliver the highest performance. While our three proposed metrics demonstrate strong correlations with EM (0.65~0.84), both BLEU and CrystalBLEU show almost no correlation with this intuitive metric (0.01~0.08), highlighting their largely unpredictable behavior when applied directly to the entire code rather than the diff. Our work contributes a robust benchmark, improved evaluation metrics, and a comprehensive evaluation of LLMs, advancing research on automated SATD repayment.

The output brightness temporal response of non-gated image intensifying systems was examined. Generation III gallium arsenide image intensifiers (image tubes) using the P43 and P20 phosphors were tested for image persistence, minimum event detection, and minimum event temporal separation (multiple burst testing, where "event" refers to a burst of light of determined duration and brightness). Persistence tests showed some correlation with previously published data and suggest that both phosphors (P43 and P20) display multiple stages of decay. The minimum event detection tests showed the Generation III Image Intensifier tubes capable of detecting pulses with durations in the low microsecond region and suggested that for faster test setups the tubes may have the capability to operate in the nanosecond region. Finally, the multiple burst testing shows the complex temporal nature of image tube power supplies when encountering multiple, rapid flashes of light.

The development of machine learning models for symptom-based health checkers is a rapidly evolving area with significant implications for healthcare. Accurate and efficient diagnostic tools can enhance patient outcomes and optimize healthcare resources. This study focuses on evaluating and optimizing machine learning models using a dataset of 10 diseases and 9,572 samples. The dataset was divided into training and testing sets to facilitate model training and evaluation. The following models were selected and optimized: Decision Tree, Random Forest, Naive Bayes, Logistic Regression and K-Nearest Neighbors. Evaluation metrics included accuracy, F1 scores, and 10-fold cross-validation. ROC-AUC and precision-recall curves were also utilized to assess model performance, particularly in scenarios with imbalanced datasets. Clinical vignettes were employed to gauge the real-world applicability of the models. The performance of the models was evaluated using accuracy, F1 scores, and 10-fold cross-validation. The use of ROC-AUC curves revealed that model performance improved with increasing complexity. Precision-recall curves were particularly useful in evaluating model sensitivity in imbalanced dataset scenarios. Clinical vignettes demonstrated the robustness of the models in providing accurate diagnoses. The study underscores the importance of comprehensive model evaluation techniques. The use of clinical vignette testing and analysis of ROC-AUC and precision-recall curves are crucial in ensuring the reliability and sensitivity of symptom-based health checkers. These techniques provide a more nuanced understanding of model performance and highlight areas for further improvement. This study highlights the significance of employing diverse evaluation metrics and methods to ensure the robustness and accuracy of machine learning models in symptom-based health checkers. The integration of clinical vignettes and the analysis of ROC-AUC and precision-recall curves are essential steps in developing reliable and sensitive diagnostic tools.

Federated Learning (FL) enables collaborative training among multiple data owners without direct sharing of users’ local data, which not only protects user privacy but also enhances model performance. However, due to the invisibility of local model training processes to the aggregation server, FL is vulnerable to Byzantine attacks and backdoor attacks initiated by adversaries. Backdoor attacks are highly covert and difficult to detect, thus posing significant threats to model security. Current backdoor attack defense mechanisms frequently fail to adequately account for the sophisticated capabilities of adversaries and the heterogeneous nature of data distributions encountered in practical deployment environments. Conventional defense approaches relying on singular evaluation metrics exhibit notable limitations when confronting highly self-adaptive adversarial attacks, with their defensive efficacy being particularly compromised under non-independent and identically distributed (Non-IID) data conditions. To overcome these limitations, we present a novel multi-metric cascading defense framework against backdoor attacks in federated learning systems. The proposed methodology systematically integrates five complementary yet mutually constraining evaluation metrics that collectively capture different aspects of model behavior. Through layer-wise feature extraction from distributed local models and adaptive threshold determination via rigorous statistical hypothesis testing, our framework achieves robust detection of both static and adaptive backdoor attacks. Comprehensive experimental evaluations confirm that the proposed solution maintains detection efficacy against sophisticated adversarial strategies while preserving the baseline accuracy of the global model.

Towards robust deep learning-based autosegmentation in MRI-planned gynecological brachytherapy: Importance of scalable development and comprehensive evaluation.

Single-view 3D object reconstruction (SVOR) aims to recover the 3D shape of an object from a single 2D image. Despite advances in deep learning (DL), challenges such as incomplete image information, scarce 3D data annotation, and highly variable object shapes still limit the performance of SVOR. Meanwhile, with the rapid development of novel view synthesis (NVS) techniques, the SVOR field has received significant advancements. However, existing reviews have not comprehensively covered the rapid developments in NVS-based approaches. This article aims to fill this gap by highlighting the latest progress in SVOR, particularly advancements related to NVS-based methods. Additionally, we observed discrepancies between existing quality evaluation metrics in SVOR and human visual perception. This is because some critical object parts are essential to consider during the evaluation. For example, when reconstructing airplanes, critical parts like the empennage and wings are often overlooked in evaluation metrics due to their smaller size compared to the fuselage. Consequently, poor reconstruction of these parts may not significantly affect overall evaluation scores. To address this issue, we propose a more comprehensive evaluation method that reflects human visual perception accurately. To achieve this, we introduce a weighted evaluation method that considers part saliency and proposes a novel technique for automatically perceiving reconstruction discrepancies. This study effectively enhances the accuracy and consistency of evaluations through these approaches, offering new insights and methodologies, filling a void in the existing literature, and providing valuable contributions to both research and practical applications in SVOR.

Despite the impressive advances in text-to-image models, they often struggle to effectively compose complex scenes with multiple objects, displaying various attributes and relationships. To address this challenge, we present T2I-CompBench++, an enhanced benchmark for compositional text-to-image generation. T2I-CompBench++ comprises 8,000 compositional text prompts categorized into four primary groups: attribute binding, object relationships, generative numeracy, and complex compositions. These are further divided into eight sub-categories, including newly introduced ones like 3D-spatial relationships and numeracy. In addition to the benchmark, we propose enhanced evaluation metrics designed to assess these diverse compositional challenges. These include a detection-based metric tailored for evaluating 3D-spatial relationships and numeracy, and an analysis leveraging Multimodal Large Language Models (MLLMs), i.e. GPT-4 V, ShareGPT4v as evaluation metrics. Our experiments benchmark 11 text-to-image models, including state-of-the-art models, such as FLUX.1, SD3, DALLE-3, Pixart-$\alpha$α, and SD-XL on T2I-CompBench++. We also conduct comprehensive evaluations to validate the effectiveness of our metrics and explore the potential and limitations of MLLMs.