Ontology Evaluation and Multilingualism

Since more than a decade, theoretical research on ontology evolution has been published in literature and several frameworks for managing ontology changes have been proposed. However, there are less studies that analyze widely used ontologies that were developed in a collaborative manner to understand community-driven ontology evolution in practice. In this paper, we perform an empirical analysis on how four well-known ontologies (DBpedia, Schema.org, PROV-O, and FOAF) have evolved through their lifetime and an analysis of the data quality issues caused by some of the ontology changes. To that end, the paper discusses the composition of the communities that developed the aforementioned ontologies and the ontology development process followed. Further, the paper analyses the changes in those ontologies in the 53 versions of them examined in this study. Depending of the use case, the community involved, and other factors different approaches for the ontology development and evolution process are used (e.g., bottom-up approach with high automation or top-down approach with a lot of manual curation). This paper concludes that one model for managing changes does not fit all. Furthermore, it is also clear that none of the selected ontologies follow the theoretical frameworks found in literature. Nevertheless, in communities where industrial participants are dominant more rigorous editorial processes are followed, largely influenced by software development tools and processes. Based on the analysis, the most common quality problems caused by ontology changes include the use of abandoned classes and properties in data and introduction of duplicate classes and properties.

Ontology helps semantic systems to establish a model of the world in order to derive machine processed knowledge. It represents formally the concepts, objects, properties and establishes the relationship between them. By providing the structure and semantics for data representation ontology has opened an avenue for appropriate representation of the world for specific and explicit sharing of knowledge without ambiguities. The use of ontology in various applications and the availability of ontology on the web have increased. Ontology should be evaluated from various perspectives to determine the quality of ontology before use or reuse. Ontology evaluation is one of the key phases in ontology development. The ontology should be accurate and provide the accurate results to a user. Unless the quality of the ontology is assured the accuracy of the results cannot be guaranteed. Metrics is used to measure the quality of ontology from different dimensions. This paper explores various metrics used for ontology evaluation and investigates the underlying mechanisms.

Development and evaluation of an ontology for guiding appropriate antibiotic prescribing

Trying out or updating an existing learning ontology and providing evaluation tools to assess its quality are essential steps before putting an e-learning system online. Ontology evaluation is a crucial task and it is usually the output of an automatic process. This paper proposes an ontology-driven model, called Ontology-Driven model for the Evaluation of Learning Ontologies (ODELO), for the evaluation of ontologies representing learning resources with respect to syntactic, semantic, pragmatic, social and cohesion metrics. Syntax deals with the formal relations between signs (e.g., words, phrases, sentences) and the production of new ones. Semantics is the study of the relationships between the system of signs and their meanings. Pragmatics is the study of natural language understanding. Social metrics reflect the fact that software agents and ontologies coexist and communicate in communities. Ontology cohesion metrics refer to the degree of relatedness of ontology classes. ODELO is a deductive evaluation model that identifies the elements of ontological quality for learning ontologies. In this paper we propose a framework for assessing the quality of learning ontologies, which constitute the basis for intelligent educational Adaptive Hypermedia (AH) systems. We introduce a new pragmatic quality metric for assessing the quality of learning ontologies.

This study examines various approaches for ontology construction and evaluation in medicinal plant ontology, focusing on the challenges associated with this domain. Thirty review articles on three databases revealed five alternative approaches for ontology generation in herbal medicine: well-structured methodology, customized methodology, semiautomatic approach, socio-technical approach, domain expert techniques, and top-level approach. However, 68% of the research did not include ontology evaluation. Ontology quality was often assessed using competency questions, expert opinion, and automated error detection. The study provides ontology developers with a better understanding of established methodologies for developing and evaluating standard ontologies in this area and identifies gaps in existing research for future studies.

Ontology evaluation is required before using the ontology within applications. Similar with software practice, the purpose of ontology evaluation is to identify the achievement of requirement criteria. Users who require coverage criteria often seeking ontology that contain the terms related to their focused domain knowledge. Users encounter the difficulty to select a suitable ontology from variety of ontology evaluation approaches. Conceptualization of information related to ontology evaluation helps to identify the important component within ontology that helps towards coverage criteria achievement. This work proposes an algorithm to extract ontology documents gained from public ontology repositories like Falcons into its vocabulary parts focused on classes and literals. The algorithm then processes the extracted ontology components with similarity algorithm and later displays the result on the coverage match of ontology with provided terms and the terms that are synonym expanded using WordNet.

Ontology Evaluation: An Example of Delay Analysis

Sak Yan Ontology (SYO) models knowledge derived from Thai tattoos in the design of cultural heritage preservation planning. Ontology Development 101 is a technique of ontology model creation. The aims of this study are to share the performance of ontology development and ontology evaluation. The study is specifically focused on validation from domain experts and automation evaluated using the OOPS! tools (OntOlogy Pitfall Scanner is a tool that helps detect some of the most common pitfalls appearing when developing ontologies). The results obtained from OOPS! show that SYO is devoid of critical errors; however, it does have one critical, three important, and three minor problems. Four of the problems are fixed, whereas the others are continuous. The combination of automatic and human validation methodologies improves the quality of the ontology being modeled. The tools enhance the traditional methodology with quicker, easier, and smaller amounts of subjective analysis. In conclusion, for the reparation movement, solutions for the above problems are suggested.

This paper presents the basis of our approach for evaluation of application ontologies. Adapting an existing task-based evaluation, this approach explains how crowdsourcing, involving application users, can efficiently help in the improvement of an application ontology all along the ontology lifecycle. A real case experiment on an application ontology designed for the semantic annotation of geobusiness user data illustrates the proposal.

Ontologies are key to information retrieval, semantic integration of datasets, and semantic similarity analyses. Evaluating ontologies (especially defining what constitutes a "good" or "better" ontology) is therefore of central importance for the Semantic Web community. Various criteria have been introduced in the literature to evaluate ontologies, and this article classifies them according to their relevance to the design or the implementation phase of ontology development. In addition, the article compiles strategies for ontology evaluation based on ontologies published until 2017 in two outlets: the Semantic Web Journal, and the Journal of Web Semantics. Gaps and opportunities for future research on ontology evaluation are exposed towards the end of the paper.

MoOnEv: Modular Ontology Evaluation and Validation tool

Ontology evaluation poses a number of difficult challenges requiring different evaluation methodologies, particularly for a “dynamic ontology” representing a complex set of concepts and generated by a combination of automatic and semi-automatic methods. We review evaluation methods that focus solely on syntactic (formal) correctness, on the preservation of semantic structure, or on pragmatic utility. We propose two novel methods for dynamic ontology evaluation and describe the use of these methods for evaluating the different taxonomic representations that are generated at different times or with different amounts of expert feedback. The proposed “violation” and “volatility” scores represent an attempt to merge syntactic and semantic considerations. Violation measures the degree of “ontological fit” to a text corpus representative of the domain. Volatility calculates the stability of the methods for ontology generation and extension. Combined, they support estimation of convergence towards a stable representation of the domain. No method of evaluation can avoid making substantive normative assumptions about what constitutes “correct” representation, but rendering those assumptions explicit can help with the decision about which methods are appropriate for selecting amongst a set of available ontologies or for tuning the design of methods used to generate a hierarchically organized representation of a domain.

This study aimed to define the concepts and learning objectives related to the polygons sub-learning area of the 7 - grade mathematics course as ontology-based and to develop a measurement tool based on this ontology within the framework of competency ontology. In this study, we determined and included the learning objectives in the ontology alongside the concepts and prerequisites. The most crucial difference that distinguishes this study from others is that it is objectively oriented. Throughout the study, we followed ontology development processes that met the system's requirements based on the ontology development. In this context, we saw that the ontology developed answered the intended competency questions. We think this study will make very important contributions in terms of bringing a different perspective to measurement and assessment in mathematics education.

In this article, we propose a metric for the automatic evaluation of restricted domain ontologies. The metric is defined in terms of the evaluation of different lexico-syntactic, statistical and semantic approaches. A syntactic approach employed is the use of lexical syntactic patterns, other approaches as grouping by formal concept analysis, similarity, latent semantic analysis and dependence graphs are used as well. These approaches focus on reference corpora to find evidence of the validity of concepts and semantic relationships stored in the target ontology. The proposed evaluation approach is able to provide a score obtained through the metric, which is based on the accuracy measure used for each ontology evaluated. The score is associated in some way with the ontology quality. This score is given with a certain degree of reliability, and it is obtained by comparing the results given against the evaluation of human experts and a baseline.

Digital agility is a critical dynamic capability that is becoming increasingly important in the context of collaborative product development processes (PDPs). This paper aims to address the complexity of today's PDPs by considering various quality aspects including safety, environment, and the entire lifecycle, along with diverse dynamic capabilities such as digital agility and circular economy. The authors employed a semantic web methodology and created an ontology-based knowledge model. The proposed ontology uses Design for X techniques, circular economy, digital agility, and the semantic web under the PDP perspective to increase performance and cooperation between designers and the project team. To validate the ontology, measures for domain ontology evaluation have been used. The paper presents a detailed guide for ontology engineering and evaluation for collaborative smart PDP, which incorporates digital agility as a critical dynamic capability. The proposed ontology can help boost PDP performance and increase customer satisfaction.