Quality control of crowdsourced classification using hierarchical class structures

Skills gained from vocational or higher education form an essential component of country’s economy, determining the structure of the national labor force. Therefore, knowledge on how people’s education converts to jobs enables data-driven choices concerning human resources within an ever-changing job market. Moreover, the relationship between education and occupation is also relevant in times of global crises, such as the COVID-19 pandemic. Healthcare system overload and skill shortage on one hand, and job losses related to lock-downs on the other, have exposed a necessity to identify target groups with relevant education backgrounds in order to facilitate their occupational transitions. However, the relationship between education and employment is complex and difficult to model. This study aims to propose the methodology that would allow us to model education-to-skill mapping. Multiple challenges arising from administrative datasets, namely imbalanced data, complex labeling, hierarchical structure and textual data, were addressed using six neural network-based algorithms of incremental complexity. The final proposed mathematical model incorporates the textual data from descriptions of education programs that are transformed into embeddings, utilizing transformer neural networks. The output of the final model is constructed as the hierarchical classification task. The effectiveness of the proposed model is demonstrated using experiments on national level data, which covers whole population of Lithuania. Finally, we provide the recommendations for the usage of proposed model. This model can be used for practical applications and scenario forecasting. Some possible applications for such model usage are demonstrated and described in this article. The code for this research has been made available on GitHub.

Hierarchical multi-label text classification (HMTC) has become rather challenging when it requires handling large sets of closely related categories. We present a novel software that provides a hybrid embedding-based text representation for HMTC, shortened as HE-HMTC. It made full use of categories’ structure and their labels semantics to enrich the text representation, and therefore attempted to improve the classification performance of the text. In addition, HE-HMTC facilitates improving the accuracy of HMTC tasks. Moreover, our HE-HMTC can easily be generalized in other hierarchical classification tasks and achieve superior performance.

The goal of electronic monitoring (EM) of longline fishing is to monitor the fish catching activities on fishing vessels, either for the regulatory compliance or catch counting. Hierarchical classification based on videos allows for inexpensive and efficient fish species identification of catches from longline fishing, where fishes are under severe deformation and self-occlusion during the catching process. More importantly, the flexibility of hierarchical classification mitigates the laborious efforts of human reviews by providing confidence scores in different hierarchical levels. Some related works either use cascaded models for hierarchical classification or make predictions per image or predict one overlapping hierarchical data structure of the dataset in advance. However, with a known non-overlapping hierarchical data structure provided by fisheries scientists, our method enforces the hierarchical data structure and introduces an efficient training and inference strategy for video-based fisheries data. Our experiments show that the proposed method outperforms the classic flat classification system significantly and our ablation study justifies our contributions in CNN model design, training strategy, and the video-based inference schemes for the hierarchical fish species classification task.

Due to the complex label hierarchy and intensive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor performance especially when low-resource or few-shot settings are considered. Recently, there is a growing trend of applying prompts on pre-trained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks. However, limited work has studied the paradigm of prompt-based learning in the HTC problem when the training data is extremely scarce. In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further explore few-shot HTC tasks. To address the issue, we propose the hierarchical verbalizer (“HierVerb”), a multi-verbalizer framework treating HTC as a single- or multi-label classification problem at multiple layers and learning vectors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning. In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably outperforms those who inject hierarchy through graph encoders, maximizing the benefits of PLMs. Extensive experiments on three popular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb significantly boosts the HTC performance, meanwhile indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks.

Hierarchical Classification is a very important classification task for arranging data in a hierarchical structure. Hierarchical arrangement of data is one of the best methods to achieve better understanding of complex data. In this paper, we propose the HMAC method to perform Hierarchical Multi-label Associative Classification. This method uses multiple and negative rules to predict class-set and to filter exceptional cases in order to improve both accuracy and explanatory ability of the resulting classifier. Redundant rule pruning method for negative rules and rule ranking method for hierarchical associative classification are developed. Moreover, we propose a rule evaluation measure, Sim, that tread-off between F-measure and Jaccard's coefficient to encode hierarchical structure meaning of actual and predicted node collections. We show that Sim is robust and simple rule evaluation measure. Experimental results show that HMAC improves accuracy and explanatory ability of hierarchical classifiers compared with various rule ranking and pruning techniques.

Fine-Grained Visual Categorization (FGVC) is challenging due to the large intra-class and small inter-class variance, and classes which are prone to be confused with one or more similar classes are called as confusing classes. Hierarchical visual data structures are widely used for image classification with a large number of classes so that the sophisticated classification problem can be processed from coarse to fine. However, hierarchical structures are accompanied with error propagation. In order to reduce the error propagation caused by the hierarchical structure, we propose the concept of minimal confusion set and its metrics. Minimal confusion set is the minimal set of classes that are highly confused with other classes in the set and much less confused with classes outside the set. Based on minimal confusion sets, a flat two-layer hierarchical structure can be formed, which reduces the error propagation. An effective mining method of minimal confusion set is also proposed, which can also be used for hierarchical learning. The mining method of minimal confusion set is based on confusion matrix, but it is different from the traditional methods using the confusion matrix obtained from test data. This method is independent on test data, and experiments on multiple datasets demonstrate the effectiveness of the method.

Hierarchical GAN-Tree and Bi-Directional Capsules for multi-label image classification

The purpose of this study was to develop eye-movement based classifying thinking model by identifying the differences in eye movement pattern shown in classification process between the elementary students who have analytic cognitive style and wholistic cognitive style. The subjects for the research consisted of 25 analytic students and 13 wholistic students. First, the participants were asked to solve the eye-tracking problem. After that, they gave an interview about the eye-tracking problem. Finally, they answered CSA test. The findings of this study were fourfold. First, 5 tasks were developed for revealed differences in classification process between students` cognitive style. The tasks were consisted of 4 non-hierarchical classification tasks and 1 hierarchical classification task. Second, there was a signi-ficant difference in students` fixation duration by students` cognitive style. Analytic students spent little time on sca-nning details and found the features successfully. They made different classification standard according to their cognitive style. Analytic students interested in features` detail, while wholistic students interested in salient features. Third, in-terviewing revealed the differences in the process of making standards. Finally, ECTM was developed. There were two steps in this model- Cognition of Elements and Making Stan-dards. This study developed eye-movement based classifying model by analyzing the differences in classifying process between analytic students and wholistic students. It suggested that classification should be taught differently by students` cognitive style and it should be taught following stage.

Question classification is an important phase in question answering systems. In this paper, we propose to apply i) hierarchical classifiers, ii) hierarchical classifiers in combination with semi-supervised learning and iii) hierarchy expansion for question classification for improving the precision. When the number of classes is large, the performance of classification algorithms may be affected. In order to improve the performance by reducing the number of classes for each classifier, we propose to use hierarchical classifiers according to the question taxonomy, in which each internal node is attached a classifier. We try to use semi-supervised learning to consume unlabeled questions with expectation to improve the performance of classifiers in the hierarchy. We explored different applications of learning methods in for each classifier of the hierarchy: a) supervised learning for all classifiers at all levels; b) semi-supervised learning for the first-level classifier and supervised learning for other classifiers; c) semi-supervised learning for all classifiers. The experiments show that the first method (a) has better results than those of flat classification; the second method (b) produces better results than those of the first method while the effort to increase the performance of fine classifiers in the last method (c) is not so successful. As another effort, we propose to automatically group question classes by clustering in order to expand a node which has a large number of classes in the question taxonomy. The experiment also shows that the overall precision is improved.

In this paper we address the task of hierarchical bird species identification from audio recordings. We evaluate three types of approaches to deal with hierarchical classification problems: the flat classification approach, the local-model per parent node classifier approach and the global-model hierarchical-classification approach. For the flat and local-model classification approach we employ the classic Naive Bayes algorithm. For the global-model approach we use the Global Model Naive Bayes (GMNB) algorithm. As in the classical Naive Bayes, the algorithm computes prior probabilities and likelihoods, but these computations take into account the hierarchical classification scenario: it assumes that any example which belongs to a given class will also belong to all its ancestor classes. In the current application, the employed class hierarchy is the standard scientific taxonomy of birds used in Biology. In order to deal with the bird songs we obtain features by computing several acoustic quantities from intervals of the audio signal. We conduct three experiments in order to compare the three different approaches to the hierarchical bird species identification problem. Our experimental results show that the use of the GMNB hierarchical classification algorithm outperforms both the flat and local-model approaches (Using the Hierarchical F-measure metric), hence the use of a global-model approach (such as the GMNB) can be a feasible way to improve the classification performance for problems with a large number of classes.

Hierarchy is an effective and common way of organizing data and representing their relationships at different levels of abstraction. However, hierarchical data dependencies cause difficulties in the estimation of "separable" models that can distinguish between the entities in the hierarchy. Extracting separable models of hierarchical entities requires us to take their relative position into account and to consider the different types of dependencies in the hierarchy. In this paper, we present an investigation of the effect of separability in text-based entity classification and argue that in hierarchical classification, a separation property should be established between entities not only in the same layer, but also in different layers. Our main findings are the followings. First, we analyse the importance of separability on the data representation in the task of classification and based on that, we introduce "Strong Separation Principle" for optimizing expected effectiveness of classifiers decision based on separation property. Second, we present Significant Words Language Models (SWLM) which capture all, and only, the essential features of hierarchical entities according to their relative position in the hierarchy resulting in horizontally and vertically separable models. Third, we validate our claims on real world data and demonstrate that how SWLM improves the accuracy of classification and how it provides transferable models over time. Although discussions in this paper focus on the classification problem, the models are applicable to any information access tasks on data that has, or can be mapped to, a hierarchical structure.

Keyword extraction of complex hierarchical categories becomes a challenge in text mining since commonly used classification for flat categories results in low accuracy. This paper presents a method to improve keyword extraction from hierarchical categories considering terms occurred in category from a hierarchy as additional factors in term-weighting. The method is an enhancement of a basic TF-IDF calculation; thus, it can comfortably be used for keyword extraction and classification. By taking term frequency and inverse document frequency of categories hierarchically related to a focused category, we can determine how important terms are in their family categories. In this work, hierarchy relations used in calculation are sub-categories, supercategories and sibling-categories. From experiment results, we found that the proposed method gained higher accuracy for about 40% from a baseline in a classification task.

Traditional classification tasks deal with assigning instances to a single label. However, some real world databases classes are structured in a hierarchy and its instances can have their classes associated with two or more paths in the hierarchical structure. In this case, such situations are referred as hierarchical multi-label classification problems. The purpose of this paper is to explore the concept of hierarchical multi-label classification problems and present a solution based on Learning Classifier Systems (LCS) to solve this kind of problem. The Hierarchical Learning Classifier System Multi-label (HLCS-Multi) proposed, presents a comprehensive solution to hierarchical multi-label classification problems building a global classifier to predict all classes in the application domain.KeywordsHierarchical Multi-label Classification ProblemsLearning Classifier SystemsProtein Function

Analyzing the performance improvement of hierarchical binary classifiers using ACO through Monte Carlo simulation and multiclass engine vibration data

Constitute is a web application developed for the analysis of the text of written constitutions. At the heart of the application is a set of constitutional excerpts that are encoded with tags derived from a conceptual inventory used in the Comparative Constitutions Project (CCP), a related dataset that records the characteristics of national constitutions since 1789. The data collection deposited with QDR is a complete set of excerpts, together with topical tags, from in-force constitutions, as well as a few draft and historical constitutions, appearing on the site as of December 26, 2017. Data Abstract The data include the constitutions of nearly all independent states, as identified by Ward and Gledistch, in force as of December 26, 2017. Constitute’s conceptual organization was informed by its intended use as a public-facing data product. Since Constitute is intended to be usable for members of the public as well as real-life constitution-writers and academics, the project’s texts, content tags, and conceptual inventory are all organized with an eye towards readability and parsimony. This guiding principle is most noticeable in the dataset’s conceptual inventory, which covers roughly half as many items as the original Comparative Constitutions Project (CCP) variable set. Constitute’s hierarchical variable organization — enabled by the Semantic Web technology underlying the project — is informed by similar concerns. To enhance usability (both visually on the project’s website and in downstream statistical applications), Constitute’s conceptual hierarchy is oriented in a shallow but wide fashion, with few hierarchical layers and many sub-categories at each layer. In compiling data for QDR, we included all information which is available on the Constitute website at the time of submission. These data consist of Semantic Web (.nt) files that include excerpts from nearly all in-force constitutions as of 2017, with accompanying content tags. The data follow a conceptual framework developed by the Principal Investigators in the Comparative Constitutions Project (CCP), a related dataset that records some 600 characteristics of national constitutions enacted since 1789. Documentation and data for that project, including data and texts for older constitutions, are available on the project website, at https://comparativeconstitutionsproject.org . Data Model The textual excerpts are derived from the CCP’s core dataset, which records a comprehensive set of characteristics of historical constitutions. The Constitute dataset consists of two main pieces: (1) a set of conceptual tags and a related conceptual ontology; and (2) a collection of cleaned and tagged constitutional texts. Constitute tracks some 300 unique conceptual tags at the paragraph level of each individual constitution, which are organized into a hierarchical, extensible ontology. Example tags might include “Free Expression,” “Head of State Powers,” or “National Motto.” The Constitute conceptual inventory is available in QDR as an OWL file, a graph-based Semantic Web format that allows for formal descriptions of hierarchical classification schemes. In accordance with the general Semantic Web framework, OWL files can be easily expanded to include new categories and relationships, allowing researchers to extend the Constitute coding scheme for their own purposes. Each constitutional text in Constitute is represented in a related hierarchical format. Currently, the QDR sample consists of nearly all in-force national constitutions as of 2017, as well as a few draft and historical texts. Files Description The data collection deposited with QDR is a complete set of excerpts -- together with topical tags – as they appeared on the site as of December 26, 2017. Metadata about each constitution is also provided (as an .nt file), along with the project’s conceptual inventory (as an .owl file) and copyright information (as a spreadsheet).