Hierarchy-Aware and Label Balanced Model for Hierarchical Text Classification

Due to the complex label hierarchy and intensive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor performance especially when low-resource or few-shot settings are considered. Recently, there is a growing trend of applying prompts on pre-trained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks. However, limited work has studied the paradigm of prompt-based learning in the HTC problem when the training data is extremely scarce. In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further explore few-shot HTC tasks. To address the issue, we propose the hierarchical verbalizer (“HierVerb”), a multi-verbalizer framework treating HTC as a single- or multi-label classification problem at multiple layers and learning vectors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning. In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably outperforms those who inject hierarchy through graph encoders, maximizing the benefits of PLMs. Extensive experiments on three popular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb significantly boosts the HTC performance, meanwhile indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks.

Hierarchical multi-label text classification (HMTC) has become rather challenging when it requires handling large sets of closely related categories. We present a novel software that provides a hybrid embedding-based text representation for HMTC, shortened as HE-HMTC. It made full use of categories’ structure and their labels semantics to enrich the text representation, and therefore attempted to improve the classification performance of the text. In addition, HE-HMTC facilitates improving the accuracy of HMTC tasks. Moreover, our HE-HMTC can easily be generalized in other hierarchical classification tasks and achieve superior performance.

Although an improvement of hierarchical text classification can be achieved by using hierarchical structure information, existing hierarchical text classification methods suffer from two problems: data skew (especially in large-scale hierarchy) and error propagation. In this paper, we first define the concept of path-based semantic vector for the presentation of categories. Then a set of additional reliable training data for data-sparse categories can be retrieved based on such representation and particular similarity metrics. This training data enhancement strategy is classifier independent and can improve the classification of categories without adequate training data. Second, we propose the occurrence probability based strategy for hierarchical text classification which can reduce error propagation efficiently. Cooccurrence probability is then introduced to correct the errors occurred on higher levels of the hierarchy. Extensive experiments show that our hierarchical classification strategies perform well on ODP dataset, even in the condition of having few training data.

Hierarchical text classification is an essential yet challenging subtask of multi-label text classification with a taxonomic hierarchy. Existing methods have difficulties in modeling the hierarchical label structure in a global view. Furthermore, they cannot make full use of the mutual interactions between the text feature space and the label space. In this paper, we formulate the hierarchy as a directed graph and introduce hierarchy-aware structure encoders for modeling label dependencies. Based on the hierarchy encoder, we propose a novel end-to-end hierarchy-aware global model (HiAGM) with two variants. A multi-label attention variant (HiAGM-LA) learns hierarchy-aware label embeddings through the hierarchy encoder and conducts inductive fusion of label-aware text features. A text feature propagation model (HiAGM-TP) is proposed as the deductive variant that directly feeds text features into hierarchy encoders. Compared with previous works, both HiAGM-LA and HiAGM-TP achieve significant and consistent improvements on three benchmark datasets.

Hierarchical text classification concerning the relationship among categories has become an interesting problem recently. Most research has focused on tree-structured categories, but in reality directed acyclic graph (DAG) – structured categories, where a child category may have more than one parent category, appear more often. In this paper, we introduce three approaches, namely, flat, tree-based, and DAG-based, for solving the multi-label text classification problem in which categories are organized as a DAG, and documents are classified into both leaf and internal categories. We also present experimental results of the methods using SVMs as classifiers on the Reuters-21578 collection and our data set of research papers in Artificial Intelligence.Keywordstext classificationhierarchiesmulti-labelsSVM

This paper presents an investigation into the summarisation of the free text element of questionnaire data using hierarchical text classification. The process makes the assumption that text summarisation can be achieved using a classification approach whereby several class labels can be associated with documents which then constitute the summarisation. A hierarchical classification approach is suggested which offers the advantage that different levels of classification can be used and the summarisation customised according to which branch of the tree the current document is located. The approach is evaluated using free text from questionnaires used in the SAVSNET (Small Animal Veterinary Surveillance Network) project. The results demonstrate the viability of using hierarchical classification to generate free text summaries.

Hierarchical classification has been becoming a popular research topic nowadays, particularly on the web as text categorization. For a large web corpus, there can be a hierarchy with hundreds of thousands of topics, so it is common to handle this task using a flat classification approach, inducing a binary classifier only for the leaf-node classes. However, it always suffers from such low prediction accuracy due to an imbalanced issue in the training data. In this paper, we propose two novel strategies: (i) “Top-Level Pruning” to narrow down the candidate classes, and (ii) “Exclusive Top-Level Training Policy” to build more effective classifiers by utilizing the top-level data. The experiments on the Wikipedia dataset show that our system outperforms the traditional flat approach unanimously on all hierarchical classification metrics.

Multi-label text classification is an increasingly important field as large amounts of text data are available and extracting relevant information is important in many application contexts. Probabilistic generative models are the basis of a number of popular text mining methods such as Naive Bayes or Latent Dirichlet Allocation. However, Bayesian models for multi-label text classification often are overly complicated to account for label dependencies and skewed label frequencies while at the same time preventing overfitting. To solve this problem we employ the same technique that contributed to the success of deep learning in recent years: greedy layer-wise training. Applying this technique in the supervised setting prevents overfitting and leads to better classification accuracy. The intuition behind this approach is to learn the labels first and subsequently add a more abstract layer to represent dependencies among the labels. This allows using a relatively simple hierarchical topic model which can easily be adapted to the online setting. We show that our method successfully models dependencies online for large-scale multi-label datasets with many labels and improves over the baseline method not modeling dependencies. The same strategy, layer-wise greedy training, also makes the batch variant competitive with existing more complex multi-label topic models.

In this paper, we propose a new model for text line segmentation and classification, which consists of convolutional and two-layer bi-directional long short-term memory (BiLSTM) networks. Trained on the synthetic text dataset, it performs excellently when predicting the real data. Without labelling every line on the real data, a generalized standard for evaluating the accuracy is proposed. We also propose a simplified IoU loss to improve the execution speed greatly. In the experiments, it achieves 98.1% line segmentation accuracy and 99.5% classification accuracy on the English fiction Pride and Prejudice by Jane Austen, and achieves 98.5% line segmentation accuracy and 99.7% classification accuracy on the The Secret Of Plato's Atlantis by John Arundell, outperforming the traditional methods. Furthermore, for 1024 × 724 input samples, it gets 2.95 FPS speed when using a Tesla K80 GPU. Index Terms—Text line segmentation, Text classification, Synthetic text, BiLSTM, Convolutional network.

Float-Zone (FZ) crystal growth process allows for producing higher purity silicon crystal with much lower concentrations of impurities, in particular low oxygen content. Nevertheless, the FZ process occasionally faces the problem of small contamination from oxidation. This can come in the form of a thin oxide layer that may form on un-melted polysilicon surface. The appearance of the oxide layer indicates degraded machine performance and the need for machine maintenance. Therefore, oxide investigation is important for improving both the FZ process and FZ machines, and the first step is oxide recognition. In this study, we characterized oxide into mainly three varieties, according to their surface texture characteristics, which are: (i) spot (ii) shadow and (iii) ghost curtain. We leveraged FZ images captured from the vision system integrated on the FZ machine to establish an oxide dataset. Targeted for data imbalance problem in our dataset, a method based on transfer learning and asymmetric loss for multi-label oxide classification is presented in this work. The results showed that the pre-trained model and the asymmetric loss used for training outperformed the baseline models and improved the classification performance. Furthermore, this study deeply investigated the effectiveness of the components of asymmetric loss. Finally, Gradient-weighted Class Activation Mapping (Grad-CAM) was employed to explain decision process of the models in order to adopt them in the industry.

Text classification is a procedure of categorizing a text into classes or predefined labels, allowing effective analysis and organizing the large volume of textual data. In the growth of online textual data like news, digital platforms, or social media, an effective model for text summarization and classification. However, traditional models of text classification had difficulty capturing deep semantic dependencies and contextual embeddings among texts or words, which led to misclassification and minimized classification accuracy. To overcome these challenges, the research proposed Token Level Class Importance (TLCI) and Layer Wise Learning Rate Decay (LWLRD) named as TLLRD is integrated with Bidirectional Encoder Representations from Transformers (BERT) named as (TLLRD-BERT), which is utilized for multiclass violence text classification and enhanced classification performance. The model integrates LLRD to stabilize fine-tuning across transformer layers and TLCI to emphasize semantically significant words associated with aggressive or violent expressions. The proposed TLLRD-BERT attains an accuracy of 98.24%, 96.54% and 99.71% through YouTube comments dataset, the news headline dataset and the SMS spam detection dataset, respectively.

Multi-label text classification has a wide range of applications in the real world. However, the data distribution in the real world is often imbalanced, which leads to serious long-tailed problems. For multi-label classification, due to the vast scale of datasets and existence of label co-occurrence, how to effectively improve the prediction accuracy of tail labels without degrading the overall precision becomes an important challenge. To address this issue, we propose A Dual-Branch Learning Model with Gradient-Balanced Loss (DBGB) based on the paradigm of existing pre-trained multi-label classification SOTA models. Our model consists of two main long-tailed module improvements. First, with the shared text representation, the dual-classifier is leveraged to process two kinds of label distributions; one is the original data distribution and the other is the under-sampling distribution for head labels to strengthen the prediction for tail labels. Second, the proposed gradient-balanced loss can adaptively suppress the negative gradient accumulation problem related to labels, especially tail labels. We perform extensive experiments on three multi-label text classification datasets. The results show that the proposed method achieves competitive performance on overall prediction results compared to the state-of-the-art methods in solving the multi-label classification, with significant improvement on tail-label accuracy.

The goal of electronic monitoring (EM) of longline fishing is to monitor the fish catching activities on fishing vessels, either for the regulatory compliance or catch counting. Hierarchical classification based on videos allows for inexpensive and efficient fish species identification of catches from longline fishing, where fishes are under severe deformation and self-occlusion during the catching process. More importantly, the flexibility of hierarchical classification mitigates the laborious efforts of human reviews by providing confidence scores in different hierarchical levels. Some related works either use cascaded models for hierarchical classification or make predictions per image or predict one overlapping hierarchical data structure of the dataset in advance. However, with a known non-overlapping hierarchical data structure provided by fisheries scientists, our method enforces the hierarchical data structure and introduces an efficient training and inference strategy for video-based fisheries data. Our experiments show that the proposed method outperforms the classic flat classification system significantly and our ablation study justifies our contributions in CNN model design, training strategy, and the video-based inference schemes for the hierarchical fish species classification task.

Quality control of crowdsourced classification using hierarchical class structures

In this paper, we model learning to rank algorithms based on structural dependencies in hierarchical multi-label text categorization (TC). Our method uses the classification probability of the binary classifiers of a standard top-down approach to generate k-best hypotheses. The latter are generated according to their global probability while at the same time satisfy the structural constraints between father and children nodes. The rank is then refined using Support Vector Machines and tree kernels applied to a structural representation of hypotheses, i.e., a hierarchy tree in which the outcome of binary one-vs-all classifiers is directly marked in its nodes. Our extensive experiments on the whole Reuters Corpus Volume 1 show that our models significantly improve over the state of the art in TC, thanks to the use of structural dependecies.