An improvement of flat approach on hierarchical text classification using top-level pruning classifiers

Due to the complex label hierarchy and intensive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor performance especially when low-resource or few-shot settings are considered. Recently, there is a growing trend of applying prompts on pre-trained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks. However, limited work has studied the paradigm of prompt-based learning in the HTC problem when the training data is extremely scarce. In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further explore few-shot HTC tasks. To address the issue, we propose the hierarchical verbalizer (“HierVerb”), a multi-verbalizer framework treating HTC as a single- or multi-label classification problem at multiple layers and learning vectors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning. In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably outperforms those who inject hierarchy through graph encoders, maximizing the benefits of PLMs. Extensive experiments on three popular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb significantly boosts the HTC performance, meanwhile indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks.

Although an improvement of hierarchical text classification can be achieved by using hierarchical structure information, existing hierarchical text classification methods suffer from two problems: data skew (especially in large-scale hierarchy) and error propagation. In this paper, we first define the concept of path-based semantic vector for the presentation of categories. Then a set of additional reliable training data for data-sparse categories can be retrieved based on such representation and particular similarity metrics. This training data enhancement strategy is classifier independent and can improve the classification of categories without adequate training data. Second, we propose the occurrence probability based strategy for hierarchical text classification which can reduce error propagation efficiently. Cooccurrence probability is then introduced to correct the errors occurred on higher levels of the hierarchy. Extensive experiments show that our hierarchical classification strategies perform well on ODP dataset, even in the condition of having few training data.

In this paper, we model learning to rank algorithms based on structural dependencies in hierarchical multi-label text categorization (TC). Our method uses the classification probability of the binary classifiers of a standard top-down approach to generate k-best hypotheses. The latter are generated according to their global probability while at the same time satisfy the structural constraints between father and children nodes. The rank is then refined using Support Vector Machines and tree kernels applied to a structural representation of hypotheses, i.e., a hierarchy tree in which the outcome of binary one-vs-all classifiers is directly marked in its nodes. Our extensive experiments on the whole Reuters Corpus Volume 1 show that our models significantly improve over the state of the art in TC, thanks to the use of structural dependecies.

This paper presents an investigation into the summarisation of the free text element of questionnaire data using hierarchical text classification. The process makes the assumption that text summarisation can be achieved using a classification approach whereby several class labels can be associated with documents which then constitute the summarisation. A hierarchical classification approach is suggested which offers the advantage that different levels of classification can be used and the summarisation customised according to which branch of the tree the current document is located. The approach is evaluated using free text from questionnaires used in the SAVSNET (Small Animal Veterinary Surveillance Network) project. The results demonstrate the viability of using hierarchical classification to generate free text summaries.

Hierarchy-Aware and Label Balanced Model for Hierarchical Text Classification

Hierarchical text categorization is a task that aims to assign predefined categories to text documents with hierarchical constraint. Recently, deep learning techniques has shown many success results in various fields, especially, in text categorization. In our previous work called Shared Hidden Layer Neural Network (SHL-NN), it has shown that sharing information between levels can improve a performance of the model. However, this work is based on a sequence of unsupervised word embedding vectors, so the performance should be limited. In this paper, we propose a supervised document embedding specifically designed for hierarchical text categorization based on Autoencoder, which is trained from both words and labels. To enhance the embedding vectors, the document embedding strategies are invented to utilize the class hierarchy information in the training process. To transfer the prediction result from the parent classes, the shared information technique has been improved to be more flexible and efficient. The experiment was conducted on three standard benchmarks: WIPO-C, WIPO-D and Wiki comparing to two baselines: SHL-NN and a top-down based SVM framework with TF-IDF inputs called “HR-SVM.” The results show that the proposed model outperforms all baselines in terms of F1 macro.

Hierarchical multi-label text classification (HMTC) has become rather challenging when it requires handling large sets of closely related categories. We present a novel software that provides a hybrid embedding-based text representation for HMTC, shortened as HE-HMTC. It made full use of categories’ structure and their labels semantics to enrich the text representation, and therefore attempted to improve the classification performance of the text. In addition, HE-HMTC facilitates improving the accuracy of HMTC tasks. Moreover, our HE-HMTC can easily be generalized in other hierarchical classification tasks and achieve superior performance.

Hierarchical text classification is an essential yet challenging subtask of multi-label text classification with a taxonomic hierarchy. Existing methods have difficulties in modeling the hierarchical label structure in a global view. Furthermore, they cannot make full use of the mutual interactions between the text feature space and the label space. In this paper, we formulate the hierarchy as a directed graph and introduce hierarchy-aware structure encoders for modeling label dependencies. Based on the hierarchy encoder, we propose a novel end-to-end hierarchy-aware global model (HiAGM) with two variants. A multi-label attention variant (HiAGM-LA) learns hierarchy-aware label embeddings through the hierarchy encoder and conducts inductive fusion of label-aware text features. A text feature propagation model (HiAGM-TP) is proposed as the deductive variant that directly feeds text features into hierarchy encoders. Compared with previous works, both HiAGM-LA and HiAGM-TP achieve significant and consistent improvements on three benchmark datasets.

Along with the exponential growth of text data on the Web, particularly of the user-generated content, comes an increasing need for hierarchically organizing documents, retrieving documents accurately, and discovering evolutionary trends of various popular topics from the data. However, all of these are challenging due to the diversity, heterogeneity, noisiness and time-sensitivity of Web 2.0 data. Motivated by this, we tackle the challenges at a fundamental level, by proposing a novel topic modeling method with ontological guidance. It may be used to discover topic language models formalizing various terms relevant to given topics using the Web data. The topic model takes into account both the ontological relationships amongst the topics defined in a topic taxonomy and also word co-occurrence patterns in the data to automatically identify the portions in the data relevant to the topics. Then, it estimates language models for these topics from these relevant portions. At an application level, we use the topic model to propose novel approaches for three different tasks, namely hierarchical text classification without labeled data, information retrieval with pseudo-relevance feedback, and discovering topic evolutionary trends. Our classification experiment on the IPTC (International Press and Telecommunications Council) taxonomy, containing more than 1100 topics, shows that our approach achieves a performance of 67% in terms of the hierarchical version of the F-1 measure, without using any labeled data. Our retrieval experiments on five benchmark datasets show that compared to baseline retrieval (without pseudo-relevance feed-back), our approach improves on average 39% in terms of mean average precision. Finally, for the last task, using blog data, our approach discovers meaningful insights on how the crowd responds to various news topics such as the language used to discuss each topic, how this language drifts over time, and when the crowd’s focus on a topic increases, reaches a peak, and declines.

Hierarchical text classification concerning the relationship among categories has become an interesting problem recently. Most research has focused on tree-structured categories, but in reality directed acyclic graph (DAG) – structured categories, where a child category may have more than one parent category, appear more often. In this paper, we introduce three approaches, namely, flat, tree-based, and DAG-based, for solving the multi-label text classification problem in which categories are organized as a DAG, and documents are classified into both leaf and internal categories. We also present experimental results of the methods using SVMs as classifiers on the Reuters-21578 collection and our data set of research papers in Artificial Intelligence.Keywordstext classificationhierarchiesmulti-labelsSVM

Hierarchical classification for acoustic scenes using deep learning

Renxuan Albert Li, Ihab Hajjar, Felicia Goldstein, Jinho D. Choi. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. 2020.

Keyword extraction of complex hierarchical categories becomes a challenge in text mining since commonly used classification for flat categories results in low accuracy. This paper presents a method to improve keyword extraction from hierarchical categories considering terms occurred in category from a hierarchy as additional factors in term-weighting. The method is an enhancement of a basic TF-IDF calculation; thus, it can comfortably be used for keyword extraction and classification. By taking term frequency and inverse document frequency of categories hierarchically related to a focused category, we can determine how important terms are in their family categories. In this work, hierarchy relations used in calculation are sub-categories, supercategories and sibling-categories. From experiment results, we found that the proposed method gained higher accuracy for about 40% from a baseline in a classification task.

Twitter is a micro logging service where worldwide users publish and share their feelings. However, sentiment analysis for Twitter messages ('tweets') is regarded as a challenging problem because tweets are short and informal. In this paper, we apply a novel approach for automatically classifying the sentiment and emotions of Twitter messages. These messages are hierarchically categorized on basis of neutrality, polarity (positive or negative) and presence of various emotions. The hierarchical classification approach (HC) is a specialization of the well-known flat classification task. The main difference between them is that when using HC, examples must be assigned to classes organized in a previously defined class hierarchy, while traditional flat classification does not take into account the hierarchical information. We applied our model to posts collected from Twitter regarding the 2011 season of the Brazilian Soccer League. Our results show that the proposed method outperforms the corresponding flat approach in emotion classification.

In the framework of multi-domain text-to-speech synthesis, it is essential (i) to design a hierarchically structured database for allowing several domains in the same speech corpus and (ii) to include a text classification module that, at run time, assigns the input sentences to a domain or set of domains from the database. We present a hierarchical text classifier based on independent component analysis (ICA), which is capable of (i) organizing the contents of the corpus in a hierarchical manner and (ii) classifying the texts to be synthesized according to the learned structure. The document organization and classification performance of our ICA-based hierarchical classifier are evaluated in several encouraging experiments conducted on a journalistic-style text corpus for speech synthesis in Catalan.