Improving the Accuracy of Question Classification with Machine Learning

Conventional application of convolutional neural networks (CNNs) for image classification and recognition is based on the assumption that all target classes are equal (i.e. no hierarchy) and exclusive of one another (i.e. no overlap). CNN-based image classifiers built on this assumption, therefore, cannot take into account an innate hierarchy among target classes (e.g. cats and dogs in animal image classification) or additional information that can be easily derived from the data (e.g. numbers larger than five in the recognition of handwritten digits), thereby resulting in scalability issues when the number of target classes is large. Combining two related but slightly different ideas of hierarchical classification and logical learning by auxiliary inputs, we propose a new learning framework called hierarchical auxiliary learning, which not only address the scalability issues with a large number of classes but also could further reduce the classification/recognition errors with a reasonable number of classes. In the hierarchical auxiliary learning, target classes are semantically or non-semantically grouped into superclasses, which turns the original problem of mapping between an image and its target class into a new problem of mapping between a pair of an image and its superclass and the target class. To take the advantage of a superclass as a hint during the learning phase, we introduce an auxiliary block into a neural network, which generates auxiliary scores used as additional information for final classification/recognition; in this paper, we add the auxiliary block between the last residual block and the fully-connected output layer of the ResNet. Experimental results show that the proposed hierarchical auxiliary learning reduces classification errors up to 0.56, 1.6 and 3.56 percent with MNIST, SVHN and CIFAR-10 datasets, respectively.

This paper examines the possibilities of using Landsat Thematic Mapper images for updating land-use information on topographic maps. Information obtained from satellite images can be kept up-todate and a large number of classes can be distinguished. However, the quality of the information still leaves much to be desired, with a reliability of no more than 40 - 90 %. By using digitized landuse information from a topographic map as a source of contest information the reliability could be improved considerably. The method permitted the Identification of a large number of additional classes and provided quantitative insight into the dynamics of landscap change. Verification using 1:18,000 scale aerial photographs and field data showed the results to have a reliability of over 95 %. It can be expected that whereever topographic maps permit the extraction of land-use information in a similar way as described in this paper, remote sensing information will meet operational accuracy requirements for many fields of application e.g. physical planning, map udating, land-use planning etc. Paper presented at the IGARSS'88 conference Remote Sensing - moving towards the 21st century in Edinburgh, Scotland, 12-16 September 1988

Quality control of crowdsourced classification using hierarchical class structures

The label tree-based classification is one of the most popular approaches for reducing the testing complexity to sublinear with the large number of classes. One of the popular approaches to generate the label tree is to apply recursively a spectral clustering algorithm to an affinity matrix for partition set of class labels into subsets, each subset corresponds to a child node of the tree. To obtain the affinity matrix from confusion matrix, a set of N binary one-versus-all classifiers is trained and applied on validation set, where N is number of classes. However, these approaches are not reliable when there are a large number of classes because it is too costly to train these classifiers. Furthermore, the affinity matrix could not reflect the real similarity among classes due to the classification accuracy can be low. In addition, the resulting label tree may not be balanced due to the objective function of spectral clustering penalizes unbalanced partitions. In this paper, to achieve better similarity measurement between classes and without using one-versus-all classifiers, we adopt the sum match kernel to get similarity matrix. Moreover, we propose a heuristic for learning balanced tree structure by adjusting the number of class labels in clusters after the spectral clustering is done. The experimental results on benchmark datasets SUN-397 and Caltech-256 show that the performance of the proposed approach outperforms significantly the other approaches.

Studies have confirmed the robust performance of machine learning classifiers for various source code modeling tasks. In general, machine learning approaches are incapable of handling imbalanced datasets, since they are sensitive to the choice of diverse classes. Therefore, these approaches may lean towards the classes with a large percentage of observations. In this work, we investigate and explore the impact of balanced and imbalanced learning on source code suggestion task otherwise known as code completion, covering a large number of imbalanced classes. We further explore the impact of vocabulary size on modeling performance. First, we provide the essentials to formulate the problem of source code suggestion as a classification task and investigate the level of imbalanced classes. Second, we train the four most adapted neural language models as a baseline to assess the modeling performance. Third, we impose two diverse class balancing techniques, TomekLinks and AllKNN, to balance the datasets and evaluate their impact on the modeling performance. Finally, we trained these models with a weighted imbalanced learning approach and compared the performance with balanced learning approaches. Additionally, we train models by varying the vocabulary size to study their impact. In total, we trained 230 models on 10 real-world software projects and extensively evaluated these models with widely used performance metrics such as Precision, Recall, FScore, mean reciprocal rank (MRR), and Receiver operating characteristics (ROC). Additionally, we employed ANOVA statistical analysis to study the statistical significance and differences between these approaches. This study has demonstrated that the modeling performance decreases during balanced model training, whereas the weighted imbalance training produces comparable results and is more efficient in terms of time cost. Additionally, this study exhibits that a large size of vocabulary does not necessarily improve the modeling performance when out-of-vocabulary predictions are disregarded.

Question categorization and classification using grammar based approach

Given a user question, the goal of a Question Answering (QA) system is to retrieve answers rather than full documents or even best-matching passages, as most Information Retrieval systems currently do. In this paper, we present BRUJA, a QA system for the management of multilingual collections. BRUJ rkstions (English, Spanish and French). The BRUJA architecture is not formed with three monolingual QA systems but instead uses English as Interlingua to make usual QA tasks such as question classifications and answer extractions. In addition, BRUJA uses Cross Language Information Retrieval (CLIR) techniques to retrieve relevant documents from a multilingual collection. On the one hand, we have more documents to find answers from but on the other hand, we are introducing noise into the system because of translations to the Interlingua (English) and the CLIR module. The question is whether the difficulty of managing three languages is worth it or whether a monolingual QA system delivers better results. We report on in-depth experimentation and demonstrate that our multilingual QA system gets better results than its monolingual counterpart whenever it uses good translation resources and, especially, CLIR techniques that are state-of-the-art.

The state-of-the-art object detection and image classification methods can perform impressively on more than 9k classes. In contrast, the number of classes in semantic segmentation datasets is relatively limited. This is not surprising when the restrictions caused by the lack of labeled data and high computation demand for segmentation are considered. In this paper, we propose a novel training methodology to train and scale the existing semantic segmentation models for a large number of semantic classes without increasing the memory overhead. In our embedding-based scalable segmentation approach, we reduce the space complexity of the segmentation model’s output from O(C) to O(1), propose an approximation method for ground-truth class probability, and use it to compute cross-entropy loss. The proposed approach is general and can be adopted by any state-of-the-art segmentation model to gracefully scale it for any number of semantic classes with only one GPU. Our approach achieves similar, and in some cases, even better mIoU for Cityscapes, Pascal VOC, ADE20k, COCO-Stuff10k datasets when adopted to DeeplabV3+ model with different backbones. We demonstrate a clear benefit of our approach on a dataset with 1284 classes, bootstrapped from LVIS and COCO annotations, with almost three times better mIoU than the DeeplabV3+. Our source code is available at: https://github.com/shipra25jain/ESSNet.

Sea ice can cause some of the most prominent marine disasters in polar and high latitude regions, and remote sensing technology provides an important means to detect such hazards. The accuracy of sea ice detection depends on the number and quality of labeled samples, but because of environmental conditions in sea ice regions, acquisition of labeled samples can be time-consuming and labor-intensive. To solve this problem, we propose a combined active learning (AL) and semisupervised learning (SSL) classification framework for sea ice detection. At first, we acquire most informative and representative samples by AL; then labeled samples acquired by AL are used as the initial labeled samples for SSL, in this framework, we not only choose the most valuable samples but also use the large number of unlabeled samples to enhance the classification accuracy. In AL phase, we use two different sampling strategies: uncertainty and diversity. In the SSL phase, we utilize a sampling function integrating AL to acquire semilabeled samples, and we use a transductive support vector machine as a classification model. We analyze three remote sensing images (hyperspectral and multispectral) and conduct detailed comparative analyses between the proposed method and others. Our proposed method achieves the highest classification accuracies (89.9734%, 97.4919%, and 89.7166%) in both experiments. These results show that the proposed method exhibits better overall performance than other methods and can be effectively applied to sea ice detection using remote sensing.

In this paper we address the task of hierarchical bird species identification from audio recordings. We evaluate three types of approaches to deal with hierarchical classification problems: the flat classification approach, the local-model per parent node classifier approach and the global-model hierarchical-classification approach. For the flat and local-model classification approach we employ the classic Naive Bayes algorithm. For the global-model approach we use the Global Model Naive Bayes (GMNB) algorithm. As in the classical Naive Bayes, the algorithm computes prior probabilities and likelihoods, but these computations take into account the hierarchical classification scenario: it assumes that any example which belongs to a given class will also belong to all its ancestor classes. In the current application, the employed class hierarchy is the standard scientific taxonomy of birds used in Biology. In order to deal with the bird songs we obtain features by computing several acoustic quantities from intervals of the audio signal. We conduct three experiments in order to compare the three different approaches to the hierarchical bird species identification problem. Our experimental results show that the use of the GMNB hierarchical classification algorithm outperforms both the flat and local-model approaches (Using the Hierarchical F-measure metric), hence the use of a global-model approach (such as the GMNB) can be a feasible way to improve the classification performance for problems with a large number of classes.

CLAIM is a tool for analyzing and classifying discrepancy reports that allows for the incorporation of domain expert knowledge into a semi-supervised machine learning (ML) process (a semi-supervised learning uses a small number of manually labeled data and a much larger amount of unlabeled for training a machine learning algorithm). By using this domain knowledge, classification accuracy is higher than conventional ML approaches. The advantages are particularly apparent with small, imbalanced data sets that are quite common in discrepancy report data sets (an imbalanced data set has an unequal distribution of documents categories within each category). The CLAIM method is robust against human bias and can tolerate misclassifications of up to 20% of the training set. The increased accuracy of the CLAIM methodology makes ML a viable tool for safety, reliability, and software development process decision making. The modest human labor requirement enables use of the method under circumstances that previously made free text discrepancy report analysis infeasible due to resource, scheduling, and cost constraints.

Semi-supervised matrixized least squares support vector machine

The classification of questions plays a key role in the system of the question-answer system, helping in the search or construction of accurate answers and thereby improving the quality of the answers provided. The following approaches are usually used to classify questions: rule-based, machine learning, and hybrid. The Kazakh language is an agglutinative language with complex morphological and syntactic involvement of polysynthetism, and therefore it is difficult to conduct machine learning research. This article presents a study on the classification of questions using rules. The question processing module helps you assign appropriate categories to questions and identify keywords from the provided input question. A prototype program was developed based on the proposed method, and an experiment was conducted involving 200 legislative questions collected from government websites such as egov.kz, adilet.kz and zan.kz . Using these question templates, the questions were categorized into specific categories. In this article, we have proposed a compact and effective method for classifying questions. The results of the experiment show that even with a small set of question categories, we can classify them with a more satisfactory and improved result.

Magnetic resonance imaging (MRI) is playing an important role in the classification of breast tumors. MRI can be used to obtain multiparametric (mp) information, such as structural, hemodynamic, and physiological information. Quantitative analysis of mp-MRI data has shown potential in improving the accuracy of breast tumor classification. In general, a large set of quantitative and texture features can be generated depending upon the type of methodology used. A suitable combination of selected quantitative and texture features can further improve the accuracy of tumor classification. Machine learning (ML) classifiers based upon features derived from MRI data have shown potential in tumor classification. There is a need for further research studies on selecting an appropriate combination of features and evaluating the performance of different ML classifiers for accurate classification of breast tumors. The objective of the current study was to develop and optimize an ML framework based upon mp-MRI features for the characterization of breast tumors (malignant vs. benign and low- vs. high-grade). This study included the breast mp-MRI data of 60 female patients with histopathology results. A total of 128 features were extracted from the mp-MRI tumor data followed by features selection. Five ML classifiers were evaluated for tumor classification using 10-fold crossvalidation with 10 repetitions. The support vector machine (SVM) classifier based on optimum features selected using a wrapper method with an adaptive boosting (AdaBoost) technique provided the highest sensitivity (0.96 ± 0.03), specificity (0.92 ± 0.09), and accuracy (94% ± 2.91%) in the classification of malignant versus benign tumors. This method also provided the highest sensitivity (0.94 ± 0.07), specificity (0.80 ± 0.05), and accuracy (90% ± 5.48%) in the classification of low- versus high-grade tumors. These findings suggest that the SVM classifier outperformed other ML methods in the binary classification of breast tumors.

Classifier combination holds the potential of improving performance by combining the results of multiple classifers. For domains with very large numbers of classes, such as biometrics, we present an axiomatic framework of desirable mathematical properties for combination functions of rank-based classifiers. This framework represents a continuum of combination rules, including the Borda Count, Logistic Regression, and Highest Rank combination methods as extreme cases [11], [23], [4], [13]. Intuitively, this framework captures how the two complementary concepts of general preference for specific classifiers and the confidence it has in any specific result (as indicated by ranks) can be balanced while maintaining consistent rank interpretation. Mixed Group Ranks (MGR) is a new combination function that balances preference and confidence by generalizing these other functions. We demonstrate that MGR is an effective combination approach by performing multiple experiments on data sets with large numbers of classes and classifiers from the FERET face recognition study.