Local ensemble learning from imbalanced and noisy data for word sense disambiguation

Multi-class imbalanced big data classification on Spark

In this paper, we introduced a classifier ensemble approach to combine heterogeneous classifiers in the presence of class label noise in the datasets. To enhance the performance of classifier ensemble, we give a preprocessing approach to filter out this class label noise. The filtered data is then used to learn individual classifier model. After that, a weight learning method is introduced to learn weights on each individual classifier to create a classifier ensemble. We applied genetic algorithm to search for an optimal weight vector on which classifier ensemble is expected to give best accuracy. The proposed approach is evaluated on variety of real life datasets. The proposed technique is also compared with existing standard ensemble techniques such as Adaboost, Bagging and RSM to show the superiority of proposed ensemble method, in the presence of class label noise, as compared to its competitors and also to show the sensitivity of competitors to class label noise.

In many real-world applications, it is common to have uneven number of examples among multiple classes. The data imbalance, however, usually complicates the learning process, especially for the minority classes, and results in deteriorated performance. Boosting methods were proposed to handle the imbalance problem. These methods need elongated training time and require diversity among the classifiers of the ensemble to achieve improved performance. Additionally, extending the boosting method to handle multi-class data sets is not straightforward. Examples of applications that suffer from imbalanced multi-class data can be found in face recognition, where tens of classes exist, and in capsule endoscopy, which suffers massive imbalance between the classes. This dissertation introduces RegBoost, a new boosting framework to address the imbalanced, multi-class problems. This method applies a weighted stratified sampling technique and incorporates a regularization term that accommodates multi-class data sets and automatically determines the error bound of each base classifier. The regularization parameter penalizes the classifier when it misclassifies instances that were correctly classified in the previous iteration. The parameter additionally reduces the bias towards majority classes. Experiments are conducted using 12 diverse data sets with moderate to high imbalance ratios. The results demonstrate superior performance of the proposed method compared to several state-of-the-art algorithms for imbalanced, multi-class classification problems. More importantly, the sensitivity improvement of the minority classes using RegBoost is accompanied with the improvement of the overall accuracy for all classes. With unpredictability regularization, a diverse group of classifiers are created and the maximum accuracy improvement reaches above 24%. Using stratified undersampling, RegBoost exhibits the best efficiency. The reduction in computational cost is significant reaching above 50%. As the volume of training data increase, the gain of efficiency with the proposed method becomes more significant.

Word Sense Disambiguation (WSD) is the process of resolving the meaning of a word unambiguously in a given natural language context. Within the scope of this thesis, it is the process of marking text with explicit sense labels. What constitutes a sense is a subject of great debate. An appealing perspective, aims to define senses in terms of their multilingual correspondences, an idea explored by several researchers, Dyvik (1998), Ide (1999), Resnik & Yarowsky (1999), and Chugur, Gonzalo & Verdejo (2002) but to date it has not been given any practical demonstration. This thesis is an empirical validation of these ideas of characterizing word meaning using cross-linguistic correspondences. The idea is that word meaning or word sense is quantifiable as much as it is uniquely translated in some language or set of languages. Consequently, we address the problem of WSD from a multilingual perspective; we expand the notion of context to encompass multilingual evidence. We devise a new approach to resolve word sense ambiguity in natural language, using a source of information that was never exploited on a large scale for WSD before. The core of the work presented builds on exploiting word correspondences across languages for sense distinction. In essence, it is a practical and functional implementation of a basic idea common to research interest in defining word meanings in cross-linguistic terms. We devise an algorithm, SALAAM for Sense Assignment Leveraging Alignment And Multilinguality, that empirically investigates the feasibility and the validity of utilizing translations for WSD. SALAAM is an unsupervised approach for word sense tagging of large amounts of text given a parallel corpus—texts in translation—and a sense inventory for one of the languages in the corpus. Using SALAAM, we obtain large amounts of sense annotated data in both languages of the parallel corpus, simultaneously. The quality of the tagging is rigorously evaluated for both languages of the corpora. The automatic unsupervised tagged data produced by SALAAM is further utilized to bootstrap a supervised learning WSD system, in essence, combining supervised and unsupervised approaches in an intelligent way to alleviate the resources acquisition bottleneck for supervised methods. Essentially, SALAAM is extended as an unsupervised approach for WSD within a learning framework; in many of the cases of the words disambiguated, SALAAM coupled with the machine learning system rivals the performance of a canonical supervised WSD system that relies on human tagged data for training. Realizing the fundamental role of similarity for SALAAM, we investigate different dimensions of semantic similarity as it applies to verbs since they are relatively more complex than nouns, which are the focus of the previous evaluations. We design a human judgment experiment to obtain human ratings on verbs' semantic similarity. The obtained human ratings are cast as a reference point for comparing different automated similarity measures that crucially rely on various sources of information. Finally, a cognitively salient model integrating human judgments in SALAAM is proposed as a means of improving its performance on sense disambiguation for verbs in particular and other word types in general.

Class label noise is a data-level difficulty associated with training objects with incorrectly assigned labels. This problem may originate from poorly documented historic data, errors during data generation process or mistakes made by human experts. Inclusion of such examples during the training process will mislead the classifier by presenting a falsified class distribution and consequently lead to degradation of models' generalization abilities. This phenomenon becomes even more troublesome in multi-class scenarios that may be affected by highly complex intra-class noise. Decomposition strategies with binary classifiers were proven to alleviate this difficulty by using simplified binary subtasks that are less affected by the noise. In this paper we propose to extend this approach by using the one-class classification decomposition. In this scenario each class has assigned individual one-class classifier that aims at capturing its distinguishing characteristics. This allows to create a robust data description and then apply a dedicated classifier combination in order to reconstruct the original multi-class task. We further extend this concept by using fuzzy one-class classifiers that allow to associate membership values with each training objects. This allows us to reduce the influence of uncertain and potentially noisy samples on the shape of learned decision boundary. Experimental study backed-up with statistical analysis shows that fuzzy one-class classifier decomposition offers an excellent robustness to noise in multi-class classification.

In the last decades, the use of ensembles of classifiers in different domains of application has received significant attention of the Machine Learning community. In a typical architecture of ensemble, a new input pattern is presented to all K components. The individual classifiers provide their output and send them to a combination method, which is responsible for providing the final output. Ensemble of classifiers can be applied to any classification problem, single-label or multi- label problems. Multi-label (ML) classification has been received much attention from various research domains, such as text categorization, bioinformatics, computer vision, among others. In this paper, we investigate the use of ensemble of classifiers when applied to multi-label problems, focusing on the use of diversity measures in these systems when applied to multi-label problems. In order to do this, the present work proposes an adapted approach for two well-known diversity measures, good and bad, for ensembles of multi-label classifiers. In order to assess the feasibility of the proposed diversity measures in ensemble of multi-label classifiers, an empirical analysis is conducted, with eight multi-label classification problems. Generally, our finds indicate that there is a strong correlation between performance of the ML ensembles with the proposed diversity measures, showing that these diversity measures can be used as an important tool in the design of efficient ensembles of multi-label classifiers.

In reality, data on many engineering problems are classified as imbalanced data, which poses significant challenges to adopting a data-driven approach. In this paper, the SOU data preprocessing method combining synthetic minority oversampling technique, random oversampling and random under sampling is used for data equalization. After, the cars evaluation, wine production regions are classified using the ensemble learning based on stacking method. Research is also conducted on the classification of imbalances in the condition assessment of aero-engines at work. The results show that the data equalization method and diagnostic framework proposed in this paper can effectively diagnose multi-class imbalanced data, and the ensemble model has higher diagnostic stability and generalization ability, which is more effective for solving multi-class imbalance problems.

Word Sense Disambiguation (WSD) is the task of extracting an appropriate sense of an ambiguous word in a sentence. WSD is an essential task for language processing, as it is a pre-requisite for determining the closest interpretations of various language-based applications. In this paper, we have made an attempt to exploit the word embedding for finding the solution for WSD for the Hindi texts. This task involves two steps - the creation of word embedding and leveraging cosine similarity to identify an appropriate sense of the word. In this process, we have considered two mostly used word2vec architectures known as Skip-Gram and Continuous Bag-Of-Words [2] models to develop the word embedding. Further, we have chosen the sense with the closest proximity to identify the meaning of an ambiguous word. To prove the effectiveness of the proposed model, we have performed experiments on large corpora and have achieved an accuracy of nearly 52%.

BPSO-Adaboost-KNN ensemble learning algorithm for multi-class imbalanced data classification

We present an unsupervised learning strategy for word sense disambiguation (WSD) that exploits multiple linguistic resources including a parallel corpus, a bilingual machine readable dictionary, and a thesaurus. The approach is based on Class Based Sense Definition Model (CBSDM) that generates the glosses and translations for a class of word senses. The model can be applied to resolve sense ambiguity for words in a parallel corpus. That sense tagging procedure, in effect, produces a semantic bilingual concordance, which can be used to train WSD systems for the two languages involved. Experimental results show that CBSDM trained on Longman Dictionary of Contemporary English, English-Chinese Edition (LDOCE E-C) and Longman Lexicon of Contemporary English (LLOCE) is very effectively in turning a Chinese-English parallel corpus into sense tagged data for development of WSD systems.

Dynamic Classification Ensembles for Handling Imbalanced Multiclass Drifted Data Streams

Word Sense Disambiguation (WSD) is crucial and its significance is prominent in every application of computational linguistics. WSD is a challenging problem of Natural Language Processing (NLP). Though there are lots of algorithms for WSD available, still little work is carried out for choosing optimal algorithm for that. Three approaches are available for WSD, namely, Knowledge-based approach, Supervised approach and Unsupervised approach. Also, one can use the combination of given approaches. Supervised approach needs large amounts of manually created sense-annotated corpus which takes computationally more amount of time and effort. Knowledge-based approach requires machine readable dictionaries, sense inventories, thesauri, etc, which are dependent on own interpretation about word's sense; Whereas unsupervised approach uses sense-unannotated corpus and it is based on the phenomenon of working that words that co-occur have similarity. This research is for Hindi language which uses Hierarchical clustering algorithm with different similarity measures which are cosine, Jaccard and dice, the result of clusters is overlapped with Hindi WordNet a product of IIT Bombay which improves result of word sense disambiguation as clustering does grouping of words which are similar.

Improving classification accuracy of motor imagery-based brain computer interface (MI-BCI) systems has been discussed widely in the BCI research community. Analyses of multi-class MI data are challenging because feature extraction and classification of these data are more difficult as compared with those applied to binary-class data. This paper introduces a new model for multi-class MI-BCI data classification. The approach encompasses two main stages: feature extraction and fuzzy classification. In the feature extraction phase, a common spatial pattern algorithm is employed to extract significant discriminant features from the multi-class data. The obtained features serve as inputs to a fuzzy classifier that is designed by fusing the fuzzy standard additive model and particle swarm optimization method. The proposed fuzzy model aims to handle uncertainty, noise, and outliers existed in MI-BCI data. Experimental studies are carried out using two benchmark BCI competition data sets. Our results show the effectiveness of the fuzzy classifier against its competing techniques. The proposed model paths the way for paralysed or disordered patients to use movement imagery to control different assistive devices in their daily activities.

This paper presents a method of word sense disambiguation that assigns a target word the sense that is most related to the senses of its neighbor words. We explore the use of measures of relatedness between word senses based on a novel hybrid approach. First, we investigate how to “literally” and “regularly” express a “concept”. We apply set algebra to Wordnet's synsets cooperating with Wordnet's word ontology. In this way we establish regular rules for constructing various representations (lexical notations) of a concept using Boolean operators and word forms in various synset(s) defined in Wordnet. Then we establish a formal mechanism for quantifying and estimating the semantic relatedness between concepts – we facilitate “concept distribution statistics” to determine the degree of semantic relatedness between two lexically expressed concepts. Human languages have words that can mean different things in different contexts, such words with multiple meanings are potentially “ambiguous”. The process of “deciding which of several meanings of a term is intended in a given context” is known as “Word Sense Disambiguation (WSD)”. The proposed method is not supervised, and does not require any manually created sense‐tagged training examples. The experimental results showed good performance on Semcor, a subset of the Brown corpus. We observe that measures of semantic relatedness are useful sources of information for word sense disambiguation.

Defect prediction in software projects plays a crucial role to reduce quality-based risk and increase the capability of detecting faulty program modules. Hence, classification approaches to anticipate software defect proneness based on static code characteristics have become a hot topic with a great deal of attention in recent years. While several novel studies show that the use of a single classifier causes the performance bottleneck, ensembles of classifiers might effectively enhance classification performance compared to a single classifier. However, the class imbalance property of software defect data severely hinders the classification efficiency of ensemble learning. To cope with this problem, resampling methods are usually combined into ensemble models. This paper empirically assesses the importance of sampling with regard to ensembles of various classifiers on imbalanced data in software defect prediction problems. Extensive experiments with the combination of seven different kinds of classification algorithms, three sampling methods, and two balanced data learning schemata were conducted over ten datasets. Empirical results indicated the positive effects of combining sampling techniques and the ensemble learning model on the performance of defect prediction regarding datasets with imbalanced class distributions.