Combining active learning suggestions

Active learning algorithms propose what data should be labeled given a pool of unlabeled data. Instead of selecting randomly what data to annotate, active learning strategies aim to select data so as to get a good predictive model with as little labeled samples as possible. Single-shot batch active learners select all samples to be labeled in a single step, before any labels are observed. We study single-shot active learners that minimize generalization bounds to select a representative sample, such as the maximum mean discrepancy (MMD) active learner. We prove that a related bound, the discrepancy, provides a tighter worst-case bound. We study these bounds probabilistically, which inspires us to introduce a novel bound, the nuclear discrepancy (ND). The ND bound is tighter for the expected loss under optimistic probabilistic assumptions. Our experiments show that the MMD active learner performs better than the discrepancy in terms of the mean squared error, indicating that tighter worst case bounds do not imply better active learning performance. The proposed active learner improves significantly upon the MMD and discrepancy in the realizable setting and a similar trend is observed in the agnostic setting, showing the benefits of a probabilistic approach to active learning. Our study highlights that assumptions underlying generalization bounds can be equally important as bound-tightness, when it comes to active learning performance. Code for reproducing our experimental results can be found at https://github.com/tomviering/NuclearDiscrepancy.

Normally, statistical methods are used to generate rankings for genes in terms of their ability to distinguish between normal and malignant tumors from a gene expression dataset. However, different statistical methods yield different ranks for same gene and there is no universally accepted method for ranking. Therefore rank aggregation is required to find the overall ranking of the set of genes. There are various rank aggregation methods in the existing literature to integrate the rankings produced by various statistical tests. Moreover, the problem of integration of some partial rankings, containing unequal numbers of genes, is more challenging. In this article, a multiobjective genetic algorithm based rank aggregation method is proposed to integrate some partial rankings in an unbiased way. The first objective is to minimize the total distance from the reference ranking to the input rankings. For distance calculation, the Scaled Footrule Distance is used. The second objective is to minimize the standard deviation among those distances in order to avoid bias toward a particular input ranking. The proposed method is applied on some real-life microarray gene expression datasets, and the performance of it is compared with that of several existing rank aggregation techniques with respect to accuracy and the AUC (Area under ROC curve) value. Again, for real-life datasets, accuracy is plotted for visual comparison.

This paper presents novel approaches for combining re-ranking and rank aggregation methods aiming at improving the effectiveness of Content-Based Image Retrieval (CBIR) systems. Given a query image as input, CBIR systems retrieve the most similar images in a collection by taking into account image visual properties. In this scenario, accurately ranking collection images is of great relevance. Aiming at improving the effectiveness of CBIR systems, re-ranking and rank aggregation algorithms have been proposed. However, different re-ranking and rank aggregation approaches, applied to different image descriptors, may produce different and complementary image rankings. In this paper, we present four novel approaches for combining these rankings aiming at obtaining more effective results. Several experiments were conducted involving shape, color, and texture descriptors. The proposed approaches are also evaluated on multimodal retrieval tasks, considering visual and textual descriptors. Experimental results demonstrate that our approaches can improve significantly the effectiveness of image retrieval systems.

Students’ interest in Indonesian history as one of the subjects in the national curriculum is still low and has implications for the lack of students’ active learning. This study aims to describe and analyze the implementation of the picture and picture learning model in increasing students’ active learning in Indonesian History subjects during the Covid-19 pandemic. This research is a classroom action research which consists of two cycles. The source of student activity data is Class X of Fashion Design at SMK Negeri 1 Tarakan, totaling 37 students. The source of data on the implementation of the learning process is the teacher (researcher). The success indicator is based on increasing students’ active learning by implementing the 7-step picture and picture learning model. The results showed that implementing the picture and picture learning model, could increase the students’ active learning of Indonesian history in class X of Fashion Design at SMK Negeri 1 Tarakan. This research has implications for the importance of active, innovative, creative, and fun principles in learning as an effort to stimulate students’ active learning in order to optimize learning outcomes.

Active learning aims to reduce annotation cost by predicting which samples are useful for a human teacher to label. However it has become clear there is no best active learning algorithm. Inspired by various philosophies about what constitutes a good criteria, different algorithms perform well on different datasets. This has motivated research into ensembles of active learners that learn what constitutes a good criteria in a given scenario, typically via multi-armed bandit algorithms. Though algorithm ensembles can lead to better results, they overlook the fact that not only does algorithm efficacy vary across datasets, but also during a single active learning session. That is, the best criteria is non-stationary. This breaks existing algorithms' guarantees and hampers their performance in practice. In this paper, we propose dynamic ensemble active learning as a more general and promising research direction. We develop a dynamic ensemble active learner based on a non-stationary multi-armed bandit with expert advice algorithm. Our dynamic ensemble selects the right criteria at each step of active learning. It has theoretical guarantees, and shows encouraging results on 13 popular datasets.

Research characterises the child as an active learner who attends more to and selectively retains information they actively elicit better than information they passively receive. At the same time, children learn best from knowledgeable others who tailor information to children's learning progress. Bringing these disparate findings together requires examining children's active learning in social interactions. The current study examines whether the active learning advantage persists in social interactions with others and is influenced by the pedagogical status of their social partner (mother, father or friend). We tested 4‐ to 5‐year‐old children with their social partners ( N friend = 47, N mother = 44, N father = 53) during a word learning task using a novel setup where two participants can interact with visual objects on a transparent touchscreen while observing each other. Participants could either actively choose objects to hear their labels or passively observe their partner's choices. Early in the task, there was an overall active benefit, although this pattern appeared to be predominantly driven by interactions between peers. Later in the task, learning appeared to be dynamic and more influenced by the social partner with whom the child was interacting, especially when considering interactions with their peers and their fathers. Together, these findings underscore the temporal and social dynamics of an active learning benefit in children's social interactions.

The analytic hierarchy process (AHP) is a method to help people make a complex decision by analyzing and synthesizing multiple criteria for the decision objective in a hierarchy. We adopt the AHP as a new strategy for the semantic concept detection (SCD) in video so that multiple factors involved in SCD, including multiple modalities and relating concepts, can be hierarchically analyzed and synthesized. In this paper, we first explain, by an example, why and how the SCD problem can be analyzed by the AHP. Then, following the idea of the AHP, we develop a new rank aggregation (RA) method, called AHP-RA. Experimental results of RA for SCD in video show the effectiveness of this method.

Rank aggregation is the process of aggregating multiple base rankers into a single but more comprehensive ranker, which plays an important role in many domains such as recommender system, meta-search, database, genomics, etc. Works related to the comparison of rank aggregation methods all don't have a suitable and general data generation mechanism to produce data with various characteristics and lack a more reasonable and effective algorithm evaluation performance index.Therefore, this paper presents a general data generation mechanism based on Mallows model to produce synthetic controllable datasets, uses generalized Kendall rank correlation coefficient and rank-biased overlap to evaluate and compare the performance of two kinds of methods under different settings.Besides, we also consider the comparison between indices and the impact of data characteristics on the algorithms.This paper may be helpful to researchers and decision-makers from multiple domains.

Metasearch engine is a system that provides unified access to multiple existing search engines. After the results returned from all used component search engines are collected, the metasearch system merges the results into a single ranked list which is expected to be better than the results of the best of the participating search systems. The success of a metasearch engine depends mainly on their rank aggregation method. The system is a better one, if the aggregated list of results displayed before the user satisfies the user with his information need. In this paper, we discuss the development of a metasearch engine that performs user feedback based metasearching using modified rough set based aggregation. Metasearching using the modified rough set based aggregation is performed in two phases namely the ranking rule learning phase and the rank aggregation phase. For each query in the training set, we mine the ranking rules and select the best rules-set by performing cross-validation test. Once the system is trained, we use the best rule set to get the overall ranking for the results returned from different search systems in response to other queries. We also present few snapshots of our system.

In the past few years, segmentation and classification techniques have become a cornerstone of many successful remote sensing algorithms aiming at delineating geographic target objects. One common strategy relies on using multiple complex features to guide the delineation process with the objective of gathering complementary information for improving classification results. However, a persistent problem in this approach is how to combine different and noncorrelated feature descriptors automatically. In this regard, one solution is to combine them through multiple classifier systems (MCSs) in which the diversity of simple/noncomplex classifiers is an essential issue in the definition of appropriate strategies for classifier fusion. In this paper, we propose a novel strategy for selecting classifiers (whereby a classifier is taken as a pair of learning method plus image descriptor) to be combined in MCS. In the proposed solution, diversity measures are used to assess the degree of agreement/disagreement between pairs of classifiers and ranked lists are created to sort them according to their diversity score. Thereafter, the classifiers are also sorted according to their performance through different evaluation measures (e.g., kappa and tau indices). In the end, a rank aggregation method is proposed to select the most suitable classifiers based on both the diversity and the effectiveness performance of classifiers. The proposed fusion framework has targeted at coffee crop classification and urban recognition but it is general enough to be used in a variety of other pattern recognition problems. Experimental results demonstrate that the novel strategy yields good results when compared to several baselines while using fewer classifiers and being much more efficient.

Rank aggregation has recently become a common approach for combining individual rankings into a consensus and for quantifying and improving performance in various applications, such as elections, web page rankings, and sports. During the past few years, rankings from many sources have become increasingly high-dimensional and partial. In this study, we develop a rank aggregation method by constructing a directed weighted competition graph. We introduce the concept of “ratio of out- and in-degrees (ROID)” to transform high-dimensional partial rankings into a single consensus. Moreover, we provide a novel effectiveness measure for the aggregate ranking according to its deviations from the ground truth ranking. The proposed method is compared with four typical methods with synthetic rankings. The results indicate that our method outperforms the other four by a significant margin and can be particularly efficient in aggregating high-dimensional rankings. The empirical results validate the effectiveness and feasibility of our method.

How does experiential learning encourage active learning in auditing education?

Mislabelling is a critical problem for stream-based active learning methods because it not only impacts the classification accuracy but also deviates the active learner from querying informative data. Dealing with label noise is omitted by most existing active learning methods. We address this issue and propose an efficient method to identify and mitigate mislabelling errors for active learning in the streaming setting. We first propose a mislabelling likelihood measure to characterize the potentially mislabelled instances. This measure is based on the degree of disagreement among the predicted and the queried class label (given by the labeller). Then, we derive a measure of informativeness that expresses how much the label of an instance needs to be corrected by an expert labeller. Specifically, an instance is worth relabelling if it shows highly conflicting information among the predicted and the queried labels. We show that filtering instances with a high mislabelling likelihood and correcting only the filtered instances with a high conflicting information greatly improves the performances of the active learner. Experiments on several real world data prove the effectiveness of the proposed method in terms of filtering efficiency and classification accuracy of the stream-based active learner.

Several users use metasearch engines directly or indirectly to access and gather data from more than one data sources. The effectiveness of a metasearch engine is majorly determined by the quality of the results and it returns and in response to user queries. The rank aggregation methods which have been proposed until now exploits very limited set of parameters such as total number of used resources and the rankings they achieved from each individual resource. In this work, we use the neural network to merge the score computation module effectively. Initially, we give a query to different search engines and the top n list from each search engine is chosen for further processing our technique. We then merge the top n list based on unique links and we do some parameter calculations such as title based calculation, snippet based calculation, content based calculation, domain calculation, position calculation and co-occurrence calculation. We give the solutions of the calculations with user given ranking of links to the neural network to train the system. The system then rank and merge the links we obtain from different search engines for the query we give. Experimentation results reports a retrieval effectiveness of about 80%, precision of about 79% for user queries and about 72% for benchmark queries. The proposed technique also includes a response time of about 76 ms for 50 links and 144 ms for 100 links.

Teaching English to non-native speakers needs to incorporate up-to-date techniques and strategies that are frequently updated to match the ever-changing needs of modern society. In order for students to start acquiring English skills effectively inside the classroom, they need to be more active in their learning and be ready to take the initiative. For this to happen, lecturers need to adopt techniques that facilitate “active learning” with the suitable environment for students to make better use of their innate abilities and become active learners. In a classroom environment where active learning is prominent, students take responsibility of their learning of English as a second language with the lecturer acting as facilitator whose role is limited to providing instructions and supervising the process. Students will have more self-confidence to express their opinions and actively participate in topic discussions. This paper discusses methods of active learning that can be applied in teaching English at Cihan University.