Fine-grained Species Recognition with Privileged Pooling: Better Sample Efficiency Through Supervised Attention.

Industrial fault diagnosis based on active learning and semi-supervised learning using small training set

Knowledge transfer is widely held to be a primary mechanism that enables humans to quickly learn new complex concepts when given only small training sets. In this paper, we apply knowledge transfer to deep convolutional neural nets, which we argue are particularly well suited for knowledge transfer. Our initial results demonstrate that components of a trained deep convolutional neural net can constructively transfer information to another such net. Furthermore, this transfer is completed in such a way that one can envision creating a net that could learn new concepts throughout its lifetime. The experiments we performed involved training a Deep Convolutional Neural Net (DCNN) on a large training set containing 20 different classes of handwritten characters from the NIST Special Database 19. This net was then used as a foundation for training a new net on a set of 20 different character classes from the NIST Special Database 19. The new net would keep the bottom layers of the old net (i.e. those nearest to the input) and only allow the top layers to train on the new character classes. We purposely used small training sets for the new net to force it to rely as much as possible upon transferred knowledge as opposed to a large and varied training set to learn the new set of hand written characters. Our results show a clear advantage in relying upon transferred knowledge to learn new tasks when given small training sets, if the new tasks are sufficiently similar to the previously mastered one. However, this advantage decreases as training sets increase in size.

Cancer centers have an urgent and unmet clinical and research need for AI that can guide patient management. A core component of advancing cancer treatment research is assessing response to therapy. Doing so by hand, for example, as per RECIST or RANO criteria, is tedious and time-consuming, and can miss important tumor response information. Most notably, the prevalent response criteria often exclude lesions, the non-target lesions, altogether. We wish to assess change in a holistic fashion that includes all lesions, obtaining simple, informative, and automated assessments of tumor progression or regression. Because genetic sub-types of cancer can be fairly specific and patient enrollment in therapy trials is often limited in number and accrual rate, we wish to make response assessments with small training sets. Deep neuroevolution (DNE) is a novel radiology artificial intelligence (AI) optimization approach that performs well on small training sets. Here, we use a DNE parameter search to optimize a convolutional neural network (CNN) that predicts progression versus regression of metastatic brain disease. We analyzed 50 pairs of MRI contrast-enhanced images as our training set. Half of these pairs, separated in time, qualified as disease progression, while the other 25 image pairs constituted regression. We trained the parameters of a CNN via "mutations" that consisted of random CNN weight adjustments and evaluated mutation "fitness" as summed training set accuracy. We then incorporated the best mutations into the next generation's CNN, repeating this process for approximately 50,000 generations. We applied the CNNs to our training set, as well as a separate testing set with the same class balance of 25 progression and 25 regression cases. DNE achieved monotonic convergence to 100% training set accuracy. DNE also converged monotonically to 100% testing set accuracy. We have thus shown thatDNE can accurately classify brain metastatic disease progression versus regression. Future work will extend the input from 2D image slices to full 3D volumes, and include the category of "no change." We believe that an approach such as ours can ultimately provide a useful and informative complement to RANO/RECIST assessment and volumetric AI analysis.

Classification algorithms require training data initially labelled by classes to build a model and then to be able to classify the new data. The amount and diversity of training data affect the classification quality and usually the larger the training set, the better the accuracy of classification. In many applications only small amounts of training data are available. This article presents a new time series classification algorithm for problems with small training sets. The algorithm was tested on hand gesture recordings in tasks of person identification and gesture recognition. The algorithm provides significantly better classification accuracy than other machine learning algorithms. For 22 different hand gestures performed by 10 people and the training set size equal to 5 gesture execution records per class, the error rate for the newly proposed algorithm is from 37% to 75% lower than for the other compared algorithms. When the training set consists of only one sample per class the new algorithm reaches from 45% to 95% lower error rate. Conducted experiments indicate that the algorithm outperforms state-of-the-art methods in terms of classification accuracy in the problem of person identification and gesture recognition.

BERT is a pre-trained language model that achieves state-of-the-art performance on natural language processing (NLP) tasks. Once it was published, it quickly became one of the most popular models in the NLP field. The official recommended method for applying BERT to downstream tasks is fine-tuning. However, we argue that transfer learning is also a very practical approach to applying BERT, and this approach has its own advantages compared to fine-tuning. In this paper, we explore the advantages of the transfer learning approach through the method that uses transfer learning and fine-tuning approaches separately to apply BERT to the same eight GLUE benchmark tasks and compare the training time spent and model performance scores, such as accuracy or F1 score, of the two approaches. Finally, this paper finds that among all eight GLUE tasks and on small training sets, transfer learning saves 30% to 50% of the time compared to a fine-tuning approach to train the models on the data of downstream tasks and can achieve very similar performance. Besides, for the same amount of time, transfer learning can obtain higher performance scores than fine-tuning on larger training sets. In conclusion, on small training sets, transfer learning has a huge advantage in time consumption under the premise of approximate performance, and it also performs better on large training sets under the same length of time, compared to fine-tuning.

Co-training is a well known semi-supervised learning algorithm, in which two classifiers are trained on two different views (feature sets): the initially small training set is iteratively updated with unlabelled samples classified with high confidence by one of the two classifiers. In this paper we address an issue that has been overlooked so far in the literature, namely, how co-training performance is affected by the size of the initial training set, as it decreases to the minimum value below which a given learning algorithm can not be applied anymore. In this paper we address this issue empirically, testing the algorithm on 24 real datasets artificially splitted in two views, using two different base classifiers. Our results show that a very small training set, even made up of one only labelled sample per class, does not adversely affect co-training performance.

A major impediment to the application of deep learning to real-world problems is the scarcity of labeled data. Small training sets are in fact of no use to deep networks as, due to the large number of trainable parameters, they will very likely be subject to overfitting phenomena. On the other hand, the increment of the training set size through further manual or semi-automatic labellings can be costly, if not possible at times. Thus, the standard techniques to address this issue are transfer learning and data augmentation, which consists of applying some sort of “transformation” to existing labeled instances to let the training set grow in size. Although this approach works well in applications such as image classification, where it is relatively simple to design suitable transformation operators, it is not obvious how to apply it in more structured scenarios. Motivated by the observation that in virtually all application domains it is easy to obtain unlabeled data, in this paper we take a different perspective and propose a label augmentation approach. We start from a small, curated labeled dataset and let the labels propagate through a larger set of unlabeled data using graph transduction techniques. This allows us to naturally use (second-order) similarity information which resides in the data, a source of information which is typically neglected by standard augmentation techniques. In particular, we show that by using known game theoretic transductive processes we can create larger and accurate enough labeled datasets which use results in better trained neural networks. Preliminary experiments are reported which demonstrate a consistent improvement over standard image classification datasets.

Learning a visual object category from few samples is a compelling and challenging problem. In several real-world applications collecting many annotated data is costly and not always possible. However, a small training set does not allow to cover the high intraclass variability typical of visual objects. In this condition, machine learning methods provide very few guarantees. This paper presents a discriminative model adaptation algorithm able to proficiently learn a target object with few examples by relying on other previously learned source categories. The proposed method autonomously chooses from where and how much to transfer information by solving a convex optimization problem which ensures to have the minimal leave-one-out error on the available training set. We analyze several properties of the described approach and perform an extensive experimental comparison with other existing transfer solutions, consistently showing the value of our algorithm.

Text classification is important to better understand online media. A major problem for creating accurate text classifiers using machine learning is small training sets due to the cost of annotating them. On this basis, we investigated how SVM and NBSVM text classifiers should be designed to achieve high accuracy and how the training sets should be sized to efficiently use annotation labor. We used a four-way repeated-measures full-factorial design of 32 design factor combinations. For each design factor combination 22 training set sizes were examined. These training sets were subsets of seven public text datasets. We study the statistical variance of accuracy estimates by randomly drawing new training sets, resulting in accuracy estimates for 98,560 different experimental runs. Our major contribution is a set of empirically evaluated guidelines for creating online media text classifiers using small training sets. We recommend uni- and bi-gram features as text representation, btc term weighting and a linear-kernel NBSVM. Our results suggest that high classification accuracy can be achieved using a manually annotated dataset of only 300 examples.

Macroscopic pKa values were calculated for all compounds in the SAMPL6 blind prediction challenge, based on quantum chemical calculations with a continuum solvation model and a linear correction derived from a small training set. Microscopic pKa values were derived from the gas-phase free energy difference between protonated and deprotonated forms together with the Conductor-like Polarizable Continuum Solvation Model and the experimental solvation free energy of the proton. pH-dependent microstate free energies were obtained from the microscopic pKas with a maximum likelihood estimator and appropriately summed to yield macroscopic pKa values or microstate populations as function of pH. We assessed the accuracy of three approaches to calculate the microscopic pKas: direct use of the quantum mechanical free energy differences and correction of the direct values for short-comings in the QM solvation model with two different linear models that we independently derived from a small training set of 38 compounds with known pKa. The predictions that were corrected with the linear models had much better accuracy [root-mean-square error (RMSE) 2.04 and 1.95 pKa units] than the direct calculation (RMSE 3.74). Statistical measures indicate that some systematic errors remain, likely due to differences in the SAMPL6 data set and the small training set with respect to their interactions with water. Overall, the current approach provides a viable physics-based route to estimate macroscopic pKa values for novel compounds with reasonable accuracy.

Background. Usually the training set of online brain-computer interface (BCI) experiment is small. For the small training set, it lacks enough information to deeply train the classifier, resulting in the poor classification performance during online testing. Methods. In this paper, on the basis of Z-LDA, we further calculate the classification probability of Z-LDA and then use it to select the reliable samples from the testing set to enlarge the training set, aiming to mine the additional information from testing set to adjust the biased classification boundary obtained from the small training set. The proposed approach is an extension of previous Z-LDA and is named enhanced Z-LDA (EZ-LDA). Results. We evaluated the classification performance of LDA, Z-LDA, and EZ-LDA on simulation and real BCI datasets with different sizes of training samples, and classification results showed EZ-LDA achieved the best classification performance. Conclusions. EZ-LDA is promising to deal with the small sample size training problem usually existing in online BCI system.

Event-related potentials (ERPs) are one of the most popular control signals for brain-computer interfaces (BCIs). However, they are very weak and sensitive to the experimental settings including paradigms, stimulation parameters and even surrounding environments, resulting in a diversity of ERP patterns across different BCI experiments. It's still a challenge to develop a general decoding algorithm that can adapt to the ERP diversities of different BCI datasets with small training sets. This study compared a recently developed algorithm, i.e., discriminative canonical pattern matching (DCPM), with seven ERP-BCI classification methods, i.e., linear discriminant analysis (LDA), stepwise LDA, bayesian LDA, shrinkage LDA, spatial-temporal discriminant analysis (STDA), xDAWN and EEGNet for the single-trial classification of two private EEG datasets and three public EEG datasets with small training sets. The feature ERPs of the five datasets included P300, motion visual evoked potential (mVEP), and miniature asymmetric visual evoked potential (aVEP). Study results showed that the DCPM outperformed other classifiers for all of the tested datasets, suggesting the DCPM is a robust classification algorithm for assessing a wide range of ERP components.

BackgroundGenetic interaction profiles are highly informative and helpful for understanding the functional linkages between genes, and therefore have been extensively exploited for annotating gene functions and dissecting specific pathway structures. However, our understanding is rather limited to the relationship between double concurrent perturbation and various higher level phenotypic changes, e.g. those in cells, tissues or organs. Modifier screens, such as synthetic genetic arrays (SGA) can help us to understand the phenotype caused by combined gene mutations. Unfortunately, exhaustive tests on all possible combined mutations in any genome are vulnerable to combinatorial explosion and are infeasible either technically or financially. Therefore, an accurate computational approach to predict genetic interaction is highly desirable, and such methods have the potential of alleviating the bottleneck on experiment design.ResultsIn this work, we introduce a computational systems biology approach for the accurate prediction of pairwise synthetic genetic interactions (SGI). First, a high-coverage and high-precision functional gene network (FGN) is constructed by integrating protein-protein interaction (PPI), protein complex and gene expression data; then, a graph-based semi-supervised learning (SSL) classifier is utilized to identify SGI, where the topological properties of protein pairs in weighted FGN is used as input features of the classifier. We compare the proposed SSL method with the state-of-the-art supervised classifier, the support vector machines (SVM), on a benchmark dataset in S. cerevisiae to validate our method's ability to distinguish synthetic genetic interactions from non-interaction gene pairs. Experimental results show that the proposed method can accurately predict genetic interactions in S. cerevisiae (with a sensitivity of 92% and specificity of 91%). Noticeably, the SSL method is more efficient than SVM, especially for very small training sets and large test sets.ConclusionsWe developed a graph-based SSL classifier for predicting the SGI. The classifier employs topological properties of weighted FGN as input features and simultaneously employs information induced from labelled and unlabelled data. Our analysis indicates that the topological properties of weighted FGN can be employed to accurately predict SGI. Also, the graph-based SSL method outperforms the traditional standard supervised approach, especially when used with small training sets. The proposed method can alleviate experimental burden of exhaustive test and provide a useful guide for the biologist in narrowing down the candidate gene pairs with SGI. The data and source code implementing the method are available from the website: http://home.ustc.edu.cn/~yzh33108/GeneticInterPred.htm

Predicting coastal tsunami impact requires the computation of inundation metrics such as maximum inundation height or momentum flux at all locations of interest. The high computational cost of inundation modelling, in both long term tsunami hazard assessment and urgent tsunami computing, comes from two major factors: (1) the high number of simulations needed to capture the source uncertainty and (2) the need to solve the nonlinear shallow water equations on high-resolution grids. We seek to mitigate the second of these factors using machine learning. The offshore tsunami wave is far cheaper to calculate than the full inundation map, and an emulator able to predict an inundation map with acceptable accuracy from simulated offshore wave height time-series would allow both more rapid hazard estimates and the processing of greater numbers of scenarios. The procedure would necessarily be specific to one stretch of coastline and a complete numerical simulation is needed for each member of the training set. Success of an inundation emulator would demand an acceptable reduction in time-to-solution, a modest number of training scenarios, an acceptable accuracy in inundation predictions, and good performance for high impact, low probability, scenarios. We have developed a convolutional encoder-decoder based neural network and applied it to a dataset of high-resolution inundation simulations for the Bay of Catania in Sicily, calculated for almost 28000 subduction earthquake scenarios in the Mediterranean Sea. We demonstrate encouraging performance in this case study for relatively small training sets (of the order of several hundred scenarios) provided that appropriate choices are made in the setting of model parameters, the loss function, and training sets. Scenarios with severe inundation need to be very well represented in the training sets for the ML-models to perform sufficiently well for the most tsunamigenic earthquakes. The importance of regularization and model parameter choices increases as the size of the training sets decrease.

Convolutional Neural Network (CNN) tend to have better results on large data sets and poor performance on small data sets, so the data augmentation is crucial for a CNN to get better performance based on the dataset with limited size. In this paper, Deep Convolution Generative Adversarial Network (DCGAN) was used to augment data to make the AlexNet perform better on an image classification task with small data sets. AlexNet was trained on a small anime face training set with only 160 samples to determine whether the anime face was male or female, and then tested its accuracy on a test set with 240 sample. Then, a pre-trained DCGAN was transferred to train on the male and female training sets respectively. And 2 DCGANs were obtained, one could generate male cartoon faces and another could generate female cartoon faces. The images generated by DCGANs were put in train set, which was used to train AlexNet again and the result was recorded. Other data augmentation methods such as cutout, cutmix and Noise Injection were compared as well. Finally, it is found that AlexNet has the best performance when using the DCGAN augmentation method, which can significantly improve the verification accuracy of the model.