SelfMatch: Robust semisupervised time‐series classification with self‐distillation

A semi-supervised deep learning approach for cropped image detection

Simple SummaryTracking the movements of chickens is important for understanding their well-being. Traditional methods for measuring chicken mobility are time-consuming and cannot provide real-time information. In this study, we, the researchers, used a combination of artificial intelligence methods and computer algorithms to track individual chickens. Using these methods, it was possible to detect and track individual chickens in two pens. We analyzed the data to see how far and how fast each chicken moved every hour and every day. Compared to manual measurements, the combined model provided more accurate measurements of the mobility of each chicken and the entire group, even when some chickens were hidden, or the detection was not perfect. This study may serve to effectively track indicators critical for broiler production performance and welfare.Mobility is a vital welfare indicator that may influence broilers’ daily activities. Classical broiler mobility assessment methods are laborious and cannot provide timely insights into their conditions. Here, we proposed a semi-supervised Deep Learning (DL) model, YOLOv5 (You Only Look Once version 5), combined with a deep sort algorithm conjoined with our newly proposed algorithm, neo-deep sort, for individual broiler mobility tracking. Initially, 1650 labeled images from five days were employed to train the YOLOv5 model. Through semi-supervised learning (SSL), this narrowly trained model was then used for pseudo-labeling 2160 images, of which 2153 were successfully labeled. Thereafter, the YOLOv5 model was fine-tuned on the newly labeled images. Lastly, the trained YOLOv5 and the neo-deep sort algorithm were applied to detect and track 28 broilers in two pens and categorize them in terms of hourly and daily travel distances and speeds. SSL helped in increasing the YOLOv5 model’s mean average precision (mAP) in detecting birds from 81% to 98%. Compared with the manually measured covered distances of broilers, the combined model provided individual broilers’ hourly moved distances with a validation accuracy of about 80%. Eventually, individual and flock-level mobilities were quantified while overcoming the occlusion, false, and miss-detection issues.

Residual placenta is one of the common types of postpartum complications in clinical practice. Residual placenta is also the main and most common cause of late postpartum hemorrhage. This article proposes a spatial pyramid loop module, which solves the problem that the existing network structure cannot effectively extract the semantic information and category information in the image at the same time. The spatial pyramid structure is used to effectively extract the semantic information and category information. In addition, this article proposes to use cyclic convolutional network to realize the transfer function of information at different scales, and build it in the spatial pyramid structure to further strengthen the ability to extract semantic information and category information. This article proposes a feature fusion module to solve the impact of image classification network used in the base network in the existing network structure. The attention mechanism is used to achieve the effective fusion of high-dimensional features and low-dimensional features in the base network to reduce the influence of the base network, so as to better recover the recognition and prediction results. A semantic category loss function is proposed to supervise the categories of objects in images. This article builds it on the feature layer with the smallest scale, which not only increases the intermediate supervision to make the network fully converge, but also reduces the difficulty of extracting category information, and makes full use of the information transfer function of the cyclic convolutional network. This article introduces uncertainty information into the field of image segmentation to provide the accuracy of segmentation. For the purpose of uncertainty information, this article improves the network structure. At the same time for the image segmentation task, this article improves the Bayesian cross entropy loss function. The experiment verifies the necessity of improving the Bayesian crossover function in this article and the effectiveness of the conditional random field used in this article, and also verifies the effectiveness of the proposed semi-supervised learning method.

Machine learning (ML), including deep learning (DL), has become increasingly popular in the last few years due to its continually outstanding performance. In this context, we apply machine learning techniques to "learn" the microstructure using both supervised and unsupervised DL techniques. In particular, we focus (1) on the localization problem bridging (micro)structure (localized) property using supervised DL and (2) on the microstructure reconstruction problem in latent space using unsupervised DL. The goal of supervised and semi-supervised DL is to replace crystal plasticity finite element model (CPFEM) that maps from (micro)structure (localized) property, and implicitly the (micro)structure (homogenized) property relationships, while the goal of unsupervised DL is (1) to represent high-dimensional microstructure images in a non-linear low-dimensional manifold, and (2) to discover a way to interpolate microstructures via latent space associating with latent microstructure variables. At the heart of this report is the applications of several common DL architectures, including convolutional neural networks (CNN), autoencoder (AE), and generative adversarial network (GAN), to multiple microstructure datasets, and the quest of neural architecture search for optimal DL architectures.

This paper proposes a semantic information extraction technology model system based on BERT, aiming to solve the problem of semantic understanding and information extraction in online public opinion analysis. The system design includes data collection, preprocessing, semantic information extraction and result display modules, using the BERT model to capture the deep semantic relationship of the text and enhance the ability to understand complex semantics and contextual information. The data collection module covers multiple sources such as social media, news comments and forum posts, and ensures the high quality and diversity of data through real-time crawling and cleaning. The semantic information extraction module is based on the multi-head attention mechanism of BERT, which can effectively capture the emotional tendency, topic information and potential intention in the public opinion text. The experiment verifies the system performance based on 1 million online public opinion data. The results show that in the semantic information extraction task, the accuracy of the BERT-based model is improved by 22.4%, the recall rate is improved by 19.8%, and the F1 value reaches 91.7<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%</sup>, which is significantly better than traditional methods such as TF-IDF and Word2Vec. The sentiment classification module has an accuracy of 93.1%, achieving accurate recognition of positive, negative and neutral emotions. The processing time of a single piece of public opinion text is reduced by an average of 25.6%, indicating the high efficiency of the system in large-scale data processing. The simulation test verifies the system's stable operation capability in the scenario of 100,000 new public opinion data per day, with a delay of less than 1 second, which can meet the needs of real-time public opinion analysis.

Ground penetrating radar (GPR) can be used for efficient and non-destructive detection of internal diseases in asphalt pavement, allowing for fast acquisition of a large amount of GPR image data. Currently, the identification of pavement diseases mainly depends on manual work, which significantly affects the efficiency of disease identification, and the identification results are prone to be affected by the operator’s personal opinions. This paper presents a method of identifying hidden diseases in asphalt pavement based on semi-supervised deep learning. Based on the YOLOv5 network, supervised learning was conducted on a limited number of existing samples to obtain the initial network model for disease identification. Then, the samples not subjected to manual labeling were identified and labeled, thus ensuring a balance in the types and the quantity of disease samples. Finally, a relatively comprehensive dataset was used for training to improve the identification precision of the intelligent model. The experimental results show that the overall precision of disease identification can reach 92% through the semi-supervised learning mode and the deep learning with expanded training dataset. The precision of identification results obtained from the semi-supervised learning model was improved by 24% compared to that obtained from the supervised learning model. The results also demonstrate that the new intelligent approach proposed in this paper can more effectively tap into the potential learning abilities of the model, thereby improving the precision of subgrade disease identification, and thus achieving accurate and efficient identification of pavement diseases.

For computer vision based appraoches such as image classification (Krizhevsky et al. 2012), object detection (Ren et al. 2015) or pixel-wise weed classification (Milioto et al. 2017) machine learning is used for both feature extraction and processing (e.g. classification or regression). Historically, feature extraction (e.g. PCA; Ch. 12.1. in Bishop 2006) and processing were sequential and independent tasks (Wöber et al. 2013). Since the rise of convolutional neuronal networks (LeCun et al. 1989), a deep machine learning approach optimized for images, in 2012 (Krizhevsky et al. 2012), feature extraction for image analysis became an automated procedure. A convolutional neuronal net uses a deep architecture of artificial neurons (Goodfellow 2016) for both feature extraction and processing. Based on prior information such as image classes and supervised learning procedures, parameters of the neuronal nets are adjusted. This is known as the learning process. Simultaneously, geometric morphometrics (Tibihika et al. 2018, Cadrin and Friedland 1999) are used in biodiversity research for association analysis. Those approaches use deterministic two-dimensional locations on digital images (landmarks; Mitteroecker et al. 2013), where each position corresponds to biologically relevant regions of interest. Since this methodology is based on scientific results and compresses image content into deterministic landmarks, no uncertainty regarding those landmark positions is taken into account, which leads to information loss (Pearl 1988). Both, the reduction of this loss and novel knowledge detection, can be done using machine learning. Supervised learning methods (e.g., neuronal nets or support vector machines (Ch. 5 and 6. in Bishop 2006)) map data on prior information (e.g. labels). This increases the performance of classification or regression but affects the latent representation of the data itself. Unsupervised learning (e.g. latent variable models) uses assumptions concerning data structures to extract latent representations without prior information. Those representations does not have to be useful for data processing such as classification and due to that, the use of supervised and unsupervised machine learning and combinations of both, needs to be chosen carefully, according to the application and data. In this work, we discuss unsupervised learning algorithms in terms of explainability, performance and theoretical restrictions in context of known deep learning restrictions (Marcus 2018, Szegedy et al. 2014, Su et al. 2017). We analyse extracted features based on multiple image datasets and discuss shortcomings and performance for processing (e.g. reconstruction error or complexity measurement (Pincus 1997)) using the principal component analysis (Wöber et al. 2013), independent component analysis (Stone 2004), deep neuronal nets (auto encoders; Ch. 14 in Goodfellow 2016) and Gaussian process latent variable models (Titsias and Lawrence 2010, Lawrence 2005).

Semi-supervised deep transfer learning for the microstructure recognition in the high-throughput characterization of nickel-based superalloys

Recently, deep learning has been highly successful in image classification. Labeling the PolSAR data, however, is time-consuming and laborious and in response semi-supervised deep learning has been increasingly investigated in PolSAR image classification. Semi-supervised deep learning methods for PolSAR image classification can be broadly divided into two categories, namely pixels-based methods and superpixels-based methods. Pixels-based semi-supervised methods are liable to be affected by speckle noises and have a relatively high computational complexity. Superpixels-based methods focus on the superpixels and ignore tiny detail-preserving represented by pixels. In this paper, a Fuzzy superpixels based Semi-supervised Similarity-constrained CNN (FS-SCNN) is proposed. To reduce the effect of speckle noises and preserve the details, FS-SCNN uses a fuzzy superpixels algorithm to segment an image into two parts, superpixels and undetermined pixels. Moreover, the fuzzy superpixels algorithm can also reduce the number of mixed superpixels and improve classification performance. To exploit unlabeled data effectively, we also propose a Similarity-constrained Convolutional Neural Network (SCNN) model to assign pseudo labels to unlabeled data. The final training set consists of the initial labeled data and these pseudo labeled data. Three PolSAR images are used to demonstrate the excellent classification performance of the FS-SCNN method with data of limited labels.

We propose a novel BIRADS-SSDL network that integrates clinically-approved breast lesion characteristics (BIRADS features) into task-oriented semi-supervised deep learning (SSDL) for accurate diagnosis of ultrasound (US) images with a small training dataset. Breast US images are converted to BIRADS-oriented feature maps (BFMs) using a distance-transformation coupled with a Gaussian filter. Then, the converted BFMs are used as the input of an SSDL network, which performs unsupervised stacked convolutional auto-encoder (SCAE) image reconstruction guided by lesion classification. This integrated multi-task learning allows SCAE to extract image features with the constraints from the lesion classification task, while the lesion classification is achieved by utilizing the SCAE encoder features with a convolutional network. We trained the BIRADS-SSDL network with an alternative learning strategy by balancing the reconstruction error and classification label prediction error. To demonstrate the effectiveness of our approach, we evaluated it using two breast US image datasets. We compared the performance of the BIRADS-SSDL network with conventional SCAE and SSDL methods that use the original images as inputs, as well as with an SCAE that use BFMs as inputs. The experimental results on two breast US datasets show that BIRADS-SSDL ranked the best among the four networks, with a classification accuracy of around 94.23 ± 3.33% and 84.38 ± 3.11% on two datasets. In the case of experiments across two datasets collected from two different institutions/and US devices, the developed BIRADS-SSDL is generalizable across the different US devices and institutions without overfitting to a single dataset and achieved satisfactory results. Furthermore, we investigate the performance of the proposed method by varying the model training strategies, lesion boundary accuracy, and Gaussian filter parameters. The experimental results showed that a pre-training strategy can help to speed up model convergence during training but with no improvement of the classification accuracy on the testing dataset. The classification accuracy decreases as the segmentation accuracy decreases. The proposed BIRADS-SSDL achieves the best results among the compared methods in each case and has the capacity to deal with multiple different datasets under one model. Compared with state-of-the-art methods, BIRADS-SSDL could be promising for effective breast US computer-aided diagnosis using small datasets.

This paper studies the classification problem of hyperspectral image (HSI). Inspired by the great success of deep neural networks in Artificial Intelligence (AI), researchers have proposed different deep learning based algorithms to improve the performance of hyperspectral classification. However, deep learning based algorithms always require a large-scale annotated dataset to provide sufficient training. To address this problem, we propose a semi-supervised deep learning framework based on the residual networks (ResNets), which use very limited labeled data supplemented by abundant unlabeled data. The core of our framework is a novel dual-strategy sample selection co-training algorithm, which can successfully guide ResNets to learn from the unlabeled data by making full use of the complementary cues of the spectral and spatial features in HSI classification. Experiments on the benchmark HSI dataset and real HSI dataset demonstrate that, with a small number of training data, our approach achieves competitive performance with maximum improvement of 41% (compare with traditional convolutional neural network (CNN) with 5 initial training samples per class on Indian Pines dataset) for HSI classification as compared with the results from those state-of-the-art supervised and semi-supervised methods.

Understanding the biology underpinning the natural regeneration of plant species in order to make plans for effective reforestation is a complex task. This can be aided by providing access to databases that contain long-term and wide-scale geographical information on species distribution, habitat, and reproduction. Although there exists widely-used biodiversity databases that contain structured information on species and their occurrences, such as the Global Biodiversity Information Facility (GBIF) and the Atlas of Living Australia (ALA), the bulk of knowledge about biodiversity still remains embedded in textual documents. Unstructured information can be made more accessible and useful for large-scale studies if there are tools and services that automatically extract meaningful information from text and store it in structured formats, e.g., open biodiversity databases, ready to be consumed for analysis (Thessen et al. 2022). We aim to enrich biodiversity occurrence databases with information on species reproductive condition and habitat, derived from text. In previous work, we developed unsupervised approaches to extract related habitats and their locations, and related reproductive condition and temporal expressions (Gabud and Batista-Navarro 2018). We built a new unsupervised hybrid approach for relation extraction (RE), which is a combination of classical rule-based pattern-matching methods and transformer-based language models that framed our RE task as a natural language inference (NLI) task. Using our hybrid approach for RE, we were able to extract related biodiversity entities from text even without a large training dataset. In this work, we implement an information extraction (IE) pipeline comprised of a named entity recognition (NER) tool and our hybrid relation extraction (RE) tool. The NER tool is a transformer-based language model that was pretrained on scientific text and then fine-tuned using COPIOUS (Conserving Philippine Biodiversity by Understanding big data; Nguyen et al. 2019), a gold standard corpus containing named entities relevant to species occurrence. We applied the NER tool to automatically annotate geographical location, temporal expression and habitat information contained within sentences. A dictionary-based approach is then used to identify mentions of reproductive conditions in text (e.g., phrases such as "fruited heavily" and "mass flowering"). We then use our hybrid RE tool to extract reproductive condition - temporal expression and habitat - geographical location entity pairs. We test our IE pipeline on the forestry compendium available in the CABI Digital Library (Centre for Agricultural and Biosciences International), and show that our work enables the enrichment of descriptive information on reproductive and habitat conditions of species. This work is a step towards enhancing a biodiversity database with the inclusion of habitat and reproductive condition information extracted from text.

With the rapid development of rail transit, the health status of trackside equipment is more and more crucial to the operation of rail transit. The use of high-precision anomaly detection methods can quickly detect the abnormal conditions of trackside equipment, ensuring the safety of rail transit. In this paper, a new framework of trackside equipment anomaly detection based on semi-supervised and multi-domain deep learning is proposed, which effectively improves the efficiency and accuracy of anomaly detection. Considering the inconspicuous changes in trackside equipment, including screw missing and cable corrosion, it is not feasible to directly use machine learning method for anomaly detection. Therefore, we first match the collected data to get the difference image. Secondly, considering the sparsity of the difference image, the incremental principal component analysis method is used to reduce the dimension of the difference image. Finally, in view of the limited number of training samples and the high cost of manual labeling, we combine semi-supervised deep learning and multi-domain deep learning to effectively train the anomaly detection model. Experimental results show that the proposed method has higher detection accuracy than the traditional artificial anomaly detection and anomaly detection based on shallow image alignment methods.

Recent advances in deep learning have made a progressive impact on BCI researches. In particular, convolutional neural networks (CNNs) with different architectural forms have been studied for spatio-temporal or spatio-spectral feature representation learning. However, there still remain many challenges and limitations due to the necessity of a large annotated training samples for robustness. In this paper, we propose a semi-supervised deep adversarial learning framework that effectively utilizes generated artificial samples along with labeled and unlabelled real samples in discovering class-discriminative features to boost robustness of a classifier, thus to enhance BCI performance. It is also noteworthy that the proposed framework allows to exploit unlabelled real samples to better uncover the underlying patterns inherent in a user’s EEG signals. In order to justify the validity of the proposed framework, we conducted exhaustive experiments with ‘Recurrent Spatio-Temporal Neural Network’ CNN architectures over the public BCI Competition IV-IIa dataset. From our experiments, we could observe statistically significant improvements on performance, compared to the competing methods with the conventional framework. We have also visualized learned convolutional weights in terms of activation pattern maps, separability of extracted features, and validity of generated artificial samples.

Semi-supervised deep learning for recognizing construction activity types from vibration monitoring data