Target Tracking Based on SE-CNN and SiameseNet

The lack of training samples remains one of the major obstacles in applying convolutional neural networks (CNNs) to the hyperspectral image (HSI) classification. In this letter, the accurate classification of HSI with limited training samples is investigated. Due to the advantages of minimizing the distance between samples in the same class and maximizing the distance between samples in different classes, siamese CNN is used for HSI classification with limited training samples. After that, to improve the classification performance, data augmentation is investigated for siamese CNN-based HSI classification. Specifically, pair data augmentation based on CutMix is proposed to generate the training pairs of the same or different classes and the new generated training pairs are used to train siamese CNN. Traditional data augmentation methods simply generate new training samples. It is not proper when data augmentation methods keep the loss function and change the content of the input at the same time. Therefore, a soft-loss-based siamese CNN, which changes its loss according to the coupled replacement data augmentation, is proposed to further address the HSI classification with limited training samples. In the experimental part, on two widely used hyperspectral datasets, the influences of different training samples and window sizes are discussed, and the experimental results reveal that the proposed soft-augmentation-based siamese CNN provides competitive results with limited training samples compared with state-of-the-art methods.

Face verification is an important task to verify people’s identities, which has sample pairs and labels with only side information about whether the two images in a pair are from the same subject or not instead of a specific label for each sample. This paper is motivated by the limitations of previous efforts that the deep neural networks with only the RGB feature are not to intra-class noise (e.g., illumination), and the deep networks with both the RGB and depth feature are not to data availability of the depth images in applications. To overcome these limitations, we focus on developing a double robust face verification architecture using the Siamese convolutional neural network (SCNN). In particular, our goal is to propose an SCNN with privileged information (SCNN+), which is inspired by Learning Using Privileged Information (LUPI), through incorporating additional depth feature along with the RGB feature in the training stage to provide privileged information to restrain the RGB feature’s prediction error. We conduct experiments to evaluate the performance of the proposed SCNN+ architecture and compare it with different categories of state-of-the-art baselines on two real-world RGB-D face datasets. The evaluation results demonstrate that SCNN+ significantly outperforms all types of baselines.

This work proposes a framework to train and deploy neural network-based gesture recognition algorithms in wearable devices. The approach is demonstrated for a high-density electromyography (HD-EMG) gesture recognition system, where a Siamese convolution neural network (SCNN) learns to associate and dissociate muscle activity patterns from the same or distinct gesture classes. This optimizes learning in low-data environments such as gesture recognition and myoelectric control, where training data must be provided by the end user. Then, using a cosine similarity-based few-shot classifier and inter-session-intra-user transfer of the SCNN’s learning, the proposed model is intended to achieve state-of-the-art results in a framework that is realistically applicable in wearable devices. For an experimented myoelectric interface user, the proposed model achieved 86.49 % accuracy in 6-class gesture recognition, using inter-session-intra-user transfer learning by the SCNN. Few-shot learning demonstrates convergence to this accuracy within 20 shots with the cosine similarity classifier. Across a group of 13 able-bodied participants, 90.66 % and 97.34 % mean and median accuracy were obtained from the 6-way 20-shot learning approach with a majority vote (100 votes) over time-distributed inferences.

Since hyperspectral images (HSI) captured by different sensors often contain different number of bands, but most of the convolutional neural networks (CNN) require a fixed-size input, the generalization capability of deep CNNs to use heterogeneous input to achieve better classification performance has become a research focus. For classification tasks with limited labeled samples, the training strategy of feeding CNNs with sample-pairs instead of single sample has proven to be an efficient approach. Following this strategy, we propose a Siamese CNN with three-dimensional (3D) adaptive spatial-spectral pyramid pooling (ASSP) layer, called ASSP-SCNN, that takes as input 3D sample-pair with varying size and can easily be transferred to another HSI dataset regardless of the number of spectral bands. The 3D ASSP layer can also extract different levels of 3D information to improve the classification performance of the equipped CNN. To evaluate the classification and generalization performance of ASSP-SCNN, our experiments consist of two parts: the experiments of ASSP-SCNN without pre-training and the experiments of ASSP-SCNN-based transfer learning framework. Experimental results on three HSI datasets demonstrate that both ASSP-SCNN without pre-training and transfer learning based on ASSP-SCNN achieve higher classification accuracies than several state-of-the-art CNN-based methods. Moreover, we also compare the performance of ASSP-SCNN on different transfer learning tasks, which further verifies that ASSP-SCNN has a strong generalization capability.

In recent years, deep learning-based methods have achieved great success in remote sensing image analysis. However, especially in the context of hyperspectral image classification, there is still a lack of labelled samples to feed those data-hungry deep models. To augment the amount of input data, models operate on pixel-pairs have been proposed and Siamese convolutional neural network (S-CNN) is a typical one. S-CNN is used as a pixel-pair feature extractor and an additional classifier like SVM is required. In this paper, we propose an end-to-end version of S-CNN. Taking advantage of the pairwise input and to make better use of spatial information, a voting strategy using neighbouring pixels is also employed to determine the final class label of the centre pixel. Experimental results on real hyperspectral datasets show that the proposed method outperforms the original S-CNN by a considerable margin.

In this paper, a novel indoor localization system with channel state information (CSI) fingerprints is proposed, which learns the spatial and frequency features of CSI in the fifth-generation (5G) cellular network by a Siamese convolution neural network. In particular, considering that the CSI continuously collected by a moving target possesses the implicit spatial association, we locate the target by the successive CSI data gathered within a time interval which can be regarded as an information subspace of the fingerprint database. Therefore, the fingerprint localization can be modeled as a subspace matching problem and solved by the Siamese network-based similarity learning. In the proposed system, we design a structure of CSI fingerprint which includes the information from multiple base stations in spatial and frequency domains. Then, the proposed Siamese architecture extracts the CSI feature and estimates the location of the target by feature similarity comparison. Compared with the existing algorithms, it can increase the positioning accuracy significantly by the feature relevance among the CSI data collected at different positions. The field tests indicate that compared to other CSI fingerprint-based positioning methods, our proposed algorithm can effectively reduce the localization error.

With the rapid proliferation of smart mobile devices, users now take millions of photos every day. These include large numbers of clothing and accessory images. We would like to answer questions like 'What outfit goes well with this pair of shoes?' To answer these types of questions, one has to go beyond learning visual similarity and learn a visual notion of compatibility across categories. In this paper, we propose a novel learning framework to help answer these types of questions. The main idea of this framework is to learn a feature transformation from images of items into a latent space that expresses compatibility. For the feature transformation, we use a Siamese Convolutional Neural Network (CNN) architecture, where training examples are pairs of items that are either compatible or incompatible. We model compatibility based on co-occurrence in large-scale user behavior data, in particular co-purchase data from Amazon.com. To learn cross-category fit, we introduce a strategic method to sample training data, where pairs of items are heterogeneous dyads, i.e., the two elements of a pair belong to different high-level categories. While this approach is applicable to a wide variety of settings, we focus on the representative problem of learning compatible clothing style. Our results indicate that the proposed framework is capable of learning semantic information about visual style and is able to generate outfits of clothes, with items from different categories, that go well together.

An improved approach for medical image fusion using sparse representation and Siamese convolutional neural network

The similarity measure is a fundamental and key problem for time series analysis, which is widely used in classification, cluster, motif discovery and etc. Time series similarity calculation has been studied a lot and many methods have been proposed, such as Euclidean distance and elastic distance measures. However, all these methods are distance-based which get the similarity by accumulating the distances between optimal match pairs and ignore the intrinsic differences of time series. In order to solve the drawback, we discard this distance-based technique and design a Siamese Convolutional Neural Network(SCNN) to obtain the similarity of time series in this paper. Concretely, we first extract the essential features for similarity calculation by Convolutional Neural Network and then ontain the absolute difference between the essential features of time series. To conveniently train the proposed model, we make full use of the label information of datasets and construct a new binary dataset which each example contains two original time series and a binary label. In the proposed Siamese network, the parameters are shared between two branches and we take binary cross-entropy function to train our model. Experimental results on synthetic and real time series datasets show that our SCNN model is effective and achieves improved accuracy for time series classification task compared to other time series similarity calculation methods.

Semantical text understanding holds significant importance in natural language processing (NLP). Numerous datasets, such as Quora Question Pairs (QQP), have been devised for this purpose. In our previous study, we developed a Siamese Convolutional Neural Network (S-CNN) that achieved an F1 score of 82.02% (95% C.I.: 81.83%-82.20%). Given the growing attention toward large language models (LLMs) like ChatGPT, we aimed to explore their effectiveness in text similarity tasks. In this research, we leveraged 5 pretrained LLMs, conducted various fine-tuning approaches (prompt engineering, n-shot learning, and supervised learning using the low-rank adaptation [LoRA]), and compared their performance using F1 score. To ensure a fair comparison, we followed our previous study's design and dataset by employing a 10-fold cross-validation for supervised model training and evaluation. Additionally, we conducted a secondary study by introducing a recent larger LLM with 70B parameters and comparing it with the 7B model using the GLUE benchmark, and both models were finetuned with the corpus. The fine-tuned LLaMA model with 7B parameters (qLLaMA_LoRA-7B) using 100,000 QQP corpus yielded the best results, achieving an F1 score of 84.9% (95% C.I.: 84.13%-85.67%), which outperformed the Alpaca_LoRA-65B (finetuned based on LLaMA-65B) (F1: 64.98% [64.72%-65.25%]; P<0.01) and had a 3% improvement compared to our previously published best model, S-CNN. The finetuned LLaMA3.1-70B (qLLaMA3.1_LoRA-70B) with 70B parameters (F1: 74.4%) outperformed the qLLaMA_LoRA-7B (F1: 71.9%) using the GLUE benchmark. The study demonstrated an effective LLM finetuning framework, which highlights the importance of finetuning LLMs for improved performance. Our task-specific supervised finetuning demonstrated improved LLM performance compared to larger pretrained models with or without n-shot learning; moreover, finetuning a larger LLM further improved performance compared to finetuning a smaller LLM. Our LLM-based finetuning framework may potentially improve various document similarity tasks, such as matching resumes with job descriptions, recommending subject-matter experts, or identifying potential reviewers for grant proposals or manuscript submissions.

This work introduces a new approach to localize anomalies in surveillance video. The main novelty is the idea of using a Siamese convolutional neural network (CNN) to learn a distance function between a pair of video patches (spatio-temporal regions of video). The learned distance function, which is not specific to the target video, is used to measure the distance between each video patch in the testing video and the video patches found in normal training video. If a testing video patch is not similar to any normal video patch then it must be anomalous. We compare our approach to previously published algorithms using 4 evaluation measures and 3 challenging target benchmark datasets. Experiments show that our approach either surpasses or performs comparably to current state-of-the-art methods.

Automatic change detection is an important and difficult task in the field of remote sensing. In this study, a deep Siamese convolutional network based on the fusion of high- and low-level features is proposed for change detection in remote sensing images. Given that low-level features correspond to low-order ones (e.g., texture) that are sensitive to change and that high-level features can accurately reflect image category information (e.g., semantic information), we fuse these features to enhance the abstractness and robustness of the extracted features in the change detection framework. The whole system is end-to-end and does not require any pre- or post-processing. Experimental results on three datasets show that our method is superior to other advanced methods by adding a high- and low-level fusion framework.

Text similarity based on two independent channels: Siamese Convolutional Neural Networks and Siamese Recurrent Neural Networks

Offline signature verification is one of the most challenging tasks in biometric authentication. Despite recent advances in this field using image recognition and deep learning, there are many remaining things to be explored. The most recent technique, which is Siamese convolutional neural network, has been used a lot in this field and has achieved great results. This paper presents an architecture that combines the power of Siamese Triplet CNN and a fully-connected neural network for binary classification to automatically verify genuine and forgery signatures even if the forged signature is highly skilled. On the challenging public dataset for signature verification BHSig260, the proposed model can achieve a low False Acceptance Rate = 13.66, which is slightly better than the reference model. Based on this approach, the one-shot learning should make it possible to determine if the input image is genuine or fraudulent just from one base image. Therefore, our model is expected to be extremely suitable for practical problems, such as banking systems or mobile authentication applications, in which the amount of data for each identity is limited in quantity and variety.KeywordsOne-shot learningOffline signature verificationSiamese convolutional neural networkTriplet loss

paper, we focus on exploring the relationship between weave patterns and their mechanical properties in woven fiber composites through Machine Learning. Specifically, we explore the interactions between woven architectures and in-plane stiffness properties through Deep Convolutional Neural Network (DCNN) and Generative Adversarial Network (GAN). Our research is important for exploring how woven composite’s pattern is related to its mechanical properties and accelerating woven composite design as well as optimization. We focus on two tasks: (1) Stiffness prediction: Predicting in-plane stiffness properties for given weave patterns. Our DCNN extracts high-level features through several convolutional and fully connected layers to determine the final predictions. (2) Weave pattern prediction: Predicting weave patterns for target stiffness properties, which can be treated as the reverse task of the first one. Due to many-to-one mapping between weave patterns and the composite properties, we utilize a Decoder Neural Network as our baseline model and compare its performance with GAN and Genetic Algorithm. We represent the weave patterns as 2D checkerboard models and use finite element analysis (FEA) to determine in-plane stiffness properties, which serve as input data for our ML framework. We show that: (1) for stiffness prediction, DCNN can predict stiffness values for a given weave pattern with relatively high accuracy (above 93%); (2) for weave pattern prediction, the GAN model gives the best prediction accuracy (above 92%) while Decoder Neural Network has the best time efficiency. HAOTIAN FENG