Image Captioning Method Based on CLIP-Combined Local Feature Enhancement and Multi-Scale Semantic Guidance

Decision letter: Visual and motor signatures of locomotion dynamically shape a population code for feature detection in Drosophila

Point cloud completion using multiscale feature fusion and cross-regional attention

Object based video retrieval with local region tracking

This study analyzes the effectiveness of the global (the whole face) and local (regions of eyes, nose, and mouth) features for face recognition. Features describing human faces are encoded in local ternary patterns. The two-class support vector machine is used as the supervised learning algorithm for training recognition models. In the recognition process, recognition modes based on the global features and local features are cascaded. For identifying a face image, the local features are used iteratively for filtering out candidates that can not be clearly identified by the global features, until the one with highest possibility is concluded. The experimental results show that cascading the recognition models of global and local features obtains better classification accuracy than the single classification process.

Learning discriminative shape representation directly on point clouds is still challenging in 3D shape analysis and understanding. Recent studies usually involve three steps: first splitting a point cloud into some local regions, then extracting the corresponding feature of each local region, and finally aggregating all individual local region features into a global feature as shape representation using simple max-pooling. However, such pooling-based feature aggregation methods do not adequately take the spatial relationships (e.g. the relative locations to other regions) between local regions into account, which greatly limits the ability to learn discriminative shape representation. To address this issue, we propose a novel deep learning network, named Point2SpatialCapsule, for aggregating features and spatial relationships of local regions on point clouds, which aims to learn more discriminative shape representation. Compared with the traditional max-pooling based feature aggregation networks, Point2SpatialCapsule can explicitly learn not only geometric features of local regions but also the spatial relationships among them. Point2SpatialCapsule consists of two main modules. To resolve the disorder problem of local regions, the first module, named geometric feature aggregation, is designed to aggregate the local region features into the learnable cluster centers, which explicitly encodes the spatial locations from the original 3D space. The second module, named spatial relationship aggregation, is proposed for further aggregating the clustered features and the spatial relationships among them in the feature space using the spatial-aware capsules developed in this paper. Compared to the previous capsule network based methods, the feature routing on the spatial-aware capsules can learn more discriminative spatial relationships among local regions for point clouds, which establishes a direct mapping between log priors and the spatial locations through feature clusters. Experimental results demonstrate that Point2SpatialCapsule outperforms the state-of-the-art methods in the 3D shape classification, retrieval and segmentation tasks under the well-known ModelNet and ShapeNet datasets.

Change detection via remote sensing data is a popular method for monitoring land cover/land use. Graph attention network (GAT) is a method that can improve the change detection performance of land cover/land use monitoring by enhancing the feature representation of remote sensing images. However, the shortcomings of connection sparsity and insufficient sample feature mining of the GAT affect the application of these methods in change detection. This paper proposes a Siamese autoencoder graph attention networks for object-oriented land cover/land use change detection via high-resolution remote sensing, which is useful for semi-supervised problem methods with poor simplicity. First, we reduce the pressure of graph network model operations and obtain multidimensional features via the growing multilevel segmentation strategy (GMSS). The adjacency matrix is established by adding strongly connected edges via the weighted difference similarity matching method (WDSM) with image objects as nodes. Second, we use the Siamese autoencoder to pretrain the node features and migrate the weight parameters to the GAT feature extraction layer. Finally, a small number of samples are selected to train the GAT and predict all nodes. The experimental results show that the average overall accuracy of the proposed method is 95.41%, and the average F1 score is 89.35%, which are at least 5.04% and 10.84% better than those of other typical methods, such as the graph convolutional network (GCN). In particular, detecting roads and bare soil is significantly better than that of other methods.

Local image features around interest-points have been widely used in order to exploit the similarities between different views of an object in different images. While there are numerous algorithms on detecting the interest-points and defining the local features, few have focused on the importance of the matching process. In this paper, we presented a method that matches interest-points detected via any algorithm. The method is motivated from human perceptual rules, particularly the Gestalt Psychology, and realizes the fact that “The whole is different from the sum of its parts”. The efficacy of the algorithm is not only the ability to decrease the number of false positive matches but also to increase the number of true positives, yielding rock-steady results for any algorithm based on matching local features.

Aging is a gradual process, and it is difficult to see changes in the face as a whole, but signs of aging can be found in local areas. Therefore, fusing local face features on the basis of global features can improve the accuracy of age prediction. This paper proposes an age prediction method based on the combination of global facial features and local facial features. First, the local regions such as eyes, nose and mouth are cropped from the pre-processed facial image. Then, feature extraction is performed on the facial image and the cropped local area image, and the extracted features are subjected to feature selection, standardization and PCA dimension reduction. The processed features are then used for age prediction using multi-class support vector machines and neural networks. Experimental results show that this method outperforms previous methods.

Most of existing local-based person re-identification (re-ID) methods extract powerful feature representations from multiple body regions of a pedestrian. However, local-based person re-ID methods employ horizontal or grid slicing to divide pedestrian images, which can easily cause network redundancy and misalignment of pedestrian body parts. Attention mechanisms have been recently introduced to computer vision tasks to strengthen local salient features and suppress irrelevant features simultaneously. However, attention mechanisms overemphasize local salient features, which may lead to potential significant information being missed. To address the above issues, we proposed a local information interaction enhancement network (LIEN) for person re-ID. The network integrates attention mechanism with local-based strategy to achieve mutually facilitating and complementary effects. Specifically, LIEN was composed of three parts: a backbone network, a dual attention module (DAM), and a local information interaction module (LIM). DAM was used to guide the backbone network to extract the local salient information, and LIM was used to mine the potential significant information that DAM ignored by exploring the correlations between different channel-level local features. With the combination of DAM and LIM, LIEN mined the salient features while obtaining more potential effective features. In particular, the way that it extracts local features, LIEN effectively avoided pedestrian misalignment caused by uniformly dividing pedestrian images. We evaluated the performance of LIEN on public person datasets, and experimental results indicated that LIEN achieved a more advanced performance.

The vast growth of image databases creates many challenges for computer vision applications, for instance image retrieval and object recognition. Large variation in imaging conditions such as illumination and geometrical properties (including scale, rotation, and viewpoint) gives rise to the need for invariant features; i.e. image features should have minimal differences under these conditions. Local image features in the form of key points are widely used because of their invariant properties. In this chapter, we analyze different issues relating to existing local feature detectors. Based on this analysis, we present a new approach for detecting and filtering local features. The proposed approach is tested in a real-life application which supports navigation in urban environments based on visual information. The study shows that our approach performs as well as existing methods but with a significantly lower number of features.KeywordsFeature VectorLocal FeatureInterest PointQuery ImageImage RecognitionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A multi-scale semantic attention representation for multi-label image recognition with graph networks

Protein abnormalities disrupt various cellular and contribute to disease development. Identifying disease-associated proteins is crucial for precision medicine, but traditional methods are time-consuming and costly, necessitating computational approaches. Existing computational methods rely on manual feature engineering and fail to leverage deep features from amino acid sequences and protein structures. In this article, we propose Model for predicting protein-phenotype associations by Fusing multi-view Features (MFF-HPO), a model for predicting protein-phenotype associations by fusing multi-view features from amino acid sequences. First, we generate three-dimensional protein structure from amino acid sequence to derive contact graphs and secondary structures then integrate these with direct sequence encoding and physicochemical properties. Using a Graph Attention Network, we extract structural features from contact graphs, while deep neural networks capture global and local features from secondary structures, physicochemical properties, and sequence encoding. Finally, concatenated features are used to predict phenotype annotations. MFF-HPO outperforms state-of-the-art methods with a mean area under the precision-recall curve of 0.314 and a mean Fmax of 0.371. Ablation studies confirm that multi-view feature fusion enhances predictions, and case studies validate its practicality.

Cooperative Multi-Agent Reinforcement Learning (MARL) relies on effective coordination among agents to maximize team performance in complex environments. However, existing coordination graph-based approaches often overlook dynamic group structures and struggle to accurately capture fine-grained inter-agent dependencies. In this paper, we introduce a novel method called the Group Attention Aware Coordination Graph (G2ACG), which builds upon the group modeling capabilities of the Group-Aware Coordination Graph (GACG). G2ACG incorporates a dynamic attention mechanism to dynamically compute edge weights in the coordination graph, enabling a more flexible and fine-grained representation of agent interactions. These learned edge weights guide a Graph Attention Network (GAT) to perform message passing and representation learning, and the resulting features are integrated into a global policy via QMIX for cooperative decision-making. Experimental results on the StarCraft II Multi-Agent Challenge (SMAC) benchmark show that G2ACG consistently outperforms strong baselines, including QMIX, DICG, and GACG, across various scenarios with diverse agent types and population sizes. Ablation studies further confirm the effectiveness of the proposed attention mechanism, demonstrating that both the number of attention heads and the number of GAT layers significantly affect performance, with a two-layer GAT and multi-head attention configuration yielding the best results.

Joint and collaborative representation with local adaptive convolution feature for face recognition with single sample per person

Biometric features such as fingerprints, iris patterns, and face features help to identify people and restrict access to secure areas by performing advanced pattern analysis and matching. Face recognition is one of the most promising biometric methodologies for human identification in a non-cooperative security environment. However, the recognition results obtained by face recognition systems are a affected by several variations that may happen to the patterns in an unrestricted environment. As a result, several algorithms have been developed for extracting different facial features for face recognition. Due to the various possible challenges of data captured at different lighting conditions, viewing angles, facial expressions, and partial occlusions in natural environmental conditions, automatic facial recognition still remains as a difficult issue that needs to be resolved. In this paper, we propose a novel approach to tackling some of these issues by analyzing the local textural descriptions for facial feature representation. The textural information is extracted by an enhanced local binary pattern (ELBP) description of all the local regions of the face. The relationship of each pixel with respect to its neighborhood is extracted and employed to calculate the new representation. ELBP reconstructs a much better textural feature extraction vector from an original gray level image in different lighting conditions. The dimensionality of the texture image is reduced by principal component analysis performed on each local face region. Each low dimensional vector representing a local region is now weighted based on the significance of the sub-region. The weight of each sub-region is determined by employing the local variance estimate of the respective region, which represents the significance of the region. The final facial textural feature vector is obtained by concatenating the reduced dimensional weight sets of all the modules (sub-regions) of the face image. Experiments conducted on various popular face databases show promising performance of the proposed algorithm in varying lighting, expression, and partial occlusion conditions. Four databases were used for testing the performance of the proposed system: Yale Face database, Extended Yale Face database B, Japanese Female Facial Expression database, and CMU AMP Facial Expression database. The experimental results in all four databases show the effectiveness of the proposed system. Also, the computation cost is lower because of the simplified calculation steps. Research work is progressing to investigate the effectiveness of the proposed face recognition method on pose-varying conditions as well. It is envisaged that a multilane approach of trained frameworks at different pose bins and an appropriate voting strategy would lead to a good recognition rate in such situation.