Transformer-CNN hybrid network for crowd counting

Medical image classification is essential for contemporary clinical diagnosis and decision support systems. However, medical images generally have similar inter-class features and complex structure patterns, making it a challenging task. While both local and global features are critical for noise reduction and discriminative pattern extraction in medical images, conventional approaches exhibit limitations. Specifically, convolutional neural networks (CNNs) focus on local features extraction but lack a comprehensive understanding of global semantic. Conversely, vision transformers (ViTs) can model long-range feature dependencies but may cause disruption to local features. To address these limitations, we propose Hierarchical Enhanced Multi-attention Feature (HEMF), an adaptive hierarchical enhanced multi-attention feature fusion framework to synergistically extract and fuse multi-scale local and global features. It comprises two core components: (1) the enhanced local and global feature extraction modules to extract multi-scale local and global features in parallel; (2) the hierarchical enhanced feature fusion module integrating a novel attention mechanism named Mixed Attention (MA) and a novel inverted residual block named Squeezed Inverted Residual Multi-Layer Perceptron (SIRMLP) to effectively fuse multi-scale features. Experimental results demonstrate that with nearly minimal model parameters compared to other advanced models, HEMF achieves the accuracy and F1-score of 87.34% and 78.89% on the ISIC2018 dataset, 87.03% and 87.02% on the Kvasir dataset, and 82.26% and 82.20% on the COVID-19 CT dataset, which are the state-of-the-art performance. Our code is open source and available from https://github.com/Esgjgd/HEMF.

LocalFace: Learning significant local features for deep face recognition

Learning a deep network with cross-hierarchy aggregation for crowd counting

In recent years, significant progress has been made in crowd counting with the development of convolutional neural networks (CNNs). However, while CNNs excel at extracting local features, their limited receptive fields restrict their ability to model global context. In contrast, Transformers can effectively model long-distance dependencies, but are inferior to CNN in capturing local detail features. Local details and global context information are crucial to handle large-scale changes in crowds. To address this problem, we propose a novel dual backbone network (DBNet) that integrates CNN and Transformer architectures, aiming to capture and aggregate both global semantic information and local detail features at multiple levels. Specifically, the dual backbone structure is designed to extract fine-grained local features while modeling long-range contextual relationships. Additionally, we introduce a multi-attention hierarchical fusion module (MAHF) that integrates global and local features from the two backbones while suppressing background noise. To further enhance accuracy in the presence of multi-scale variations, we also employ a Feature Enhancement Module (FEM), which enables the network to more effectively identify edge features and facilitates more effective multi-scale feature modeling. Extensive experiments on ShanghaiTech, UCF-QNRF, and JHU-Crowd++ datasets demonstrate that DBNet achieves competitive performance, validating the effectiveness of our approach.

B-cell epitopes (BCEs) are a set of specific sites on the surface of an antigen that binds to an antibody produced by B-cell. The recognition of BCEs is a major challenge for drug design and vaccines development. Compared with experimental methods, computational approaches have strong potential for BCEs prediction at much lower cost. Moreover, most of the currently methods focus on using local information around target residue without taking the global information of the whole antigen sequence into consideration. We propose a novel deep leaning method through combing local features and global features for BCEs prediction. In our model, two parallel modules are built to extract local and global features from the antigen separately. For local features, we use Graph Convolutional Networks (GCNs) to capture information of spatial neighbors of a target residue. For global features, Attention-Based Bidirectional Long Short-Term Memory (Att-BLSTM) networks are applied to extract information from the whole antigen sequence. Then the local and global features are combined to predict BCEs. The experiments show that the proposed method achieves superior performance over the state-of-the-art BCEs prediction methods on benchmark datasets. Also, we compare the performance differences between data with or without global features. The experimental results show that global features play an important role in BCEs prediction. Our detailed case study on the BCEs prediction for SARS-Cov-2 receptor binding domain confirms that our method is effective for predicting and clustering true BCEs.

Most of existing local-based person re-identification (re-ID) methods extract powerful feature representations from multiple body regions of a pedestrian. However, local-based person re-ID methods employ horizontal or grid slicing to divide pedestrian images, which can easily cause network redundancy and misalignment of pedestrian body parts. Attention mechanisms have been recently introduced to computer vision tasks to strengthen local salient features and suppress irrelevant features simultaneously. However, attention mechanisms overemphasize local salient features, which may lead to potential significant information being missed. To address the above issues, we proposed a local information interaction enhancement network (LIEN) for person re-ID. The network integrates attention mechanism with local-based strategy to achieve mutually facilitating and complementary effects. Specifically, LIEN was composed of three parts: a backbone network, a dual attention module (DAM), and a local information interaction module (LIM). DAM was used to guide the backbone network to extract the local salient information, and LIM was used to mine the potential significant information that DAM ignored by exploring the correlations between different channel-level local features. With the combination of DAM and LIM, LIEN mined the salient features while obtaining more potential effective features. In particular, the way that it extracts local features, LIEN effectively avoided pedestrian misalignment caused by uniformly dividing pedestrian images. We evaluated the performance of LIEN on public person datasets, and experimental results indicated that LIEN achieved a more advanced performance.

Chinese agricultural diseases and pests named entity recognition with multi-scale local context features and self-attention mechanism

A multi-scale and multi-level feature aggregation network for crowd counting

During the past five years, there has been an increasing trend of weakly supervised crowd counting methods being developed since such methods just rely on count-level annotations and avoid a laborious labeling process. But, the existing weakly supervised methods usually fail to achieve comparable counting performance to the fully supervised methods. To improve the accuracy of crowd counting tasks, we propose to combine the convolutional neural network (CNN) and Transformer frameworks. Since CNN focuses on capturing local detail information and Transformer can effectively extract global context information, we believe that the combination of CNN and Transformer could learn more efficient feature representations for crowd images. Our proposed framework is named CrowdCCT (Crowd Counting via CNN and Transformer), and it is composed of a CNN feature extraction part, a Transformer feature extraction part, and a counting regression part. In the CNN part, we utilize DenseNet121 to learn rich semantic features with its inherent dense connection structure. In the Transformer part, we introduce two attention modules, Multi-Scale Dilated Attention (MSDA) and Location-Enhanced Attention (LEA), working together to extract more expressive features. The output features are then fed into the regression part to generate the predicted counting results. Experiments on four crowd counting benchmark datasets demonstrate that our proposed CrowdCCT can achieve superior performance. Also, the experimental results validate the feasibility and effectiveness of combining CNN and Transformer for weakly supervised counting tasks. Our work could be expected to promote further combination research on CNN and Transformer.

Abstract. Most of the existing crowd counting research methods are based on convolutional neural network (CNN), which has a strong ability to extract local features, but is limited by the size of the receptive field, making it difficult to model the global context. At the same time, the background of crowd images is complex, the targets are densely distributed, and they are easily disturbed by external conditions such as occlusion and lighting. Therefore, it is extremely cumbersome and error-prone to label the heads of pedestrians in the image. In response to the above problems, this paper proposes a weakly supervised crowd counting method based on Swin Transformer. First, Swin Transformer is used as the backbone network for feature extraction to capture global context information and realize the modeling of feature interactions between targets. Secondly, attention-based multi-scale feature fusion module is designed to aggregate global spatial position features on multiple scales to improve the detection effect of small objects. Finally, global average pooling is used for feature dimensionality reduction and a regression layer is designed to predict the number of people. Tests were carried out on three crowd datasets including Shanghai Tech, UCF_CC_50 and UCF_QNRF. The experimental results show that the overall performance of the proposed method is better than other common crowd counting methods.

The rotating parts of large and complex equipment are key components that ensure the normal operation of the equipment. Accurate fault diagnosis is crucial for the safe operation of these systems. To simultaneously extract both local and global valuable fault feature information from key components of complex equipment, this study proposes a fault diagnosis network model, named MultiDilatedFormer, which is based on the fusion of transformer and multi-head dilated convolution. The newly designed multi-head dilated convolution module is sequentially integrated into the transformer-encoder architecture, constructing a feature extraction module where the complementary advantages of both components enhance overall performance. Firstly, the sample is expanded into a two-dimensional feature map and then input into the newly designed feature extraction module. Finally, the diagnostic output is performed by the designed patch feature fusion module and classifier module. Additionally, interpretability research is conducted on the proposed model, aiming to understand the decision-making mechanism of the model through visual analysis of the entire decision process. The experimental results on three different datasets indicate that the proposed model achieved high accuracy in fault diagnosis with relatively short data windows. The highest accuracy reached 97.95%, which was up to 10.97% higher than other models. Furthermore, the feasibility of the model is also verified in the actual dataset of the rotating parts of the injection molding machine. The excellent performance of the model on different datasets demonstrates its effectiveness in extracting comprehensive fault feature information and also proves its great potential in practical industrial applications.

Android, the most widely used mobile operating system, is also the most vulnerable to malware due to its high popularity. This has significantly focused on Android malware detection in mobile security. While extensive research has been conducted using various methods, new malware’s emergence underscores this field’s dynamic nature and the need for continuous research. The motivation that drives malware developers to create Android malware constantly is the potential to access Android devices, thereby gaining access to sensitive user information. This study, which is a complex and in-depth exploration, aims to detect Android malware using a meta-classifier that combines the single-classifier light gradient boosting machine, support vector machine, and random forest. The process involves converting disassembled malware codes into grey images for global and local feature extraction. The classification accuracy is 97% at best on a malware dataset of 3,963 samples. The main contribution of this paper is to produce an Android APK malware detector model that works by combining multiple machine learning algorithms trained using the dataset resulting from local and global feature extraction algorithms.

Crowd counting is a highly challenging problem in computer vision and machine learning. Most previous methods have focused on consistent density crowds, i.e., either a sparse or a dense crowd, meaning they performed well in global estimation while neglecting local accuracy. To make crowd counting more useful in the real world, we propose a new perspective, named pan-density crowd counting, which aims to count people in varying density crowds. Specifically, we propose the Pan-Density Network (PaDNet) which is composed of the following critical components. First, the Density-Aware Network (DAN) contains multiple subnetworks pretrained on scenarios with different densities. This module is capable of capturing pandensity information. Second, the Feature Enhancement Layer (FEL) effectively captures the global and local contextual features and generates a weight for each density-specific feature. Third, the Feature Fusion Network (FFN) embeds spatial context and fuses these density-specific features. Further, the metrics Patch MAE (PMAE) and Patch RMSE (PRMSE) are proposed to better evaluate the performance on the global and local estimations. Extensive experiments on four crowd counting benchmark datasets, the ShanghaiTech, the UCF-CC-50, the UCSD, and the UCFQNRF, indicate that PaDNet achieves state-of-the-art recognition performance and high robustness in pan-density crowd counting.

CNN-based methods have made great progress in single-image rain removal. Most recent methods improve performance by increasing the depth of the network. To fully extract local and global features while reducing inference time, we propose a top-to-down attribute-insensitive multiscale hourglass network for rain streak and raindrop removal. For the rain removal task, we expect that the constructed network can accurately identify the various attributes of the rain information characteristics of the small target. Considering the difference in the size, shape, direction and density of rain streak and raindrop, inspired by the performance of hourglass architecture to capture multiscale features in human pose estimation, we introduce an attribute-insensitive hourglass module to recognize the attributes of rain streak and raindrop in a unified framework. This feature extraction module could capture the characteristics of rain streak and raindrop with different attributes. This stacked hourglass blocks down-sample features and then up-samples them back to the original resolution based on discrete wavelet transform and inverse discrete wavelet transform. We perform extensive experiments on five synthetic and real-world de-raining datasets to validate the effectiveness of our proposed network on rain streak and raindrop removal. The qualitative and quantitative results show that our method is suitable for removing rain streak and raindrop in a unified framework. We present the results of generalization and ablation study for key components, we also report the accuracy of semantic segmentation after preprocessing with all rain removal methods. Our source code will be available on the GitHub: https://github.com/Ruini94/AIMHNet .

Medical image segmentation is a fundamental task in medical image analysis, playing a crucial role in disease diagnosis, treatment planning, and clinical decision-making. Accurate segmentation of anatomical structures, such as blood vessels, organs, and lesions, is essential for reliable medical interpretations. However, existing segmentation models often face challenges in effectively capturing both local and global features within medical images. Traditional convolutional neural networks (CNNs) excel at extracting fine-grained local features but struggle to model long-range dependencies due to their limited receptive fields. Conversely, transformer-based models can capture global contextual relationships but often fail to preserve detailed local structures, leading to suboptimal segmentation performance. The purpose of this study is to develop SEFormer, a novel hybrid network that integrates SENet, ResNet, and Transformer to enhance medical image segmentation. SEFormer aims to effectively capture both local and global features, addressing the limitations of traditional CNNs and transformer-based models.SENet is incorporated to recalibrate feature maps by adaptively emphasizing informative features, thereby compensating for the global representation limitations of CNNs. Meanwhile, the ResNet backbone ensures deep feature extraction while maintaining computational efficiency. Additionally, the Transformer module is introduced to capture long-range dependencies and enhance contextual awareness, complementing the local feature extraction capabilities of CNNs. This hybrid approach allows SEFormer to balance fine-grained spatial details with broader contextual information, leading to more precise segmentation results.Furthermore, to mitigate information loss during feature extraction and ensure a more complete and hierarchical representation of image information, we introduce a feature pyramid structure inspired by multi-scale image pyramid models. By progressively increasing the receptive field across different scales and employing skip connections, SEFormer effectively fuses multi-scale local and global features at each stage of the pyramid. This hierarchical fusion mechanism ensures a richer and more robust feature representation, which is particularly beneficial for segmenting complex medical structures such as blood vessels. We evaluate SEFormer on the CHASE_DB1 dataset, a widely used benchmark for retinal vessel segmentation. Experimental results demonstrate that SEFormer outperforms existing state-of-the-art segmentation methods, achieving a 3.25% improvement in segmentation accuracy. Additionally, we conduct ablation studies to verify the contributions of different components within SEFormer. The results show that incorporating SENet enhances feature recalibration and channel attention, while the Transformer module significantly improves global context awareness. The feature pyramid structure further contributes to performance gains by ensuring a multi-scale representation of vascular structures. Compared to conventional CNN-based methods, our approach achieves better segmentation accuracy with improved robustness to variations in vessel thickness, noise, and image contrast. Furthermore, SEFormer maintains computational efficiency, making it suitable for real-world clinical applications where both accuracy and processing speed are critical. SEFormer provides an efficient and accurate approach for medical image segmentation by leveraging the complementary strengths of CNNs, SENet, and Transformers while integrating a feature pyramid structure to maximize feature representation. Future research directions include extending SEFormer to other medical imaging modalities, such as CT and MRI scans, as well as exploring further optimization techniques to reduce computational costs while maintaining high segmentation performance. Additionally, integrating SEFormer with active learning frameworks could further improve segmentation performance by leveraging expert-annotated data more efficiently.