Fusing Transformer and YOLOX for Smoke Detection

Pre-trained DCNN trained on a large-scale image database can be used as a universal feature representation for image classification, which has achieved significant progress in some image recognition tasks. Compared with other image recognition tasks, directly utilizing a single convolutional feature as feature representation for vein recognition task cannot achieve the impressive result due to the sparse distribution of vein information. Therefore, to obtain more representative and discriminative convolutional feature for vein recognition, a novel multi-layer convolutional features' concatenation with semantic feature selector is proposed in this paper. In the pre-trained DCNN, different convolutional layers can encode different-level feature information. High-level convolutional features with vein information cover more semantic information and low-level convolutional features with vein information cover more detail information. However, low-level convolutional features also contain some background information. Therefore, in order to remove the background information of low-level convolutional features, a novel semantic feature selector is presented. First, the proposed local max-pooling of preserving spatial position (LMP-PSP) information is applied on activation map obtained by adding up all feature maps of the high-level convolutional layer to generate the semantic weighting map, which reflects key vein information of high-level convolutional features. Then, semantic weighting map is regarded as a feature selector to discard the background information of the low-level convolutional features and preserve the detail information of low level convolutional features. Finally, low-level convolutional features with vein information are selectively linked to high-level convolutional features with vein information based on the proposed semantic feature selector. A series of rigorous experiments on two lab-made vein databases named CUMT-Hand-Dorsa Vein database and CUMT-Palm Vein database is conducted to verify the effectiveness and feasibility of the proposed model. Besides, additional experiments with PUT Palm Vein database and the subset of PolyU database illustrate its generalization ability and robustness.

Representations of visual and semantic information can overlap in human visual cortex, with the same neural populations exhibiting sensitivity to low-level features (orientation, spatial frequency, retinotopic position) and high-level semantic categories (faces, scenes). It has been hypothesized that this relationship between low-level visual and high-level category neural selectivity reflects natural scene statistics, such that neurons in a given category-selective region are tuned for low-level features or spatial positions that are diagnostic of the region's preferred category. To address the generality of this “natural scene statistics” hypothesis, as well as how well it can account for responses to complex naturalistic images across visual cortex, we performed two complementary analyses. First, across a large set of rich natural scene images, we demonstrated reliable associations between low-level (Gabor) features and high-level semantic categories (faces, buildings, animate/inanimate objects, small/large objects, indoor/outdoor scenes), with these relationships varying spatially across the visual field. Second, we used a large-scale functional MRI dataset (the Natural Scenes Dataset) and a voxelwise forward encoding model to estimate the feature and spatial selectivity of neural populations throughout visual cortex. We found that voxels in category-selective visual regions exhibit systematic biases in their feature and spatial selectivity, which are consistent with their hypothesized roles in category processing. We further showed that these low-level tuning biases are not driven by selectivity for categories themselves. Together, our results are consistent with a framework in which low-level feature selectivity contributes to the computation of high-level semantic category information in the brain.

With the development of theories and technologies in medical imaging, most of the tumors can be detected in the early stage. However, the nature of ovarian cysts lacks accurate judgement, leading to that many patients with benign nodules still need Fine Needle Aspiration (FNA) biopsies or surgeries, increasing the physical pain and mental pressure of patients as well as unnecessary medical health care costs. Therefore, we present an image diagnosis system for classifying the ovarian cysts in color ultrasound images, which novelly applies the image features fused by both high-level features from deep learning network and low-level features from texture descriptor. Firstly, the ultrasound images are enhanced to improve the quality of training data set and the rotation invariant uniform local binary pattern (ULBP) features are extracted from each of the images as the low-level texture features. Then the high-level deep features extracted by the fine-tuned GoogLeNet neural network and the low-level ULBP features are normalized and cascaded as one fusion feature that can represent both the semantic context and the texture patterns distributed in the image. Finally, the fusion features are input to the Cost-sensitive Random Forest classifier to classify the images into "malignant" and "benign". The high-level features extracted by the deep neural network from the medical ultrasound image can reflect the visual features of the lesion region, while the low-level texture features can describe the edges, direction and distribution of intensities. Experimental results indicate that the combination of the two types of features can describe the differences between the lesion regions and other regions, and the differences between lesions regions of malignant and benign ovarian cysts.

Automatic liver tumor segmentation from 3D Computed Tomography (CT) is a necessary prerequisite in the interventions of hepatic abnormalities and surgery planning. However, accurate liver tumor segmentation remains challenging due to the large variability of tumor sizes and inhomogeneous texture. Recent advances based on Fully Convolutional Network (FCN) in liver tumor segmentation draw on success of learning discriminative multi-level features. In this paper, we propose a Deep Attentive Refinement Network (DARN) for improved liver tumor segmentation from CT volumes by fully exploiting both low and high level features embedded in different layers of FCN. Different from existing works, we exploit attention mechanism to leverage the relation of different levels of features encoded in different layers of FCN. Specifically, we introduce a Semantic Attention Refinement (SemRef) module to selectively emphasize global semantic information in low level features with the guidance of high level ones, and a Spatial Attention Refinement (SpaRef) module to adaptively enhance spatial details in high level features with the guidance of low level ones. We evaluate our network on the public MICCAI 2017 Liver Tumor Segmentation Challenge dataset (LiTS dataset) and it achieves state-of-the-art performance. The proposed refinement modules are an effective strategy to exploit multi-level features and has great potential to generalize to other medical image segmentation tasks.

Image retrieval is one of the most interesting and fastest growing research areas in all fields. It is an effective and efficient tool for managing large image databases. In most Content-Based Image Retrieval (CBIR) systems, an image is represented by a set of low-level visual features; hence a direct correlation with high-level semantic information will be absent. Therefore, a gap exists between high-level information and low-level features, which is the main reason that hinders the improvement of the image retrieval accuracy. In this work, main focus is on the semantic based image retrieval system using Gray Level Co-occurrence Matrix (GLCM) for texture feature extraction. Based on the texture features, semantic interpretations are given to the extracted textures. The images are retrieved according to user satisfaction and thereby reduce the semantic gap between low level features and high level features.

Automatic change detection is an important and difficult task in the field of remote sensing. In this study, a deep Siamese convolutional network based on the fusion of high- and low-level features is proposed for change detection in remote sensing images. Given that low-level features correspond to low-order ones (e.g., texture) that are sensitive to change and that high-level features can accurately reflect image category information (e.g., semantic information), we fuse these features to enhance the abstractness and robustness of the extracted features in the change detection framework. The whole system is end-to-end and does not require any pre- or post-processing. Experimental results on three datasets show that our method is superior to other advanced methods by adding a high- and low-level fusion framework.

A pooling-based feature pyramid network for salient object detection

Some tasks in the construction industry and urban management field such as site selection and fire response management are usually managed by using a Geographical Information System (GIS), as the tasks in these processes require a high level and amount of integrated geospatial information. Recently, a key element of this integrated geospatial information to emerge is detailed geometrical and semantic information about buildings. In parallel, Building Information Models (BIMs) of today have the capacity for storing and representing such detailed geometrical and semantic information. In this context, the research aimed to investigate the applicability of BIMs in geospatial environment by focusing specifically on these two domains; site selection and fire response management. In the first phase of the research two use case scenarios were developed in order to understand the processes in these domains in a more detailed manner and to establish the scope of a possible software development for transferring information from BIMs into the geospatial environment. In the following phase of the research two data models were developed – a Schema-Level Model View and a geospatial data model. The Schema-Level Model View was used in simplifying the information acquired from the BIM, while the geospatial data model acted as the template for creating physical files and databases in the geospatial environment. Following this, three software components to transfer building information into the geospatial environment were designed, developed, and validated. The first component served for acquiring the building information from the BIM, while the latter two served for transforming the information into the geospatial environment. The overall research demonstrated that it is possible to transfer (high level of geometric and semantic) information acquired from BIMs into the geospatial environment. The results also demonstrated that BIMs provide a sufficient level and amount of (geometric and semantic) information (about the building) for the seamless automation of data management tasks in the site selection and fire response management processes.

Obtaining animal regions and the relative position relationship of animals in the scene is conducive to further studying animal habits, which is of great significance for smart animal farming. However, the complex breeding environment still makes detection difficult. To address the problems of poor target segmentation effects and the weak generalization ability of existing semantic segmentation models in complex scenes, a semantic segmentation model based on an improved DeepLabV3+ network (Imp-DeepLabV3+) was proposed. Firstly, the backbone network of the DeepLabV3+ model was replaced by MobileNetV2 to enhance the feature extraction capability of the model. Then, the layer-by-layer feature fusion method was adopted in the Decoder stage to integrate high-level semantic feature information with low-level high-resolution feature information at multi-scale to achieve more precise up-sampling operation. Finally, the SENet module was further introduced into the network to enhance information interaction after feature fusion and improve the segmentation precision of the model under complex datasets. The experimental results demonstrate that the Imp-DeepLabV3+ model achieved a high pixel accuracy (PA) of 99.4%, a mean pixel accuracy (MPA) of 98.1%, and a mean intersection over union (MIoU) of 96.8%. Compared to the original DeepLabV3+ model, the segmentation performance of the improved model significantly improved. Moreover, the overall segmentation performance of the Imp-DeepLabV3+ model surpassed that of other commonly used semantic segmentation models, such as Fully Convolutional Networks (FCNs), Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP), and U-Net. Therefore, this study can be applied to the field of scene segmentation and is conducive to further analyzing individual information and promoting the development of intelligent animal farming.

Real-time smoke detection is of great significance for early warning of fire, which can avoid the serious loss caused by fire. Detecting smoke in actual scenes is still a challenging task due to large variance of smoke color, texture, and shapes. Moreover, the smoke detection in the actual scene is faced with the difficulties in data collection and insufficient smoke datasets, and the smoke morphology is susceptible to environmental influences. To improve the performance of smoke detection and solve the problem of too few datasets in real scenes, this paper proposes a model that combines a deep convolutional generative adversarial network and a convolutional neural network (DCG-CNN) to extract smoke features and detection. The vibe algorithm was used to collect smoke and nonsmoke images in the dynamic scene and deep convolutional generative adversarial network (DCGAN) used these images to generate images that are as realistic as possible. Besides, we designed an improved convolutional neural network (CNN) model for extracting smoke features and smoke detection. The experimental results show that the method has a good detection performance on the smoke generated in the actual scenes and effectively reduces the false alarm rate.

Infrared scene simulation generation technology is an important component of infrared imaging guidance hardware-in-the-loop simulation. However, there are significant differences in the infrared simulation images generated by different simulation platforms, and how to effectively analyze and evaluate the realism of infrared visual simulation generated images is a research difficulty. This paper proposes an infrared simulation image fidelity evaluation algorithm based on object detection tasks, IFE-YOLO. The model integrates Swin Transformer, YOLOX, and attention mechanism to achieve fidelity evaluation at the target task level. Firstly, we propose that STCNet backbone network extracts target feature information, which not only has Swin Transformer's excellent ability to model based on global information, but also uses attention mechanism to capture location information and channel relationship to enhance the ability to process feature information. Secondly, based on the improved PANet, a feature pyramid network is constructed for deep fusion of high-level and low-level features to effectively use semantic information and location information. Thirdly, decoupling detection head and improved position loss function are used in the target detection part to improve the model performance. Finally, the infrared fidelity evaluation process for detection task is designed, and the validity of the model is verified on the constructed infrared simulation image dataset.

Selective feature connection mechanism: Concatenating multi-layer CNN features with a feature selector

Most state-of-the-art deep networks proposed for biomedical image segmentation are developed based on U-Net. While remarkable success has been achieved, its inherent limitations hinder it from yielding more precise segmentation. First, its receptive field is limited due to the fixed kernel size, which prevents the network from modeling global context information. Second, when spatial information captured by shallower layer is directly transmitted to higher layers by skip connections, the process inevitably introduces noise and irrelevant information to feature maps and blurs their semantic meanings. In this article, we propose a novel segmentation network equipped with a new context prior guidance (CPG) module to overcome these limitations for biomedical image segmentation, namely context prior guidance network (CPG-Net). Specifically, we first extract a set of context priors under the supervision of a coarse segmentation and then employ these context priors to model the global context information and bridge the spatial-semantic gap between high-level features and low-level features. The CPG module contains two major components: context prior representation (CPR) and semantic complement flow (SCF). CPR is used to extract pixels belonging to the same objects and hence produce more discriminative features to distinguish different objects. We further introduce deep semantic information for each CPR by the SCF mechanism to compensate the semantic information diluted during the decoding. We extensively evaluate the proposed CPG-Net on three famous biomedical image segmentation tasks with diverse imaging modalities and semantic environments. Experimental results demonstrate the effectiveness of our network, consistently outperforming state-of-the-art segmentation networks in all the three tasks. Codes are available at https://github.com/zzw-szu/CPGNet .

This paper presents an autonomous navigation system. Our system is based on an accurate 3D map, which includes “geometric information” (e.g., curb, wall, street tree) and “semantic information” (e.g., sidewalk, roadway, crosswalk) extracted by environmental recognition. By using the semantic map, we can obtain the suitable area to keep away from undesired places. Furthermore, by comparing the map with real-time 3D geometric information from LIDAR, we obtain the robot position. To show the effectiveness of our system, we conduct a 3D semantic map construction experiment and driving test. The experiment results show that the proposed system enables accurate and highly reproducible localization and stable autonomous mobility.

Nowadays, 3D city models are used by the increasing number of applications. Most applications require not only geometric information but also semantic information. As a standard and tool for 3D city model, CityGML, provides a method for storing and managing both geometric and semantic information. Moreover, it also provides the multi-scale representation of 3D building model for efficient visualization. In CityGML, building models are represented in five LODs (Level of Detail), start from LOD0, LOD1, LOD2, LOD3, and LOD4. Each level has different accuracy and detail requirement for visualization. Usually, for obtaining multi-LOD of 3D building model, several data sources are integrated. For example, LiDAR data is used for generating LOD0, LOD1, and LOD2 as close-range photogrammetry data is used for generating more detailed model in LOD3 and LOD4. However, using additional data sources is increasing cost and time consuming. Since the development of TLS (Terrestrial Laser Scanner), data collection for detailed model can be conducted in a relative short time compared to photogrammetry. Point cloud generated from TLS can be used for generating the multi-LOD of building model. This paper gives an overview about the representation of 3D building model in CityGML and also method for generating multi-LOD of building from TLS data. An experiment was conducted using TLS. Following the standard in CityGML, point clouds from TLS were processed resulting 3D model of building in different level of details. Afterward, models in different LOD were converted into XML schema to be used in CityGML. From the experiment, final result shows that TLS can be used for generating 3D models of building in LOD1, LOD2, and LOD3.