Deep Features Homography Transformation Fusion Network—A Universal Foreground Segmentation Algorithm for PTZ Cameras and a Comparative Study

This paper proposed a novel approach for detecting lung sound disorders using deep learning feature fusion. The lung sound dataset are oversampled and converted into spectrogram images. Then, extracting deep features from CNN architectures, which are pre-trained on large-scale image datasets. These deep features capture rich representations of spectrogram images from the input signals, allowing for a comprehensive analysis of lung disorders. Next, a fusion technique is employed to combine the extracted features from multiple CNN architectures totlaly 8064 feature. This fusion process enhances the discriminative power of the features, facilitating more accurate and robust detection of lung disorders. To further improve the detection performance, an improved CNN Architecture is employed. To evaluate the effectiveness of the proposed approach, an experiments conducted on a large dataset of lung disorder signals. The results demonstrate that the deep feature fusion from different CNN architectures, combined with different CNN Layers, achieves superior performance in lung disorder detection. Compared to individual CNN architectures, the proposed approach achieves higher accuracy, sensitivity, and specificity, effectively reducing false negatives and false positives. The proposed model achieves 96.03% accuracy, 96.53% Sensitivity, 99.424% specificity, 96.52% precision, and 96.50% F1 Score when predicting lung diseases from sound files. This approach has the potential to assist healthcare professionals in the early detection and diagnosis of lung disorders, ultimately leading to improved patient outcomes and enhanced healthcare practices.

The classification of high-resolution (HR) synthetic aperture radar (SAR) images is of great importance for SAR scene interpretation and application. However, the presence of intricate spatial structural patterns and complex statistical nature makes SAR image classification a challenging task, especially in the case of limited labeled SAR data. This paper proposes a novel HR SAR image classification method, using a multi-scale deep feature fusion network and covariance pooling manifold network (MFFN-CPMN). MFFN-CPMN combines the advantages of local spatial features and global statistical properties and considers the multi-feature information fusion of SAR images in representation learning. First, we propose a Gabor-filtering-based multi-scale feature fusion network (MFFN) to capture the spatial pattern and get the discriminative features of SAR images. The MFFN belongs to a deep convolutional neural network (CNN). To make full use of a large amount of unlabeled data, the weights of each layer of MFFN are optimized by unsupervised denoising dual-sparse encoder. Moreover, the feature fusion strategy in MFFN can effectively exploit the complementary information between different levels and different scales. Second, we utilize a covariance pooling manifold network to extract further the global second-order statistics of SAR images over the fusional feature maps. Finally, the obtained covariance descriptor is more distinct for various land covers. Experimental results on four HR SAR images demonstrate the effectiveness of the proposed method and achieve promising results over other related algorithms.

Automatic classification of Lyme disease rashes on the skin helps clinicians and dermatologists’ probe and investigate Lyme skin rashes effectively. This paper proposes a new in‐depth features fusion system to classify Lyme disease rashes. The proposed method consists of two main steps. First, three different deep learning models, Densenet201, InceptionV3, and Exception, were trained independently to extract the deep features from the erythema migrans (EM) images. Second, a deep feature fusion mechanism (meta classifier) is developed to integrate the deep features before the final classification output layer. The meta classifier is a basic deep convolutional neural network trained on original images and features extracted from base level three deep learning models. In the feature fusion mechanism, the last three layers of base models are dropped out and connected to the meta classifier. The proposed deep feature fusion method significantly improved the classification process, where the classification accuracy was 98.97%, which is particularly impressive than the other state‐of‐the‐art models.

The precise and prompt identification of skin cancer is essential for efficient treatment. Variations in colour within skin lesions are critical signs of malignancy; however, discrepancies in imaging conditions may inhibit the efficacy of deep learning models. Numerous previous investigations have neglected this problem, frequently depending on deep features from a singular layer of an individual deep learning model. This study presents a new hybrid deep learning model that integrates discrete cosine transform (DCT) with multi-convolutional neural network (CNN) structures to improve the classification of skin cancer. Initially, DCT is applied to dermoscopic images to enhance and correct colour distortions in these images. After that, several CNNs are trained separately with the dermoscopic images and the DCT images. Next, deep features are obtained from two deep layers of each CNN. The proposed hybrid model consists of triple deep feature fusion. The initial phase involves employing the discrete wavelet transform (DWT) to merge multidimensional attributes obtained from the first layer of each CNN, which lowers their dimension and provides time–frequency representation. In addition, for each CNN, the deep features of the second deep layer are concatenated. Afterward, in the subsequent deep feature fusion stage, for each CNN, the merged first-layer features are combined with the second-layer features to create an effective feature vector. Finally, in the third deep feature fusion stage, these bi-layer features of the various CNNs are integrated. Through the process of training multiple CNNs on both the original dermoscopic photos and the DCT-enhanced images, retrieving attributes from two separate layers, and incorporating attributes from the multiple CNNs, a comprehensive representation of attributes is generated. Experimental results showed 96.40% accuracy after trio-deep feature fusion. This shows that merging DCT-enhanced images and dermoscopic photos can improve diagnostic accuracy. The hybrid trio-deep feature fusion model outperforms individual CNN models and most recent studies, thus proving its superiority.

Human Action Recognition (HAR) has become one of the most active research area in the domain of artificial intelligence, due to various applications such as video surveillance. The wide range of variations among human actions in daily life makes the recognition process more difficult. In this article, a new fully automated scheme is proposed for Human action recognition by fusion of deep neural network (DNN) and multiview features. The DNN features are initially extracted by employing a pre-trained CNN model name VGG19. Subsequently, multiview features are computed from horizontal and vertical gradients, along with vertical directional features. Afterwards, all features are combined in order to select the best features. The best features are selected by employing three parameters i.e. relative entropy, mutual information, and strong correlation coefficient (SCC). Furthermore, these parameters are used for selection of best subset of features through a higher probability based threshold function. The final selected features are provided to Naive Bayes classifier for final recognition. The proposed scheme is tested on five datasets name HMDB51, UCF Sports, YouTube, IXMAS, and KTH and the achieved accuracy were 93.7%, 98%, 99.4%, 95.2%, and 97%, respectively. Lastly, the proposed method in this article is compared with existing techniques. The resuls shows that the proposed scheme outperforms the state of the art methods.

Many applications for mobile robot authentication require to be able to explore a large field of view with high resolution. The proposed vision system is composed of a catadioptric sensor for full range monitoring and a pan tilt zoom (PTZ) camera leading to an innovative sensor, able to detect and track any moving objects at a higher zoom level. In our application, the catadioptric sensor is calibrated and used to detect and track regions of interest (ROIs) within its 360 degree field of view (FOV), especially face regions. Using a joint calibration strategy, the PTZ camera parameters are automatically adjusted by the system in order to detect and track the face ROI within a higher resolution.

Object detection (OD) in unmanned aerial vehicle (UAV) images faces many challenges, with diverse-scale objects and small objects being particularly prominent issues. To alleviate these challenges, we propose a novel multiscale feature learning and feature fusion network under the guidance of deformable convolution. First, a deformable convolution-guided feature learning (DCGFL) block is designed in the backbone to extract more effective multiscale features. The DCGFL block leverages the adaptability of deformable convolution to the shapes and scales of objects, akin to spatial attention. Moreover, it also employs channel attention to identify important feature maps. Hence, the proposed backbone possesses the functionality of spatial attention and channel attention. Second, in the neck, we devise a simple generalized feature pyramid network (SimpleGFPN) with several deformable convolution-guided feature fusion (DCGFF) blocks to fuse multiscale features. The proposed neck has cross-layer and cross-scale pathways, facilitating effective information exchange and fusion between shallow spatial and deep semantic features. Third, the SIoU loss is used to better model the bounding box regression loss. Finally, experimental results on the VisDrone2021 and UAVDT datasets show that the proposed method outperforms the compared OD methods. In terms of mean average precision, we obtain 37.8% on VisDrone2021 and 18.5% on UAVDT.

Due to increasing security concerns, a complete security system should consist of two major components, a computer-based face-recognition system and a real-time automated video surveillance system. A computerbased face-recognition system can be used in gate access control for identity authentication. In recent studies, multispectral imaging and fusion of multispectral narrow-band images in the visible spectrum have been employed and proven to enhance the recognition performance over conventional broad-band images, especially when the illumination changes. Thus, we present an automated method that specifies the optimal spectral ranges under the given illumination. Experimental results verify the consistent performance of our algorithm via the observation that an identical set of spectral band images is selected under all tested conditions. Our discovery can be practically used for a new customized sensor design associated with given illuminations for an improved face recognition performance over conventional broad-band images. In addition, once a person is authorized to enter a restricted area, we still need to continuously monitor his/her activities for the sake of security. Because pantilt-zoom (PTZ) cameras are capable of covering a panoramic area and maintaining high resolution imagery for real-time behavior understanding, researches in automated surveillance systems with multiple PTZ cameras have become increasingly important. Most existing algorithms require the prior knowledge of intrinsic parameters of the PTZ camera to infer the relative positioning and orientation among multiple PTZ cameras. To overcome this limitation, we propose a novel mapping algorithm that derives the relative positioning and orientation between two PTZ cameras based on a unified polynomial model. This reduces the dependence on the knowledge of intrinsic parameters of PTZ camera and relative positions. Experimental results demonstrate that our proposed algorithm presents substantially reduced computational complexity and improved flexibility at the cost of slightly decreased pixel accuracy as compared to Chen and Wang’s method [18].

Due to the complex and changeable marine environment, the active sonar target recognition problem has always been difficult in the field of underwater acoustics. Deep learning-based fusion recognition technology provides an effective way to solve this problem, but relying on simple concatenation strategies to fuse multi-domain features can cause information redundancy, and it is not easy to effectively mine correlation information between domains. Therefore, this paper proposes an attention mechanism-based multi-domain feature fusion approach for active sonar target recognition. By preprocessing active sonar echo signals and constructing a multi-domain feature extraction and fusion network, this method uses a one-dimensional convolutional neural network with long short-term memory (1DCNN-LSTM) and a two-dimensional convolutional neural network (2DCNN) with channel attention introduced to extract deep features from different domains. Subsequently, combining feature concatenation and constructing multi-domain cross-attention, intra- and cross-domain feature fusion is performed, which can effectively eliminate redundant information and promote inter-domain information interaction, while maximizing the retention of target features. Experimental results show that compared with single-domain methods, the network using an attention mechanism for multi-domain feature fusion strengthens cross-domain information interaction and significantly improves feature representation capability. Compared with other methods, the proposed method has obvious advantages in performance and maintains stable generalization ability in scenarios with low signal-clutter ratios.

The goal of infrared and visible image fusion is to generate a fused image that possesses the prominent attributes of infrared image and the rich textures of visible image. Current deep learning-based methods typically involve three main steps: feature extraction, feature fusion, and reconstruction. Extracting multi-scale features during the feature extraction stage is beneficial for fully utilizing deep features. Shallow image features exhibit higher resolution and contain more textures, while deep image features have lower resolution but stronger semantic information after more pooling and convolution. Existing methods use the same network to process features of different scales, while ignoring their differences. Moreover, they lack carefully designed texture preservation modules, leading to insufficient preservation of texture in the fused images. To overcome these issues, we propose a novel end-to-end fusion network for infrared and visible images where we develop a scaling-down network and texture attention. Based on the characteristics of features at different scales, we design a scaling-down fusion network that use deeper and more complex network to process shallow features, while using more streamlined network to process deep features. In order to better preserve image textures, we design texture attention in a relatively gentle way that focus on feature channels have rich textures to achieve the goal. We conducted experiments on publicly available datasets, and the results demonstrate that our method surpasses eleven state-of-the-art methods in terms of fusion performance. This conclusion has been verified through both subjective evaluation and objective evaluation.

Shot boundary detection is essentially to detect the position of frames where the shot changes. It has been actively studied in video analysis and management for convenience, which becomes a key technique with the rapid proliferation of rich and diverse videos. With respect to the complex characteristics of different shots in varying length and content variation property, in this paper we present a two stage method for shot boundary detection (TSSBD) which distinguishes abrupt shot by fusing color histogram and deep features, and locate gradual shot changes with C3D-based deep analysis. Abrupt shot changes are detected firstly as it occurs between two frames, which divides the complete video into segments containing gradual transitions; Over these video segments, gradual shot change detection is implemented using 3D-convolutional neural network, which classifies clips into specific gradual shot change types; Finally, an effective merging strategy is proposed to locate positions of gradual shot transitions. The experimental analysis illustrates that the proposed progressive method is capable of detecting both abrupt shot transitions and gradual shot transitions accurately.

BACKGROUND Timely identification of local failure after Stereotactic Radiosurgery (SRS) offers the opportunity for appropriate treatment modifications that may result in improved treatment outcomes, patient survival, and quality of life. Previous studies showed that the addition of either radiomics or deep learning features to clinical features increased the accuracy of the models in predicting local control (LC) of brain metastases after SRS. To date, however, no study combined both radiomics and deep learning features together with clinical features to develop machine learning algorithms to predict LC of brain metastases. In this study, we examined whether a model trained with a combination of all these features could predict LC better than models trained with only a subset of these features. MATERIAL AND METHODS Pre-treatment brain MRIs and clinical data were collected retrospectively for 129 patients at the Gamma Knife Center of Elisabeth-TweeSteden Hospital (ETZ), Tilburg, The Netherlands. The patients were split into 103 patients for training and 26 patients for testing. The segment-based radiomics features were extracted using the radiomics feature extractor of the python radiomics package. The deep learning features were extracted using a fine-tuned 3D ResNet model and then combined with the clinical and radiomics features. A Random Forest classifier was trained with the training data set and then tested with the test data set. The performance was compared across 4 different models trained with clinical features only, clinical and radiomics features, clinical and deep learning features, and clinical, radiomics and deep learning features. RESULTS The prediction model with only clinical variables demonstrated an area under the receiver operating characteristic curve (AUC) of 0.82 and an accuracy of 75.6%. The prediction model with the combination of clinical and radiomics features demonstrated an AUC of 0.880 and an accuracy of 83.3% whereas the prediction model with the combination of clinical and deep learning features demonstrated an AUC of 0.863 and an accuracy of 78.3%. The best prediction performance was associated with the model that combined the clinical, radiomics and deep learning features with an AUC of 0.886 and 87% accuracy. CONCLUSION Machine learning models built on radiomics features and deep learning features combined with patient characteristics show potential to predict LC after SRS with high accuracy. The promising findings from this study demonstrate the potential for early prediction of SRS outcome for brain metastasis prior to treatment initiation and might offer the opportunity for appropriate treatment modifications that may result in improved treatment outcomes, patient survival, and quality of life.

A novel method for auto-calibration of a PTZ (pan-tilt-zoom) camera network is proposed. The key idea on which it is based is to use pan-tilt motions generated by PTZ cameras themselves as calibration patterns. Generating and observing the pan-tilt motions of each camera makes it possible to estimate whole relative camera poses in a camera network. Cameras first observe circular trajectories of a marker set on another camera performing tilt motions with various pan angles. Although finding correspondences between marker points captured by the cameras is cumbersome due to network delays, that of between circular trajectories is easy. It is thus possible to use several hundred points in a circular trajectory to estimate its normal vectors and centers with high accuracy. Imposing geometric constraints on them makes it possible to eliminate ambiguities of them and obtain a unique relative camera pose directly. A novel refinement process to minimize differences between modeled pan-tilt motions and three or more circular trajectories is then performed. Experiments using synthetic data confirmed that the refinement process improved camera pose estimation accuracy in comparison to the direct estimation. An experiment using real data showed that the proposed method works properly for real cameras.

Fusing learned representations from Riesz Filters and Deep CNN for lung tissue classification.

Immediately after an earthquake, rapid disaster management is the main challenge for relevant organizations. While satellite images have been used in the past two decades for building-damage mapping, they have rarely been utilized for the timely damage monitoring required for rescue operations. Unmanned aerial vehicles (UAVs) have recently become very popular due to their agile deployment to sites, super-high spatial resolution, and relatively low operating cost. This paper proposes a novel deep-learning-based method for rapid post-earthquake building damage detection. The method detects damages in four levels and consists of three steps. First, three different feature types—non-deep, deep, and their fusion—are investigated to determine the optimal feature extraction method. A “one-epoch convolutional autoencoder (OECAE)” is used to extract deep features from non-deep features. Then, a rule-based procedure is designed for the automatic selection of the proper training samples required by the classification algorithms in the next step. Finally, seven famous machine learning (ML) algorithms—including support vector machine (SVM), random forest (RF), gradient boosting (GB), extreme gradient boosting (XGB), decision trees (DT), k-nearest neighbors (KNN), and adaBoost (AB)—and a basic deep learning algorithm (i.e., multi-layer perceptron (MLP)) are implemented to obtain building damage maps. The results indicated that auto-training samples are feasible and superior to manual ones, with improved overall accuracy (OA) and kappa coefficient (KC) over 22% and 33%, respectively; SVM (OA = 82% and KC = 74.01%) was the most accurate AI model with a slight advantage over MLP (OA = 82% and KC = 73.98%). Additionally, it was found that the fusion of deep and non-deep features using OECAE could significantly enhance damage-mapping efficiency compared to those using either non-deep features (by an average improvement of 6.75% and 9.78% in OA and KC, respectively) or deep features (improving OA by 7.19% and KC by 10.18% on average) alone.