Visual saliency detection for RGB-D images under a Bayesian framework

Saliency detection in images attracts much research attention for its usage in numerous multimedia applications. In this paper, we propose a saliency detection method based on optimization for RGBD images. With RGBD images, our method utilizes the depth channel to enhance the identification of background and foreground regions. We firstly generate new depth image by using non-linear transformation and outstand object region. Then, we introduce saliency optimization framework to integrate the depth cue and other low-level cues to obtain the final saliency map. The experimental results demonstrate that our method performs better in saliency detection for RGBD Images.

Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images.

The classification of hyperspectral images (HSIs) is one of the most popular topics in the remote sensing community. Numerous feature extraction methods have been proposed to improve the classification accuracy of HSIs. Recently, deep features extracted by convolution neural network (CNN) have been introduced into the classification process of HSIs. Due to the nonlinear and invariant advantages of the features, CNN methods provide a powerful tool for representing geographic objects and classifying HSIs. However, traditional deep features only extracted at pixel-level and often neglect multiscale characteristics of geographic objects. In this study, a new deep feature extraction method is proposed, which takes advantage of multi-scale object analysis and the CNN model. Firstly, multiscale image objects are obtained by the multiscale segmentation algorithm and multiscale low-level features of objects are extracted. Secondly, the CNN is devoted to obtain deep features from low-level object features at each scale, respectively. Thirdly, the obtained deep features at all scales are stacked and fed to one fully connected layer to extract the multiscale deep learning features for classification. Finally, the logistic regression classifier is applied to hyperspectral image (HSI) classification based on object-oriented multiscale deep features. The proposed method was carried out on three widely used hyperspectral data sets: University of Pavia, Salinas, and Washington DC. The results reveal that the proposed method provides better results than other state-of-the-art methods.

In recent years, with the rapid development of computer vision, increasing attention has been paid to remote sensing image scene classification. To improve the classification performance, many studies have increased the depth of convolutional neural networks (CNNs) and expanded the width of the network to extract more deep features, thereby increasing the complexity of the model. To solve this problem, in this paper, we propose a lightweight convolutional neural network based on attention-oriented multi-branch feature fusion (AMB-CNN) for remote sensing image scene classification. Firstly, we propose two convolution combination modules for feature extraction, through which the deep features of images can be fully extracted with multi convolution cooperation. Then, the weights of the feature are calculated, and the extracted deep features are sent to the attention mechanism for further feature extraction. Next, all of the extracted features are fused by multiple branches. Finally, depth separable convolution and asymmetric convolution are implemented to greatly reduce the number of parameters. The experimental results show that, compared with some state-of-the-art methods, the proposed method still has a great advantage in classification accuracy with very few parameters.

In this paper we study the problem of object detection for RGB-D images using semantically rich image and depth features. We propose a new geocentric embedding for depth images that encodes height above ground and angle with gravity for each pixel in addition to the horizontal disparity. We demonstrate that this geocentric embedding works better than using raw depth images for learning feature representations with convolutional neural networks. Our final object detection system achieves an average precision of 37.3%, which is a 56% relative improvement over existing methods. We then focus on the task of instance segmentation where we label pixels belonging to object instances found by our detector. For this task, we propose a decision forest approach that classifies pixels in the detection window as foreground or background using a family of unary and binary tests that query shape and geocentric pose features. Finally, we use the output from our object detectors in an existing superpixel classification framework for semantic scene segmentation and achieve a 24% relative improvement over current state-of-the-art for the object categories that we study. We believe advances such as those represented in this paper will facilitate the use of perception in fields like robotics.KeywordsRGB-D perceptionobject detectionobject segmentation

In view of the scene’s complexity and diversity in scene classification, this paper makes full use of the contextual semantic relationships between the objects to describe the visual attention regions of the scenes and combines with the deep convolution neural networks, so that a scene classification model using visual attention and deep networks is constructed. Firstly, the visual attention regions in the scene image are marked by using the context-based saliency detection algorithm. Then, the original image and the visual attention region detection image are superimposed to obtain a visual attention region enhancement image. Furthermore, the deep convolution features of the original image, the visual attention region detection image, and the visual attention region enhancement image are extracted by using the deep convolution neural networks pretrained on the large-scale scene image dataset Places. Finally, the deep visual attention features are constructed by using the multilayer deep convolution features of the deep convolution networks, and a classification model is constructed. In order to verify the effectiveness of the proposed model, the experiments are carried out on four standard scene datasets LabelMe, UIUC-Sports, Scene-15, and MIT67. The results show that the proposed model improves the performance of the classification well and has good adaptability.

Leveraging the power of computational methods, AI can perform effective strategies in intelligent design. Researchers are pushing the boundaries of AI, developing computational systems to solve complex questions. The authors investigate the association of user preference for UI and deep image features, aiming to predict user preference level using deep convolutional neural networks (DCNNs) trained on a UI design image dataset. A total of 12,186 UI design images were collected from UI.cn and DOOOOR.com . Users' views and likes can help understand the implicit user preference level, which is set as the ground‐truth annotation for the dataset. Six DCNNs, including VGG‐19, InceptionNet‐V3, MobileNet, EfficientNet, ResNet‐50 and NASNetLarge were trained to learn the user preference of UI images. The experiment achieves an optimal result with a mean‐squared error of 0.000214 and a mean absolute error of 0.0103 based on EfficientNet, which indicates that the proposed method provides the possibility in learning the pattern of user aesthetics preference for UI design. On the basis of the prediction model, a mobile application named ‘HotUI’ was developed for UI design recommendations.

Due to large amount of image data, the accuracy and real-time performance of classification are difficult problems in image classification. On the one hand, the broad learning system (BLS) has achieved good results in the timeliness of classification. On the other hand, deep learning feature extraction effect is good, but its structure is complex and the training time is long. Therefore, based on the above advantages of both methods, this paper proposed BLS based on deep features for image classification. Firstly, a feature extractor is constructed based on the ResNet101 to obtain the deep features of the classification image. Then, the feature nodes and enhancement nodes of BLS are constructed based on the deep features. Through the experiment, our method has two performance on benchmark datasets: high classification accuracy, good real-time.

Image saliency detection is very useful in many computer vision tasks while it still remains a challenging problem. In this paper, we propose a new computational saliency detection model which is implemented with a coarse to fine strategy under the Bayesian framework. First, saliency points are applied to get a coarse location of the saliency region. And then, based on the rough region, we compute a prior map for the Bayesian model to achieve the final saliency map. Experimental results on a public available dataset show the effectiveness of the proposed prior map and the strength of our saliency map compared with several previous method.

Recently, monkeypox virus is slowly evolving and there are fears it will spread as COVID-19. Computer-aided diagnosis (CAD) based on deep learning approaches especially convolutional neural network (CNN) can assist in the rapid determination of reported incidents. The current CADs were mostly based on an individual CNN. Few CADs employed multiple CNNs but did not investigate which combination of CNNs has a greater impact on the performance. Furthermore, they relied on only spatial information of deep features to train their models. This study aims to construct a CAD tool named "Monkey-CAD" that can address the previous limitations and automatically diagnose monkeypox rapidly and accurately. Monkey-CAD extracts features from eight CNNs and then examines the best possible combination of deep features that influence classification. It employs discrete wavelet transform (DWT) to merge features which diminishes fused features' size and provides a time-frequency demonstration. These deep features' sizes are then further reduced via an entropy-based feature selection approach. These reduced fused features are finally used to deliver a better representation of the input features and feed three ensemble classifiers. Two freely accessible datasets called Monkeypox skin image (MSID) and Monkeypox skin lesion (MSLD) are employed in this study. Monkey-CAD could discriminate among cases with and without Monkeypox achieving an accuracy of 97.1% for MSID and 98.7% for MSLD datasets respectively. Such promising results demonstrate that the Monkey-CAD can be employed to assist health practitioners. They also verify that fusing deep features from selected CNNs can boost performance.

Segmentation ability map: Interpret deep features for medical image segmentation.

Depth cameras started to gain popularity in agricultural applications during the last years. This type of cameras has been implemented mainly for three-dimensional (3D) reconstruction of objects in indoor or outdoor sceneries. The use of such cameras in the construction of 3D models for simulation purposes of complex structures that are usually met in nature, such as trees and other plants, is a great challenge. Remarkably, agricultural environments are extremely complex. Thus, the proper setup and implementation of such technologies is particularly important in order to attain useable data. So far, the depth information collected using these cameras varies among different objects’ structure and sensing conditions due to the uncertainty of the outdoor environment. The use of a specific methodology using color and depth images gives the opportunity to extract geometrical characteristics information about point clouds of the targeted objects. This chapter explores the different technologies used by depth cameras and presents several applications concerning indoor and outdoor environments by presenting indicative scenarios for agricultural applications. Towards that direction, a 3D reconstruction of trees was established producing point clouds from Red Green Blue Depth (RGB-D) images acquired in real field conditions. The point cloud samples of trees were collected using an unmanned ground vehicle (UGV) and imported in Gazebo in order to visualize a simulation of the environment. This simulation technique can be used for testing and evaluating the navigation of robotic systems. By further analyzing the resulted 3D point clouds, various geometrical measurements of the simulated samples, such as the volume or the height of tree canopies, can be calculated. Possible weaknesses of this procedure are mainly attributed to the camera’s limitations and the sampling parameters. However, results show that it is possible to establish a suitable simulation environment to implement it in several agricultural applications by utilizing automated unmanned robotic platforms.KeywordsDepth camerasUnmanned ground vehiclesScene reconstruction3D point cloudAgricultural robotsSituation awareness

Efficient and reliable grasp pose generation plays a crucial role in robotic manipulation tasks. The advancement of deep learning techniques applied to point cloud data has led to rapid progress in grasp detection. However, point cloud data has limitations: no appearance information and susceptibility to sensor noise. In contrast, color Red, Green, Blue (RGB) images offer high-resolution and intricate textural details, making them a valuable complement to the 3-D geometry offered by point clouds or depth (D) images. Nevertheless, the effective integration of appearance information to enhance point cloud-based grasp detection remains an open question. In this study, we extend the concepts of VoteGrasp <xref ref-type="bibr" rid="ref1">[1]</xref> and introduce an innovative deep learning approach referred to as VoteGrasp Red, Green, Blue, Depth (RGBD). To build robustness to occlusion, the proposed model generates candidates by casting votes and accumulating evidence for feasible grasp configurations. This methodology revolves around fuzing votes extracted from images and point clouds. To further enhance the collaborative effect of merging appearance and geometry features, we introduce a context learning module. We exploit contextual information by encoding the dependency of objects in the scene into features to boost the performance of grasp generation. The contextual information enables our model to increase the likelihood that the generated grasps are collision-free. The efficacy of our model is verified through comprehensive evaluations on the demanding GraspNet-1Billion dataset, leading to a significant improvement of 9.3 in average precision (AP) over the existing state-of-the-art results. Additionally, we provide extensive analyses through ablation studies to elucidate the contributions of each design decision.

Cross-modal retrieval, which aims to solve the problem that the query and the retrieved results are from different modality, becomes more and more essential with the development of the Internet. In this paper, we mainly focus on the exploration of high-level semantic representation of image and text for cross-modal matching. Deep convolutional image features and Fisher Vector with neural word embeddings are utilized as visual and textual features respectively. To further investigate the correlation among heterogeneous multimodal characteristics, we use multiclass logistic classifier for semantic matching across modalities. Experiments on Wikipedia and Pascal Sentence dataset demonstrate the robustness and effectiveness for both Img2Text and Text2Img retrieval tasks.

Glioblastoma is a highly aggressive brain tumor with limited survival that poses challenges in predicting patient outcomes. The Karnofsky Performance Status (KPS) score is a valuable tool for assessing patient functionality and contributes to the stratification of patients with poor prognoses. This study aimed to develop a 6-month postoperative KPS prediction model by combining clinical data with deep learning-based image features from pre- and postoperative MRI scans, offering enhanced personalized care for glioblastoma patients. Using 1,476 MRI datasets from the Brain Tumor Segmentation Challenge 2020 public database, we pretrained two variational autoencoders (VAEs). Imaging features from the latent spaces of the VAEs were used for KPS prediction. Neural network-based KPS prediction models were developed to predict scores below 70 at 6 months postoperatively. In this retrospective single-center analysis, we incorporated clinical parameters and pre- and postoperative MRI images from 150 newly diagnosed IDH wild-type glioblastoma, divided into training (100 patients) and test (50 patients) sets. In training set, the performance of these models was evaluated using the area under the curve (AUC), calculated through fivefold cross-validation repeated 10 times. The final evaluation of the developed models assessed in the test set. Among the 150 patients, 61 had 6-month postoperative KPS scores below 70 and 89 scored 70 or higher. We developed three models: a clinical-based model, an MRI-based model, and a multimodal model that incorporated both clinical parameters and MRI features. In the training set, the mean AUC was 0.785±0.051 for the multimodal model, which was significantly higher than the AUCs of the clinical-based model (0.716±0.059, P = 0.038) using only clinical parameters and the MRI-based model (0.651±0.028, P<0.001) using only MRI features. In the test set, the multimodal model achieved an AUC of 0.810, outperforming the clinical-based (0.670) and MRI-based (0.650) models. The integration of MRI features extracted from VAEs with clinical parameters in the multimodal model substantially enhanced KPS prediction performance. This approach has the potential to improve prognostic prediction, paving the way for more personalized and effective treatments for patients with glioblastoma.