An Efficient Network for Lane Segmentation

With the leaps and bounds of computer performance and the advent of the era of big data, deep learning has drawn more and more attention from all walks of life. It can combine low-level features to form more abstract high-level features and describe the data more essentially. Therefore, it is widely used in various fields such as computer vision. Image segmentation is one of the most basic research topics in the field of computer vision. The main purpose is to extract regions of interest from images for later image processing research. In 3D reconstruction based on sequence images, the segmentation accuracy and speed of sequence images determine the quality and efficiency of target reconstruction. Therefore, when facing large-scale sequence images, the biggest problem is how to improve the segmentation speed while ensuring accuracy. Based on the above background, the research content of this article is an indoor scene segmentation algorithm based on full convolutional neural network. According to the characteristics of indoor application scenes, this paper proposes a fast convolutional neural network image segmentation method to segment the indoor scene image and construct the fast fully convolutional networks (FFCN) for indoor scene image segmentation uses inter-layer fusion to reduce the amount of network calculation parameters and avoid the loss of picture feature information by continuous convolution. In order to verify the effectiveness of the network, in this paper, a basic living object data set (XAUT data set) in an indoor environment is created. The XAUT data set is used to train the FFCN network under the Caffe framework to obtain an indoor scene segmentation model. In order to compare the effectiveness of the model, the structure of the worn FCN8s, FCN16s, and FCN32s models was fine-tuned, and the corresponding algorithm model for indoor scene segmentation was obtained by training with the XAUT data set. The experimental results show that the pixel recognition accuracy of all types of networks has reached 86%, and the mean IU ratio has reached more than 63%. The mean IU of the FCN8s network is the highest at 70.38%, but its segmentation speed is only 1/5 of FFCN. On the premise that other types of indicators are not much different, the average segmentation speed on FFCN fast segmentation convolutional neural network reaches 40 fps. It can be seen that the scale fusion technology can well avoid the loss of image feature information in the network convolution and reddening process. Compared with other FCN networks, it has a faster speed and is conducive to real-time image preprocessing.

Convolution neural networks (CNNs) have proven to be effective clinical imaging methods. This study highlighted some of the key issues within these systems. It is difficult to train these systems in a limited clinical image databases, and many publications present strategies including such learning algorithm. Furthermore, these patterns are known for making a highly reliable prognosis. In addition, normalization of volume and losses of dice have been used effectively to accelerate and stabilize the training. Furthermore, these systems are improperly regulated, resulting in more confident ratings for correct and incorrect classification, which are inaccurate and difficult to understand. This study examines the risk assessment of Fully Convolutional Neural Networks (FCNNs) for clinical image segmentation. Essential contributions have been made to this planned work: 1) dice loss and cross-entropy loss are compared on the basis of segment quality and uncertain assessment of FCNNs; 2) proposal for a group model for assurance measurement of full convolutional neural networks trained with dice loss and group normalization; And 3) the ability of the measured FCNs to evaluate the segment quality of the structures and to identify test examples outside the distribution. To evaluate the study’s contributions, it conducted a series of tests in three clinical image division applications such as heart, brain and prostate. The findings of the study provide significant insights into the predictive ambiguity assessment and a practical strategies for outside-distribution identification and reliable measurement in the clinical image segmentation. The approaches presented in this research significantly enhance the reliability and accuracy rating of CNN-based clinical imaging methods.

Abdominal image segmentation is a challenging, yet important clinical problem. Variations in body size, position, and relative organ positions greatly complicate the segmentation process. Historically, multi-atlas methods have achieved leading results across imaging modalities and anatomical targets. However, deep learning is rapidly overtaking classical approaches for image segmentation. Recently, Zhou et al. showed that fully convolutional networks produce excellent results in abdominal organ segmentation of computed tomography (CT) scans. Yet, deep learning approaches have not been applied to whole abdomen magnetic resonance imaging (MRI) segmentation. Herein, we evaluate the applicability of an existing fully convolutional neural network (FCNN) designed for CT imaging to segment abdominal organs on T2 weighted (T2w) MRI's with two examples. In the primary example, we compare a classical multi-atlas approach with FCNN on forty-five T2w MRI's acquired from splenomegaly patients with five organs labeled (liver, spleen, left kidney, right kidney, and stomach). Thirty-six images were used for training while nine were used for testing. The FCNN resulted in a Dice similarity coefficient (DSC) of 0.930 in spleens, 0.730 in left kidneys, 0.780 in right kidneys, 0.913 in livers, and 0.556 in stomachs. The performance measures for livers, spleens, right kidneys, and stomachs were significantly better than multi-atlas (p < 0.05, Wilcoxon rank-sum test). In a secondary example, we compare the multi-atlas approach with FCNN on 138 distinct T2w MRI's with manually labeled pancreases (one label). On the pancreas dataset, the FCNN resulted in a median DSC of 0.691 in pancreases versus 0.287 for multi-atlas. The results are highly promising given relatively limited training data and without specific training of the FCNN model and illustrate the potential of deep learning approaches to transcend imaging modalities.

Land cover and its change are crucial for many environmental applications. This study focuses on the land cover classification and change detection with multitemporal and multispectral Sentinel-2 satellite data. To address the challenging land cover change detection task, we rely on two different deep learning architectures and selected pre-processing steps. For example, we define an excluded class and deal with temporal water shoreline changes in the pre-processing. We employ a fully convolutional neural network (FCN), and we combine the FCN with long short-term memory (LSTM) networks. The FCN can only handle monotemporal input data, while the FCN combined with LSTM can use sequential information (multitemporal). Besides, we provided fixed and variable sequences as training sequences for the combined FCN and LSTM approach. The former refers to using six defined satellite images, while the latter consists of image sequences from an extended training pool of ten images. Further, we propose measures for the robustness concerning the selection of Sentinel-2 image data as evaluation metrics. We can distinguish between actual land cover changes and misclassifications of the deep learning approaches with these metrics. According to the provided metrics, both multitemporal LSTM approaches outperform the monotemporal FCN approach, about 3 to 5 percentage points (p.p.). The LSTM approach trained on the variable sequences detects 3 p.p. more land cover changes than the LSTM approach trained on the fixed sequences. Besides, applying our selected pre-processing improves the water classification and avoids reducing the dataset effectively by 17.6%. The presented LSTM approaches can be modified to provide applicability for a variable number of image sequences since we published the code of the deep learning models. The Sentinel-2 data and the ground truth are also freely available.

With the success of convolutional neural network (CNN) in the field of image processing and image recognition, an improved fully convolutional neural network (FCNN) based on CNN is proposed. Although FCNN has better performance than CNN, the number of parameters of FCNN has also increased significantly compared to CNN. Consequently, this paper proposes a multi‐feature fusion decision convolutional neural network (MFFD‐CNN), which use two kinds down‐sample methods in the pooling layer, and chooses the concatenate or add operation as the way that consolidates the feature of the convolutional layer with a stride of 2 and the max‐pooling layer to execute fusion operations. Meanwhile replaces the traditional softmax loss with an additive margin Softmax (AM‐Softmax) loss. The structure, which is based on the multi‐target classification experiment results of the MSTAR data set, shows can not only obtain an average correct recognition rate 1.2% higher than FCNN, but also better stability without augmenting the training samples of MSTAR data.

Automatic segmentation of uterus and fetus from 3D fetal ultrasound images remains a challenging problem due to multiple issues of fetal ultrasound, e.g., the relatively low image quality, intensity variations. In this work, we present a novel framework for the joint segmentation of uterus and fetus. It consists of two main components: a task-specific fully convolutional neural network (FCN) and a bidirectional convolutional LSTM (BiCLSTM). Our framework is inspired by a simple observation: the segmentation task can be decomposed into multiple easier-to-solve subproblems. More specifically, the encoder of the FCN extracts object-relevant features from the ultrasound slices. The BiCLSTM layer is responsible for modeling the inter-slice correlations. The final two branches of the FCN decoder produce the uterus and fetus predictions. In this way, the burden of the whole problem is evenly distributed among different parts of our network, thereby maximally exploiting the capacity of our network. Furthermore, we propose a spatially constrained loss to restrict the spatial positions of the segmented uterus and fetus to boost the performance. Quantitative results demonstrate the effectiveness of the proposed method.KeywordsFetal ultrasonographyUterus and fetus segmentationBidirectional convolutional LSTMTask-specific FCN

Automated identification and localization of vertebrae in spinal computed tomography (CT) imaging is a complicated hybrid task. This task requires detecting and indexing a long sequence in a 3-D image, and both image feature extraction and sequence modeling are needed to address the problem. In this paper, the powerful fully convolutional neural network (FCN) technique performs both of these tasks simultaneously because FCNs directly encode and decode the spatial interdependence of different components in images. The key module of our proposed framework is a 3-D FCN trained in an end-to-end manner at the spine level to capture the long-range contextual information in CT volumes. The large increase in the calculation due to the full-size image inputs is alleviated by the scale-down of the inputs and the use of an auxiliary FCN to compensate for the loss of details. The composite network pipeline design enables the integration of local image details and global image patterns. Furthermore, explicit spatial and sequential constraints are imposed by the hidden Markov model (HMM) for a higher robustness and a clearer interpretation of network outputs. The proposed framework is quantitatively evaluated on the public dataset from the MICCAI 2014 Computational Challenge on Vertebrae Localization and Identification and demonstrates an identification rate (within 20 mm) of 94.67%, a mean identification rate of 87.97%, and a mean error distance of 2.56 mm on the test set, thus achieving the highest performance reported on this dataset.

Diabetic retinopathy data augmentation and vessel segmentation through deep learning based three fully convolution neural networks

Reverberation clutter is image noise resulting from multiple reflections between tissue layers and leads to image degradation. We propose a 3D fully convolutional neural network (FCNN) to selectively remove reverberation noise from ultrasound channel data. A training set was generated using Field II Pro to simulate full synthetic aperture channel data. Reverberation noise was approximated by adding bandpass filtered noise to the noise-free simulated channel data. This channel data, with and without noise, was used to train a 3D FCNN with a custom architecture to remove the reverberation and thermal noise. The network was evaluated on a simulated validation set and on an ATS 549 phantom using a L12-3v transducer connected to a Verasonics Vantage 256 system with reverberation noise. The resulting network shows significant reverberation noise reduction. Across the validation set, the normalized RMS loss between the noisy and noise free images decreased from 77% to 39% after being decluttered using 3D FCNN.

This research is motivated by sustainability problems of oil palm expansion. Fast-growing industrial Oil Palm Plantations (OPPs) in the tropical belt of Africa, Southeast Asia and parts of Brazil lead to significant loss of rainforest and contribute to the global warming by the corresponding decrease of carbon dioxide absorption. We propose a novel approach to monitoring of the expansion of OPPs based on an application of state-of-the-art Fully Convolutional Neural Networks (FCNs) to solve Semantic Segmentation Problem for Landsat imagery. The proposed approach significantly outperforms per-pixel classification methods based on Random Forest using texture features, NDVI, and all Landsat bands. Moreover, the trained FCN is robust to spatial and temporal shifts of input data. The paper provides a proof of concept that FCNs as semi-automated methods enable OPPs mapping of entire countries and may serve for yearly detection of oil palm expansion.

Documenting the impacts of climate change and human activities on tropical rainforests is imperative for protecting tropical biodiversity and for better implementation of REDD+ and UN Sustainable Development Goals. Recent advances in very high-resolution satellite sensor systems (i.e., WorldView-3), computing power, and machine learning (ML) have provided improved mapping of fine-scale changes in the tropics. However, approaches so far focused on feature extraction or the extensive tuning of ML parameters, hindering the potential of ML in forest conservation mapping by not using textural information, which is found to be powerful for many applications. Additionally, the contribution of shortwave infrared (SWIR) bands in forest cover mapping is unknown. The objectives were to develop end-to-end mapping of the tropical forest using fully convolution neural networks (FCNNs) with WorldView-3 (WV-3) imagery and to evaluate human impact on the environment using the Betampona Nature Reserve (BNR) in Madagascar as the test site. FCNN (U-Net) using spatial/textural information was implemented and compared with feature-fed pixel-based methods including Support Vector Machine (SVM), Random Forest (RF), and Deep Neural Network (DNN). Results show that the FCNN model outperformed other models with an accuracy of 90.9%, while SVM, RF, and DNN provided accuracies of 88.6%, 84.8%, and 86.6%, respectively. When SWIR bands were excluded from the input data, FCNN provided superior performance over other methods with a 1.87% decrease in accuracy, while the accuracies of other models—SVM, RF, and DNN—decreased by 5.42%, 3.18%, and 8.55%, respectively. Spatial–temporal analysis showed a 0.7% increase in Evergreen Forest within the BNR and a 32% increase in tree cover within residential areas likely due to forest regeneration and conservation efforts. Other effects of conservation efforts are also discussed.

Fully convolutional neural network (FCN) has been dominating the game of face detection task for a few years with its congenital capability of sliding-window-searching with shared kernels, which boiled down all the redundant calculation, and most recent state-of-the-art methods such as Faster-RCNN, SSD, YOLO and FPN use FCN as their backbone. So here comes one question: Can we find a universal strategy to further accelerate FCN with higher accuracy, so could accelerate all the recent FCN-based methods? To analyze this, we decompose the face searching space into two orthogonal directions, 'scale' and 'spatial'. Only a few coordinates in the space expanded by the two base vectors indicate foreground. So if FCN could ignore most of the other points, the searching space and false alarm should be significantly boiled down. Based on this philosophy, a novel method named scale estimation and spatial attention proposal (S2AP) is proposed to pay attention to some specific scales in image pyramid and valid locations in each scales layer. Furthermore, we adopt a masked-convolution operation based on the attention result to accelerate FCN calculation. Experiments show that FCN-based method RPN can be accelerated by about 4A— with the help of S2AP and masked-FCN and at the same time it can also achieve the state-of-the-art on FDDB, AFW and MALF face detection benchmarks as well.

Real-time path planning is the core of autonomous driving. It plans the optimal drive path for the autonomous vehicle (AV) based on concurrent perception and localization information. Due to the uncertainties in those information, it is necessary to plan multiple paths in real-time for selection, which, however, is expensive for traditional graph search methods. In recent years, path learning methods based on semantic segmentation proved that the driving area of an AV can be predicted by learning perception information and its historical paths. Inspired by these methods, this paper proposes a real-time multiple path planning method combining semantic segmentation with the traditional graph-based search. A fully convolutional neural network (FCN) was first designed to learn the optimal path area generated by an A* based path planning method in various real and simulated environments. By injecting noises into localization information, the generalization ability of the neural network is greatly enhanced facing inaccurate localization results. Then, multiple possible path areas inferred by the FCN are adopted as constraints for the following A* based path planning. The results showed that the optimal path area reduces the computational cost of subsequent path searches, which makes real-time multiple path planning possible. Besides, even if the neural network infers an unreasonable path area, the A* based path search method can still guarantee the correctness of the path searching.

Abstract. Fully convolutional neural networks (FCN) are successfully used for pixel-wise land cover classification - the task of identifying the physical material of the Earth’s surface for every pixel in an image. The acquisition of large training datasets is challenging, especially in remote sensing, but necessary for a FCN to perform well. One way to circumvent manual labelling is the usage of existing databases, which usually contain a certain amount of label noise when combined with another data source. As a first part of this work, we investigate the impact of training data on a FCN. We experiment with different amounts of training data, varying w.r.t. the covered area, the available acquisition dates and the amount of label noise. We conclude that the more data is used for training, the better is the generalization performance of the model, and the FCN is able to mitigate the effect of label noise to a high degree. Another challenge is the imbalanced class distribution in most real-world datasets, which can cause the classifier to focus on the majority classes, leading to poor classification performance for minority classes. To tackle this problem, in this paper, we use the cosine similarity loss to force feature vectors of the same class to be close to each other in feature space. Our experiments show that the cosine loss helps to obtain more similar feature vectors, but the similarity of the cluster centers also increases.

Instrument segmentation plays a vital role in 3D ultrasound (US) guided cardiac intervention. Efficient and accurate segmentation during the operation is highly desired since it can facilitate the operation, reduce the operational complexity, and therefore improve the outcome. Nevertheless, current image-based instrument segmentation methods are not efficient nor accurate enough for clinical usage. Lately, fully convolutional neural networks (FCNs), including 2D and 3D FCNs, have been used in different volumetric segmentation tasks. However, 2D FCN cannot exploit the 3D contextual information in the volumetric data, while 3D FCN requires high computation cost and a large amount of training data. Moreover, with limited computation resources, 3D FCN is commonly applied with a patch-based strategy, which is therefore not efficient for clinical applications. To address these, we propose a POI-FuseNet, which consists of a patch-of-interest (POI) selector and a FuseNet. The POI selector can efficiently select the interested regions containing the instrument, while FuseNet can make use of 2D and 3D FCN features to hierarchically exploit contextual information. Furthermore, we propose a hybrid loss function, which consists of a contextual loss and a class-balanced focal loss, to improve the segmentation performance of the network. With the collected challenging ex-vivo dataset on RF-ablation catheter, our method achieved a Dice score of 70.5%, superior to the state-of-the-art methods. In addition, based on the pre-trained model from ex-vivo dataset, our method can be adapted to the in-vivo dataset on guidewire and achieves a Dice score of 66.5% for a different cardiac operation. More crucially, with POI-based strategy, segmentation efficiency is reduced to around 1.3 seconds per volume, which shows the proposed method is promising for clinical use.