Trapezoidal Segmented Regression: A Novel Continuous-scale Real-time Annotation Approximation Algorithm

Identification of clouds and their shadows are two major preprocessing steps for providing an effective interpretation of remotely sensed optical images. Although deep learning-based methods have proved to deliver competent performance for detecting clouds and cloud shadows, improving their generalization ability for reliable results requires a large number of training images and accurate ground truths. As creating ground truth of cloud and cloud shadow is expensive, a practical way to generate more images and their ground truths is to use data augmentation methods. We propose two new data augmentation approaches (one for generating synthetic clouds in scenes and the other one for creating cloud shadows with different levels of shade) so as to achieve natural-looking images with little or no effort required for creating their ground truths. Our experiments show that while each of these approaches is capable of boosting cloud and cloud shadow segmentation individually, the fusion of them improves detecting cloud and shadow in a simultaneous learning pipeline, leading to outperforming the state-of-the-art results.

In this paper, we study a practical omnidirectional depth estimation with neural networks that enables effective learning on real world data obtained using wide-baseline multiple fish-eye cameras. Most previous approaches only used synthetic data providing dense and accurate depth ground truth (GT). However, it is unrealistic to acquire such high quality GT data in real world due to limitations of the existing depth sensors. We first introduce two critical problems that can reduce the accuracy of depth estimation: depth GT sparsity and sensor calibration error. We then propose a novel semi-supervised learning method using pixel-level loss that selectively uses supervised loss and unsupervised re-projection loss according to existence of GT. Empirical results demonstrate that our method efficiently reduces the performance degradation in both simulation on synthetic data and real world data using sparse depth sensor.

Effective land monitoring and land use classification are critical for proper management of resources especially in heterogeneous and climate diverse areas. Consequently, this study seeks to test the hypothesis that the integration of Sentinel-1 radar and Sentinel-2 optical data enhances the degree of discrimination of crops in major farming areas of Morocco from the years 2020 to 2022. A three-dimensional coordinate system was established which included a series of processing stages that started with cloud masking, scaling of reflectance, and radar optical integration. At each year’s end, temporal averages and composites were created using selected Sentinel-2 spectral bands B2, B3, B4, B8, B11, B12 and Sentinel-1 VV & VH dual polarization channels. Ground truth samples from four major crops; Baley, Crop, D. Wheat and S. Wheat were used as the training set in a Random Forest classifier. The results for the three agricultural zones indicated high overall accuracies greater than 80% for each year, with the application of a combination of radar and optical data sets contributing greatly towards the ability to differentiate the crops located in cloud folded and spectral overlapping areas. Many classes had high consumer accuracy (≥70%) levels, yet several crops, like D. Wheat, had poor producer accuracy, possibly due to the uneven distribution of ground truth data sets. The small amount of Kappa coefficients between 0.50 and 0.60 also indicate moderate agreement similar to the validation data and thus more accurate ground truth and class targeted feature detection is needed. This study emphasizes the relevance notes of the multi-sensor data fusion technology for crop monitoring and also landcover classification which contributes to precision farming and resources management. Future work will focus on including temporal characteristics as well as state-of-the-art machine learning techniques to solve class balance issues and improve classification performance.

A number of real-time object detection, tracking, and autonomy artificial intelligence (AI) and machine learning (ML) algorithms are being proposed for unmanned aerial vehicles (UAVs). A big challenge is can we stress test these algorithms, identify their strengths and weaknesses, and assess if the UAV is safe and trustworthy? The process of collecting real-world UAV data is costly, time consuming, and riddled by lack of quality geospatial ground truth and metadata. Herein, we outline a fully automated framework and work ow to address the above challenges using free or low-cost assets, the photorealistic Unreal Engine (UE), and AirSim aerial platform simulator. Specifically, we discuss the rapid prototyping of an outdoor environment combined with the robotic operating system (ROS) for abstracting UAV data collection, control, and processing. Real and accurate ground truth is collected and metrics are presented for individual frame and entire flight collection evaluation. Metrics recorded and analyzed include percentage of scene mapped, 3D mapping accuracy, time to complete task, object detection and tracking statistics, battery usage, altitude (from ground), collisions, and other statistics. These metrics are computed in general and with respect to context, e.g., clutter, view angle, etc. Overall, the proposed work is an automated way to explore UAV operation before real-world testing or deployment. Promising preliminary results are discussed for an outdoor environment with vegetation, short and long range objects, buildings, people, vehicles, and other features for a UAV performing loitering and interrogation.

Confidence measures for stereo analysis are not yet a subject of detailed comparative evaluations. There have been some studies, but still insufficient for estimating the performance of these measures. We comparatively discuss confidence measures whose performance appeared to be ‘promising’ to us, by evaluating their performance on commonly used stereo test data. Those data are either engineered and come with accurate ground truth (for disparities), or they are recorded outdoors and come with approximate ground truth. The performance of confidence measures varies widely between these two types of data. We propose modifications of confidence measures which can improve their performance on outdoor data.

Esophageal adenocarcinoma (EAC) is a deadly cancer that is rising rapidly in incidence. The early detection of EAC with curative intervention greatly improves the prognoses of patients. A scanning fiber endoscope (SFE) using fluorescence-labeled peptides that bind rapidly to epidermal growth factor receptors showed a promising performance for early EAC detection. Target-to-background (T/B) ratios were calculated to quantify the fluorescence images for neoplasia lesion classification. This T/B calculation is generally based on lesion segmentation with the Chan–Vese algorithm, which may require hyperparameter adjustment when segmenting frames with different brightness and contrasts, which impedes automation to real-time video. Deep learning models are more robust to these changes, while accurate pixel-level segmentation ground truth is challenging to establish in the medical field. Since within our dataset the ground truth contained only a frame-level diagnosis, we proposed a computer-aided diagnosis (CAD) system to calculate the T/B ratio in real time. A two-step process using convolutional neural networks (CNNs) was developed to achieve automatic suspicious frame selection and lesion segmentation for T/B calculation. In the segmentation model training for Step 2, the lesion labels were generated with a manually tuned Chan–Vese algorithm using the labeled and predicted suspicious frames from Step 1. In Step 1, we designed and trained deep CNNs to select suspicious frames using a diverse and representative set of 3427 SFE images collected from 25 patient videos from two clinical trials. We tested the models on 1039 images from 10 different SFE patient videos and achieved a sensitivity of 96.4%, a specificity of 96.6%, a precision of 95.5%, and an area under the receiver operating characteristic curve of 0.989. In Step 2, 1006 frames containing suspicious lesions were used for training for fluorescence target segmentation. The segmentation models were tested on two clinical datasets with 100 SFE frames each and achieved mean intersection-over-union values of 0.89 and 0.88, respectively. The T/B ratio calculations based on our segmentation results were similar to the manually tuned Chan–Vese algorithm, which were 1.71 ± 0.22 and 1.72 ± 0.28, respectively, with a p-value of 0.872. With the graphic processing unit (GPU), the proposed two-step CAD system achieved 50 fps for frame selection and 15 fps for segmentation and T/B calculation, which showed that the frame rejection in Step 1 improved the diagnostic efficiency. This CAD system with T/B ratio as the real-time indicator is designed to guide biopsies and surgeries and to serve as a reliable second observer to localize and outline suspicious lesions highlighted by fluorescence probes topically applied in organs where cancer originates in the epithelia.

In lung cancer computer-aided diagnosis CAD systems, having an accurate and available ground truth is critical and time consuming. In this study, we have explored Lung Image Database Consortium LIDC database containing pulmonary computed tomography CT scans, and we have implemented content-based image retrieval CBIR approach to exploit the limited amount of diagnostically labeled data in order to annotate unlabeled images with diagnoses. By applying CBIR method iteratively and using pathologically confirmed cases, we expand the set of diagnosed data available for CAD systems from 17 nodules to 121 nodules. We evaluated the method by implementing a CAD system that uses various combinations of lung nodule sets as queries and retrieves similar nodules from the diagnostically labeled dataset. In calculating the precision of this system Diagnosed dataset and computer-predicted malignancy data are used as ground truth for the undiagnosed query nodules. Our results indicate that CBIR expansion is an effective method for labeling undiagnosed images in order to improve the performance of CAD systems. It also indicated that little knowledge of biopsy confirmed cases not only assist the physician's as second opinion to mark the undiagnosed cases and avoid unnecessary biopsies too.

In many remote sensing and hyperspectral image analysis applications, precise ground truth information is unavailable or impossible to obtain. Imprecision in ground truth often results from highly mixed or sub-pixel spectral responses over classes of interest, a mismatch between the precision of global positioning system (GPS) units and the spatial resolution of collected imagery, and misalignment between multiple sources of data. Given these sorts of imprecision, training of traditional supervised machine learning models which rely on the assumption of accurate and precise ground truth becomes intractable. Multiple instance learning (MIL) is a methodology that can be used to address these challenging problems. This chapter investigates the topic of hyperspectral image analysis given imprecisely labeled data and reviews MIL methods for hyperspectral target detection, classification, data fusion, and regression.

Monitoring the extent of plateau forests has drawn much attention from governments given the fact that the plateau forests play a key role in global carbon circulation. Despite the recent advances in the remote-sensing applications of satellite imagery over large regions, accurate mapping of plateau forest remains challenging due to limited ground truth information and high uncertainties in their spatial distribution. In this paper, we aim to generate a better segmentation map for plateau forests using high-resolution satellite imagery with limited ground-truth data. We present the first 2 m spatial resolution large-scale plateau forest dataset of Sanjiangyuan National Nature Reserve, including 38,708 plateau forest imagery samples and 1187 handmade accurate plateau forest ground truth masks. We then propose an few-shot learning method for mapping plateau forests. The proposed method is conducted in two stages, including unsupervised feature extraction by leveraging domain knowledge, and model fine-tuning using limited ground truth data. The proposed few-shot learning method reached an F1-score of 84.23%, and outperformed the state-of-the-art object segmentation methods. The result proves the proposed few-shot learning model could help large-scale plateau forest monitoring. The dataset proposed in this paper will soon be available online for the public.

Drilling and logging of Integrated Ocean Drilling Program (IODP) Hole 1256D have provided a unique opportunity for systematically studying a fundamental problem in marine geophysics: What influences the seismic structure of oceanic crust, porosity or composition? Compressional wave velocities (Vp) logged in open hole or from regional refraction measurements integrate both the host rock and cracks in the crust. To determine the influence of cracks on Vp at several scales, we first need an accurate ground truth in the form of laboratory Vp on crack-free, or nearly crack-free samples. We measured Vp on 46 water-saturated samples at in situ pressures to determine the baseline velocities of the host rock. These new results match or exceed Vp logs throughout most of the hole, especially in the lower dikes and gabbros, where porosities are low. In contrast, samples measured at sea under ambient laboratory conditions, had consistently lower Vp than the Vp logs, even after correction to in situ pressures. Crack-free Vp calculated from simple models of logging and laboratory porosity data for different lithologies and facies suggest that crustal velocities in the lavas and upper dikes are controlled by porosity. In particular, the models demonstrate significant large-scale porosity in the lavas, especially in the sections identified as fractured flows and breccias. However, crustal velocities in the lower dikes and gabbros are increasingly controlled by petrology as the layer 2-3 boundary is approached.

Publicly available satellite imagery can be an ubiquitous, cheap, and powerful tool for vehicle localisation when a prior sensor map is unavailable.However, satellite images are not directly comparable to data from ground range sensors because of their starkly different modalities.We present a learned metric localisation method that not only handles the modality difference, but is cheap to train, learning in a self-supervised fashion without metrically accurate ground truth.By evaluating across multiple real-world datasets, we demonstrate the robustness and versatility of our method for various sensor configurations.We pay particular attention to the use of millimetre wave radar, which, owing to its complex interaction with the scene and its immunity to weather and lighting, makes for a compelling and valuable use case.

Driver attention estimation is one of the key technologies for intelligent vehicles. The existing related methods only focus on the scene image or the driver's gaze or head pose. The purpose of this article is to propose a more reasonable and feasible method based on a dual-view scene with calibration-free gaze direction. According to human visual mechanisms, the low-level features, static visual saliency map, and dynamic optical flow information are extracted as input feature maps, which combine the high-level semantic descriptions and a gaze probability map transformed from the gaze direction. A multiresolution neural network is proposed to handle the calibration-free features. The proposed method is verified on a virtual reality experimental platform that collected more than 550 000 samples and obtained a more accurate ground truth. The experiments show that the proposed method is feasible and better than the state-of-the-art methods based on multiple widely used metrics. This study also provides a discussion of the effects of different landscapes, times, and weather conditions on the performance.

Targets interrogated during synthetic aperture sonar (SAS) applications may exhibit elastic scattering behavior and re-radiate sound after the initial geometric returns. This behavior can be a discriminating feature in target identification, but may also obscure nearby targets and make the full scene difficult to comprehend. Advancing algorithms to leverage elastic scattering for target localization and identification requires data with accurate ground truth. There is a shortage of field data containing elastic targets with sufficient quality for algorithm development. Stanton’s 1984 model presents the scattering amplitude of an elastic cylinder as a function of frequency and spherical angles. This presentation uses Stanton’s scattering model to simulate the time series of pressure scattered by the cylinder. This approach allows the target scattering model to be readily coupled with time series models for the environment to generate high fidelity simulated data. Simulated time series are generated for SAS collection geometries. The presentation will explore this modeling approach as a substitute for field data during initial stages of algorithm development for improved target localization. It will investigate the impact of elastic effects on imagery and discuss the limitations of methods used to isolate such effects.

The vast majority of modern consumer cameras employ a rolling shutter (RS) mechanism which has a price and electronic advantage to global shutter (GS). However, in geometric computer vision applications such as visual simultaneous localization and mapping (VSLAM), performances of accuracy and robustness are usually deteriorated due to the rolling shutter effect when using the RS cameras. This paper introduced the Wuhan University Rolling Shutter Visual-Inertial (WHU-RSVI) synthetic dataset for evaluating VSLAM and VI-SLAM (visual-inertial SLAM) methods in which RS cameras or IMU data are typically used. The proposed synthetic dataset contains RS images, time-synchronized GS images, inertial measurement unit (IMU) measurements, and accurate ground truth. It provides camera images with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480 $ </tex-math></inline-formula> resolution at 30 Hz and IMU measurements from 90 Hz to 14400 Hz. The cubic B-spline curves are used to model the motion of trajectories. Based on the known trajectories, an image of each pose can be rendered, and the corresponding IMU measurement model is then established. The dataset provides realistic images and IMU measurements by modeling the sensor noise in RGB and IMU data. Two trajectories with three sequences of different motion speeds (i. e., slow, medium and fast corresponding to different rolling shutter effects) are contained in the proposed dataset. Herein, the proposed dataset can be applied to compare the impact of different rolling shutter effects on a specific method.

Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we propose a novel learning-based multi-view stereo method to estimate 3D information. To better utilize the information from multiple frames, an elevation plane sweeping method is proposed to generate the depth-azimuth-elevation cost volume. The volume after regularization can be considered as a probabilistic volumetric representation of the target. Instead of performing regression on the elevation angles, we use pseudo front depth from the cost volume to represent the 3D information which can avoid the 2D-3D problem in acoustic imaging. High-accuracy results can be generated with only two or three images. Synthetic datasets were generated to simulate various underwater targets. We also built the first real dataset with accurate ground truth in a large scale water tank. Experimental results demonstrate the superiority of our method, compared to other state-of-the-art methods.