Deriving High Spatiotemporal Remote Sensing Images Using Deep Convolutional Network

To design a more accurate and reproducible technique for the measurement of blood-brain barrier (BBB) permeability in gadolinium-enhancing multiple sclerosis (MS) lesions. Four MS patients were scanned using a new dynamic contrast-enhanced (DCE)-magnetic resonance imaging (MRI) protocol based on an uninterrupted two-part acquisition consisting of an initial part at high temporal and low spatial resolutions and a second part at low temporal and high spatial resolutions. The method preserves both the high spatial resolution needed for the often small size of lesions and the high temporal resolution required during the first minute after injection to sufficiently sample the first-pass bolus. Simulations compared the performance of this new protocol with the conventional one at low temporal and high spatial resolutions throughout. The BBB permeability estimates changed by up to 33% between the two protocols. The new protocol led to simulated error on K(trans) of 7%-10%, versus 7%-30% with the conventional protocol, and was more robust with respect to offsets between acquisition and injection start times, differences in shape of the first-pass peak, and permeability values. The dual-temporal resolution protocol produces improved BBB permeability estimates and provides a more complete view of active inflammatory MS lesion pathology.

MODIS and ASTER synergy for characterizing thermal volcanic activity

Hyperspectral images (HSI) have high spectral and low spatial resolutions. However, multispectral images (MSI) usually have low spectral and high spatial resolutions. In various applications HSI with high spectral and spatial resolutions are required. In this paper, a new method for spatial resolution enhancement of HSI using high resolution MSI based on sparse coding and linear spectral unmixing (SCLSU) is introduced. In the proposed method (SCLSU), high spectral resolution features of HSI and high spatial resolution features of MSI are fused. In this case, the sparse representation of some high resolution MSI and linear spectral unmixing (LSU) model of HSI and MSI is simultaneously used in order to construct high resolution HSI (HRHSI). The fusion process of HSI and MSI is formulated as an ill-posed inverse problem. It is solved by the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) and an orthogonal matching pursuit (OMP) algorithm. Finally, the proposed algorithm is applied to the Hyperion and ALI datasets. Compared with the other state-of-the-art algorithms such as Coupled Nonnegative Matrix Factorization (CNMF) and local spectral unmixing, the SCLSU has significantly increased the spatial resolution and in addition the spectral content of HSI is well maintained.

Evapotranspiration (ET) estimates are particularly needed for monitoring the available water of arid lands. Remote sensing data offer the ideal spatial and temporal coverage needed by irrigation water management institutions to deal with increasing pressure on available water. Low spatial resolution (LR) products present strong advantages. They cover larger zones and are acquired more frequently than high spatial resolution (HR) products. Current sensors such as Moderate-Resolution Imaging Spectroradiometer (MODIS) offer a long record history. However, validation of ET products at LR remains a difficult task. In this context, the objective of this study is to evaluate scaling properties of ET fluxes obtained at high and low resolution by two commonly used Energy Balance models, the Surface Energy Balance System (SEBS) and the Two-Source Energy Balance model (TSEB). Both are forced by local meteorological observations and remote sensing data in Visible, Near Infra-Red and Thermal Infra-Red spectral domains. Remotely sensed data stem from Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) and MODIS sensors, respectively, resampled at 100 m and 1000 m resolutions. The study zone is a square area of 4 by 4 km2 located in a semi-arid irrigated agricultural zone in the northwest of Mexico. Wheat is the dominant crop, followed by maize and vegetables. The HR ASTER dataset includes seven dates between the 30 December 2007 and 13 May 2008 and the LR MODIS products were retrieved for the same overpasses. ET retrievals from HR ASTER products provided reference ET maps at LR once linearly aggregated at the km scale. The quality of this retrieval was assessed using eddy covariance data at seven locations within the 4 by 4 km2 square. To investigate the impact of input aggregation, we first compared to the reference dataset all fluxes obtained by running TSEB and SEBS models using ASTER reflectances and radiances previously aggregated at the km scale. Second, we compared to the same reference dataset all fluxes obtained with SEBS and TSEB models using MODIS data. LR fluxes obtained by both models driven by aggregated ASTER input data compared well with the reference simulations and illustrated the relatively good accuracy achieved using aggregated inputs (relative bias of about 3.5% for SEBS and decreased to less than 1% for TSEB). Results also showed that MODIS ET estimates compared well with the reference simulation (relative bias was down to about 2% for SEBS and 3% for TSEB). Discrepancies were mainly related to fraction cover mapping for TSEB and to surface roughness length mapping for SEBS. This was consistent with the sensitivity analysis of those parameters previously published. To improve accuracy from LR estimates obtained using the 1 km surface temperature product provided by MODIS, we tested three statistical and one deterministic aggregation rules for the most sensible input parameter, the surface roughness length. The harmonic and geometric averages appeared to be the most accurate.

Monitoring crop water content for corn and soybean fields through data fusion of MODIS and Landsat measurements in Iowa

Most modern satellites can provide two types of images: 1) panchromatic (PAN) image and 2) multispectral (MS) image. The former has high spatial resolution and low spectral resolution, while the latter has high spectral resolution and low spatial resolution. To obtain images with both high spectral and spatial resolution, pansharpening has emerged to fuse the spatial information of the PAN image and the spectral information of the MS image. However, most pansharpening methods fail to preserve spatial and spectral information simultaneously. In this article, we propose a framelet-based convolutional neural network (CNN) for pansharpening which makes it possible to pursue both high spectral and high spatial resolution. Our network consists of three subnetworks: 1) feature embedding net; 2) feature fusion net; and 3) framelet prediction net. Different from conventional CNN methods directly inferring high-resolution MS images, our approach learns to predict their framelet coefficients from available PAN and MS images. The introduction of multilevel feature aggregation and hybrid residual connection makes full use of spatial information of PAN image and spectral information of MS image. Quantitative and qualitative experiments at reduced- and full-resolution demonstrate that the proposed method achieves more appealing results than other state-of-the-art pansharpening methods. The source code and trained models are available at https://github.com/TingMAC/FrMLNet.

Hyperspectral remote sensing images (HSIs) are characterized by having a low spatial resolution and a high spectral resolution, whereas multispectral images (MSIs) are characterized by low spectral and high spatial resolutions. These complementary characteristics have stimulated active research in the inference of images with high spatial and spectral resolutions from HSI-MSI pairs. In this paper, we formulate this data fusion problem as the minimization of a convex objective function containing two data-fitting terms and an edge-preserving regularizer. The data-fitting terms are quadratic and account for blur, different spatial resolutions, and additive noise; the regularizer, a form of vector Total Variation, promotes aligned discontinuities across the reconstructed hyperspectral bands. The optimization described above is rather hard, owing to its non-diagonalizable linear operators, to the non-quadratic and non-smooth nature of the regularizer, and to the very large size of the image to be inferred. We tackle these difficulties by tailoring the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) — an instance of the Alternating Direction Method of Multipliers (ADMM) — to this optimization problem. By using a convenient variable splitting and by exploiting the fact that HSIs generally “live” in a low-dimensional subspace, we obtain an effective algorithm that yields state-of-the-art results, as illustrated by experiments.

Hyperspectral images (HSIs) usually have high spectral and low spatial resolution. Conversely, multispectral images (MSIs) usually have low spectral and high spatial resolution. The fusion of HSI and MSI aims to create spectral images with high spectral and spatial resolution. In this paper, we propose a fusion algorithm by combining linear spectral unmixing with the local low-rank property. By taking advantage of the local low-rank property, we first partition the corresponding spectral image into patches. For each patch pair, we cast the fusion problem as a coupled spectral unmixing problem that extracts the abundance and the endmembers of MSI and HSI, respectively. It then updates the abundance and the endmember through an alternating update algorithm. In fact, the convergence of the alternative update algorithm can be mathematically and empirically supported. We also propose a multiscale postprocessing procedure to combine fusion results obtained under different patch sizes. In experiments on three data sets, the proposed fusion algorithms outperformed state-of-the-art fusion algorithms in both spatial and spectral domains.

A robust merging method for Sentinel-2 Multispectral Imager (MSI) and Landsat-8 Operational Land Imager (OLI) data products over inland and coastal waters greatly improves our capability for monitoring water quality indicators (WQIs) by significantly increasing the temporal observation frequency with high spatial resolution. Such merged surface reflectance products (e.g. HLS L30 and HLS S30) are publicly available through Harmonized Landsat and Sentinel-2 (HLS) initiative for land applications. However, such a product for water reflectance, a principal quantity for deriving a series of WQIs, has not yet been available. Here, we propose a comprehensive spectral harmonization method for OLI and MSI imagery to improve the accuracy and consistency of water reflectance for retrieving chlorophyll-a (Chl-a) concentration and the light absorption coefficient of colored dissolved organic matter at 440nm (a<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cdom</sub>(440)) with reasonable uncertainties. To do so, we first evaluate the performance of atmospheric correction (AC) schemes including Acolite, Polymer, C2RCC, OC-SMART, and ICOR against the in situ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ρ</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</sub>. Our results indicate that compared to other AC schemes, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ρ</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</sub> products derived from OC-SMART processing are in good agreement with in situ data, with a median absolute percentage difference (MAPD) < 30% across all the visible bands for OLI and MSI. To acquire consistent <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ρ</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</sub> estimates from OLI and MSI, we apply a recently published machine learning (ML)-based bandpass adjustment (BA) model [1], trained on near-simultaneous OLI and MSI <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ρ</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w</sub> obtained through OC-SMART, to MSI images, with OLI as a reference. To better estimate Chl-a and a<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cdom</sub>(440), the performance of retrieval models, including the recently developed ocean color net (OCN) [2], the Gaussian Process Regression (GPR) and the band ratio based algorithms, are tuned and evaluated against in situ data. Chl-a and acdom(440) estimates show that OCN is the top performer with a mean absolute error of 38% and 20%, respectively. The tuned OCN model is then applied to OLI and MSI images. Our results demonstrate that the proposed BA method properly aligns the OLI and MSI water reflectance spectra, resulting in improved Chl-a and a<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cdom</sub>(440) estimates in a variety of optically complex waters.

In this paper, we present a novel spatial and spectral fusion model (SASFM) that uses sparse matrix factorization to fuse remote sensing imagery with different spatial and spectral properties. By combining the spectral information from sensors with low spatial resolution (LSaR) but high spectral resolution (HSeR) (hereafter called HSeR sensors), with the spatial information from sensors with high spatial resolution (HSaR) but low spectral resolution (LSeR) (hereafter called HSaR sensors), the SASFM can generate synthetic remote sensing data with both HSaR and HSeR. Given two reasonable assumptions, the proposed model can integrate the LSaR and HSaR data via two stages. In the first stage, the model learns from the LSaR data a spectral dictionary containing pure signatures, and in the second stage, the desired HSaR and HSeR data are predicted using the learned spectral dictionary and the known HSaR data. The SASFM is tested with both simulated data and actual Landsat 7 Enhanced Thematic Mapper Plus (ETM+) and Terra Moderate Resolution Imaging Spectroradiometer (MODIS) acquisitions, and it is also compared to other representative algorithms. The experimental results demonstrate that the SASFM outperforms other algorithms in generating fused imagery with both the well-preserved spectral properties of MODIS and the spatial properties of ETM+. Generated imagery with simultaneous HSaR and HSeR opens new avenues for applications of MODIS and ETM+.

Particle image velocimetry (PIV) techniques have a limited field of view of the flow field and can only capture high-resolution flow fields in localized areas. To obtain a larger measurement range, multiple cameras must be used to capture the flow field simultaneously and then stitch the parts together. However, this method can be expensive. We propose the local-global fusion convolutional neural network (LGF-CNN) for reconstructing large-field flow fields with high spatial resolution based only on two flow data types: local small-field high spatial resolution wake velocity fields and global large-field low spatial resolution velocity fields. The core of the model consists of convolutional neural network (CNN) architecture to learn the mapping relationship between the small field of view with high spatial resolution and the large field of view with low spatial resolution. Using the effectively trained LGF-CNN model, we demonstrate its ability to reconstruct high-resolution velocity fields around the circular cylinder. The LGF-CNN is rigorously validated on a number of representative datasets, including simulated data for Reynolds numbers of 200 and 500, as well as experimental data for a Reynolds number of 3.3 × 104 with the steady jet at the rear stagnation point of the cylinder. The results demonstrate the ability of LGF-CNN to generate accurate velocity fields with high spatial resolution, including reliable detection of high-frequency components. The proposed method could reduce the number of cameras required for large-field, high spatial resolution PIV measurements, thereby reducing experimental costs.

Land surface temperature (LST) images retrieved from the thermal infrared (TIR) band data of Moderate Resolution Imaging Spectroradiometer (MODIS) have much lower spatial resolution than the MODIS visible and near-infrared (VNIR) band data. The coarse pixel scale of MODIS LST images (1000 m under nadir) have limited their capability in applying to many studies required high spatial resolution in comparison of the MODIS VNIR band data with pixel scale of 250–500 m. In this paper we intend to develop an efficient approach for pixel decomposition to increase the spatial resolution of MODIS LST image using the VNIR band data as assistance. The unique feature of this approach is to maintain the thermal radiance of parent pixels in the MODIS LST image unchanged after they are decomposed into the sub-pixels in the resulted image. There are two important steps in the decomposition: initial temperature estimation and final temperature determination. Therefore the approach can be termed double-step pixel decomposition (DSPD). Both steps involve a series of procedures to achieve the final result of decomposed LST image, including classification of the surface patterns, establishment of LST change with normalized difference of vegetation index (NDVI) and building index (NDBI), reversion of LST into thermal radiance through Planck equation, and computation of weights for the sub-pixels of the resulted image. Since the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) with much higher spatial resolution than MODIS data was on-board the same platform (Terra) as MODIS for Earth observation, an experiment had been done in the study to validate the accuracy and efficiency of our approach for pixel decomposition. The ASTER LST image was used as the reference to compare with the decomposed LST image. The result showed that the spatial distribution of the decomposed LST image was very similar to that of the ASTER LST image with a root mean square error (RMSE) of 2.7 K for entire image. Comparison with the evaluation DisTrad (E-DisTrad) and re-sampling methods for pixel decomposition also indicate that our DSPD has the lowest RMSE in all cases, including urban region, water bodies, and natural terrain. The obvious increase in spatial resolution remarkably uplifts the capability of the coarse MODIS LST images in highlighting the details of LST variation. Therefore it can be concluded that, in spite of complicated procedures, the proposed DSPD approach provides an alternative to improve the spatial resolution of MODIS LST image hence expand its applicability to the real world.

Satellite data holds considerable potential as a source of information on rice crop growth which can be used to inform agronomy. However, given the typical field sizes in many rice-growing countries such as China, data from coarse spatial resolution satellite systems such as the Moderate Resolution Imaging Spectroradiometer (MODIS) are inadequate for resolving crop growth variability at the field scale. Nevertheless, systems such as MODIS do provide images with sufficient frequency to be able to capture the detail of rice crop growth trajectories throughout a growing season. In order to generate high spatial and temporal resolution data suitable for mapping rice crop phenology, this study fused MODIS data with lower frequency, higher spatial resolution Landsat data. An overall workflow was developed which began with image preprocessing, calculation of multi-temporal normalized difference vegetation index (NDVI) images, and spatiotemporal fusion of data from the two sensors. The Spatial and Temporal Adaptive Reflectance Fusion Model was used to effectively downscale the MODIS data to deliver a time-series of 30 m spatial resolution NDVI data at 8-day intervals throughout the rice-growing season. Zonal statistical analysis was used to extract NDVI time-series for individual fields and signal filtering was applied to the time-series to generate rice phenology curves. The downscaled MODIS NDVI products were able to characterize the development of paddy rice at fine spatial and temporal resolutions, across wide spatial extents over multiple growing seasons. These data permitted the extraction of key crop seasonality parameters that quantified inter-annual growth variability for a whole agricultural region and enabled mapping of the variability in crop performance between and within fields. Hence, this approach can provide rice crop growth data that is suitable for informing agronomic policy and practice across a wide range of scales.

In recent years, ultra-high field functional MRI has allowed researchers to study cortical activity at high spatiotemporal resolution. Advancements in technology have made it possible to perform fMRI of cortical laminae, which is crucial for understanding and mapping of local circuits and overall brain function. Unlike invasive electrophysiology, fMRI provides a non-invasive approach to studying human and animal brain function. However, achieving high spatial resolution has often meant sacrificing temporal resolution. In contrast, line-scanning fMRI maintains both high spatial and temporal resolution, and has been successfully applied to animals to detect laminar differences of the hemodynamic response. Although this method has been extended to human brain imaging in initial studies, staying within SAR safety limits while maintaining a well-defined saturation profile at a short TR is a major challenge. We present a method for gradient-echo-based human line-scanning that uses four saturation regions to achieve a line with narrow FWHM (3.9 mm) at high spatiotemporal resolution (voxel size 0.39 x 3.0 x 3.0 mm3, TR = 250 ms). We demonstrate its use for laminar fMRI by measuring laminar time courses in the hand knob of the primary human motor cortex during a finger-tapping task. Our findings indicate differences in the onset and temporal characteristics of the hemodynamic response across cortical layers. Deeper layers exhibited distinct temporal dynamics compared with the gray matter near the cortical surface. Specifically, the BOLD response reached 95% of the maximum amplitude earlier than the superficial layers, and demonstrated a faster return to baseline after stimulus offset. We demonstrate that line-scanning fMRI offers a valuable tool for investigating recordings at a very high temporal and spatial resolution and could help advance our understanding of the mechanistic nature of the BOLD response.

While data like HJ-1 CCD images have advantageous spatial characteristics for describing crop properties, the temporal resolution of the data is rather low, which can be easily made worse by cloud contamination. In contrast, although Moderate Resolution Imaging Spectroradiometer (MODIS) can only achieve a spatial resolution of 250 m in its normalised difference vegetation index (NDVI) product, it has a high temporal resolution, covering the Earth up to multiple times per day. To combine the high spatial resolution and high temporal resolution of different data sources, a new method (Spatial and Temporal Adaptive Vegetation index Fusion Model [STAVFM]) for blending NDVI of different spatial and temporal resolutions to produce high spatial–temporal resolution NDVI datasets was developed based on Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM). STAVFM defines a time window according to the temporal variation of crops, takes crop phenophase into consideration and improves the temporal weighting algorithm. The result showed that the new method can combine the temporal information of MODIS NDVI and spatial difference information of HJ-1 CCD NDVI to generate an NDVI dataset with both high spatial and high temporal resolution. An application of the generated NDVI dataset in crop biomass estimation was provided. An average absolute error of 17.2% was achieved. The estimated winter wheat biomass correlated well with observed biomass (R 2 of 0.876). We conclude that the new dataset will improve the application of crop biomass estimation by describing the crop biomass accumulation in detail. There is potential to apply the approach in many other studies, including crop production estimation, crop growth monitoring and agricultural ecosystem carbon cycle research, which will contribute to the implementation of Digital Earth by describing land surface processes in detail.