A seamless global daily 5 km soil moisture product from 1982 to 2021 using AVHRR satellite data and an attention-based deep learning model

The Advanced Scatterometer (ASCAT), Soil Moisture Active Passive (SMAP), and European Space Agency-Climate Change Initiative (ESA–CCI) soil moisture (SM) products are widely used in agricultural drought monitoring, water resource management, and climate analysis applications. However, the performance of these SM products varies significantly across regions and environmental conditions, due to in sensor characteristics, retrieval algorithms, and the lack of localized calibration. This study proposes a multi-source long short-term memory (MLSTM) for improving ASCAT, ESA–CCI, and SMAP SM products by combining in-situ SM measurements and four key auxiliary variables: precipitation (PRE), land surface temperature (LST), fractional vegetation cover (FVC), and evapotranspiration (ET). First, the in-situ measured data from four in-situ observation networks were corrected using the LSTM method to match the grid sizes of ASCAT (0.1°), ESA–CCI (0.25°), and SMAP (0.1°) SM products. The RPE, LST, FVC, and ET were used as inputs to the LSTM to obtain loss data against in-situ SM measurements. Second, the ASCAT, ESA–CCI, and SMAP SM datasets were used as inputs to the LSTM to generate loss data, which were subsequently corrected using LSTM-derived loss data based on in-situ SM measurements. When the mean squared error (MSE) loss values were minimized, the improvement for ASCAT, ESA–CCI, and SMAP products was considered the best. Finally, the improved ASCAT, ESA–CCI, and SMAP were produced and evaluated by the correlation coefficient (R), root mean square error (RMSE), and standard deviation (SD). The results showed that the RMSE values of the improved ASCAT, ESA–CCI, and SMAP products against the corrected in-situ SM data in the OZNET network were lower, i.e., 0.014 cm3/cm3, 0.019 cm3/cm3, and 0.034 cm3/cm3, respectively. Compared with the ESA–CCI and SMAP products, the ASCAT product was greatly improved, e.g., in the SNOTEL network, the Root Mean-Square Deviation (RMSD) values of 0.1049 cm3/cm3 (ASCAT) and 0.0662 cm3/cm3 (improved ASCAT). Overall, the MLSTM-based algorithm has the potential to improve the global satellite SM product.

The Essential Climate Variable (ECV) soil moisture (SM) datasets, originated from the European Space Agency, have revealed great potential for application in hydrology and agriculture. Hence, it is essential to continuously enhance the data quality and spatial completeness to satisfy the increasing scientific research requirements. In this study, we explore the potential possibility of Soil Moisture Active Passive (SMAP) datasets in filling the gaps of ECV SM. The comprehensive assessment results show that: (1) The data missing percent of gap-filled ECV decreases 20% on average, which can be one step closer to generate a seamlessly covered global land surface SM product with favorable quality. (2) Compared to the original ECV, the gap-filled ECV products express similar good response to the in-situ measurements, suggesting that the SMAP SM products could be taken to efficiently fill the gaps and consistently maintain favorable accuracy at the same time. (3) Compared to the in-situ measurements, the original ECV SM products demonstrate extremely high probability density peak percentages. Fortunately, this eminent high value could be effectively rectified through gap-filling progress using SMAP. Overall, this study conducts objective and detailed evaluation on the performance of applying SMAP to fill the gaps of ECV, and is expected to act as a valuable reference in ECV SM gap-filling method.

Abstract. Soil moisture (SM) datasets are critical to understanding the global water, energy, and biogeochemical cycles and benefit extensive societal applications. However, individual sources of SM data (e.g., in situ and satellite observations, reanalysis, offline land surface model simulations, Earth system model – ESM – simulations) have source-specific limitations and biases related to the spatiotemporal continuity, resolutions, and modeling and retrieval assumptions. Here, we developed seven global, gap-free, long-term (1970–2016), multilayer (0–10, 10–30, 30–50, and 50–100 cm) SM products at monthly 0.5∘ resolution (available at https://doi.org/10.6084/m9.figshare.13661312.v1; Wang and Mao, 2021) by synthesizing a wide range of SM datasets using three statistical methods (unweighted averaging, optimal linear combination, and emergent constraint). The merged products outperformed their source datasets when evaluated with in situ observations (mean bias from −0.044 to 0.033 m3 m−3, root mean square errors from 0.076 to 0.104 m3 m−3, Pearson correlations from 0.35 to 0.67) and multiple gridded datasets that did not enter merging because of insufficient spatial, temporal, or soil layer coverage. Three of the new SM products, which were produced by applying any of the three merging methods to the source datasets excluding the ESMs, had lower bias and root mean square errors and higher correlations than the ESM-dependent merged products. The ESM-independent products also showed a better ability to capture historical large-scale drought events than the ESM-dependent products. The merged products generally showed reasonable temporal homogeneity and physically plausible global sensitivities to observed meteorological factors, except that the ESM-dependent products underestimated the low-frequency temporal variability in SM and overestimated the high-frequency variability for the 50–100 cm depth. Based on these evaluation results, the three ESM-independent products were finally recommended for future applications because of their better performances than the ESM-dependent ones. Despite uncertainties in the raw SM datasets and fusion methods, these hybrid products create added value over existing SM datasets because of the performance improvement and harmonized spatial, temporal, and vertical coverages, and they provide a new foundation for scientific investigation and resource management.

The study aims to develop an Attention-Based LSTM (Long Short-Term Memory) model with Conditional Variation Autoencoders (CVAE), and LIME (Local Interpretable Model-Agnostic Explanations), to accurately predict students’ performance. The study employed data Processing and Augmentation using Conditional Variational Autoencoders (CVAE), an Attention-Based Long Short-Term Memory (LSTM) framework, and a final ensemble approach for refined predictions classifying students into three categories. The study utilized a primary dataset consisting of 500 students’ records, the study establishes a performance benchmark. The study revealed that students who frequently raised their hands during lessons were identified by the model as more attentive and participatory, traits that strongly correlated with higher academic performance. It was evident that the developed LSTM with CVAE, and Ensemble model accurately predicts students’ performance. The study concluded the Attention-Based LSTM outperformed the Deep Neural Network, K-Nearest Neighbor, Decision Tree, Support Vector Machine, Naïve Bayes, Random Forest, and Artificial Neural Network which uses the same dataset in predicting student performance. The Attention-Based LSTM consistently outperforms these models, showcasing superior metrics in accuracy, precision, recall, and loss with an accuracy of 89.6%. The study recommended that the Attention-Based LSTM model’s performance highlights how crucial it is for the educational industry to make use of cutting-edge analytical techniques. Also, when developing curricula, policymakers need to take the research on student involvement into account, including components that encourage active participation can result in better academic performance.

A preliminary study toward consistent soil moisture products from the Advanced Microwave Scanning Radiometer 2 (AMSR2) is presented. Its predecessor, the Advanced Microwave Scanning Radiometer for Earth Observing System (AMSR-E), has provided Earth scientists with a consistent and continuous global soil moisture dataset. A major challenge remains to achieve synergy between these soil moisture datasets, which is hampered by the lack of an overlapping observation period of the sensors. Here, observations of the multifrequency microwave radiometer on board the Tropical Rainfall Measuring Mission (TRMM) satellite were used to improve consistency between AMSR-E and AMSR2. Several scenarios to achieve synergy between the AMSR-E and AMSR2 soil moisture products were evaluated. The novel soil moisture retrievals from C-band observations, a frequency band that is lacking on board the TRMM satellite, are also presented. A global comparison of soil moisture retrievals against ERA-Interim soil moisture demonstrates the need for an intercalibration procedure. Several different scenarios based on filtering were tested, and the impact on the soil moisture retrievals was evaluated against two independent reference soil moisture datasets (reanalysis and in situ soil moisture) that cover both individual observation periods of the AMSR-E and AMSR2 sensors. Results show a high degree of consistency between both satellite products and two independent reference products for the soil moisture products retrieved from X-band observations. Care should be taken in the interpretation of the presented soil moisture products, and future research is needed to further align the AMSR2 and AMSR-E sensor calibrations.

Abstract. An improved fine-scale soil moisture (SM) dataset at 1 km grid spacing, covering much of the eastern continental US, was generated by assimilating 9 km Soil Moisture Active Passive (SMAP) SM data into the v4.0.1 Noah-MP land surface model. With 12 ensemble members, the assimilation was carried out using the ensemble Kalman filter algorithm within NASA's Land Information System. The SM analysis for 2016 was fully validated against in situ observations from four different networks and compared with four other existing datasets. Results indicate that this SM analysis surpasses other datasets in top-layer SM distribution, including a machine-learning-based product, despite all SM estimates being less heterogeneous than observed. The analysis of anomalous errors suggests that large similarity in intrinsic errors is likely due to overlapping data sources among the selected SM datasets. More detailed evaluations were performed over two geographic areas. The observations collected by the Atmospheric Radiation Measurement facility in Oklahoma suggest that soil temperature and surface heat fluxes are concurrently simulated with good accuracy. Investigation into the 2016 southeastern US drought response further indicates drier conditions and higher evapotranspiration estimates compared to GLEAMv4.1. Notably, large errors are associated with grids having clay soil textures, underscoring the need for refined model treatments for specific soil types to further improve SM estimates. The dataset is publicly available on Zenodo at https://doi.org/10.5281/zenodo.14370563 (Tai et al., 2024).

The European Space Agency (ESA) launched the SMOS (Soil Moisture Ocean Salinity) mission in 2009; currently, multiple global soil moisture (SM) products are based on the measurements of its L-band (1.4 GHz) radiometer. We compared four SMOS products with each other: Level 2, Level 3, IC (INRA-CESBIO), and Near Real Time products. The comparisons focused on core validation sites (CVS), whose spatial representativeness errors allow the estimation of the SM product performance for bias-insensitive metrics (unbiased root mean square error (ubRMSE) and correlation (R), and anomaly R) with negligible uncertainty and for bias-sensitive metrics (mean difference (MD) and root mean square difference or RMSD) with acceptable uncertainty. When the products were compared with CVS independently, the results showed that the ubRMSE, R, and anomaly R of the IC product were better than those of the other products, while the MD was larger. However, the differences between the performances were smaller when the products were assessed using only the data points when each product had a valid retrieval. This indicates that the algorithms have similar performance and that data screening and quality flagging of the retrievals markedly affects the performance. The NASA Soil Moisture Active Passive (SMAP) mission produces a similar SM product as SMOS using an L-band radiometer. The closeness of the ubRMSE, R, and anomaly R performance of the IC product and the SMAP product (0.039 m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> /m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> vs. 0.041 m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> /m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> , 0.80 vs. 0.81, and 0.75 vs. 0.75) demonstrate that the SMOS and SMAP radiometers can achieve similar SM sensitivity.

High-quality and long time-series soil moisture (SM) data are increasingly required for the Qinghai-Tibet Plateau (QTP) to more accurately and effectively assess climate change. In this study, to evaluate the accuracy and effectiveness of SM data, five passive microwave remotely sensed SM products are collected over the QTP, including those from the soil moisture active passive (SMAP), soil moisture and ocean salinity INRA-CESBIO (SMOS-IC), Fengyun-3B microwave radiation image (FY3B), and two SM products derived from the advanced microwave scanning radiometer 2 (AMSR2). The two AMSR2 products are generated by the land parameter retrieval model (LPRM) and the Japan Aerospace Exploration Agency (JAXA) algorithm, respectively. The SM products are evaluated through a two-stage data comparison method. The first stage is direct validation at the grid scale. Five SM products are compared with corresponding in situ measurements at five in situ networks, including Heihe, Naqu, Pali, Maqu, and Ngari. Another stage is indirect validation at the regional scale, where the uncertainties of the data are quantified by using a three-cornered hat (TCH) method. The results at the regional scale indicate that soil moisture is underestimated by JAXA and overestimated by LPRM, some noise is contained in temporal variations in SMOS-IC, and FY3B has relatively low absolute accuracy. The uncertainty of SMAP is the lowest among the five products over the entire QTP. In the SM map composed by five SM products with the lowest pixel-level uncertainty, 66.64% of the area is covered by SMAP (JAXA: 19.39%, FY3B: 10.83%, LPRM: 2.11%, and SMOS-IC: 1.03%). This study reveals some of the reasons for the different performances of these five SM products, mainly from the perspective of the parameterization schemes of their corresponding retrieval algorithms. Specifically, the parameterization configurations and corresponding input datasets, including the land-surface temperature, the vegetation optical depth, and the soil dielectric mixing model are analyzed and discussed. This study provides quantitative evidence to better understand the uncertainties of SM products and explain errors that originate from the retrieval algorithms.

&amp;lt;p&amp;gt;Merging data from different instruments is required to construct long time data records of soil moisture (SM). This is the goal of projects such as the ESA Climate Change Initiative (CCI) for SM (Gruber et al., 2019), which uses both active and passive microwave sensors. Currently, the GLDAS v2.1 model is used as reference to re-scale active and passive time series by matching their Cumulative Density Function (CDF) to that of the model. Removing the dependency on models is important, in particular for data assimilation applications into hydrological or climate models, and it has been proposed (Van der Schalie et al., 2018) to use L-band data from one of the two instruments specifically designed to measure SM, ESA Soil Moisture and Ocean Salinity (SMOS) and NASA Soil Moisture Active Passive (SMAP) satellites, as reference to re-scale other time series.&amp;lt;br&amp;gt;To investigate this approach, AMSR-2 SM time series obtained from C1-, C2- and X-band observations using LPRM (Land Parameter Retrieval Model) were re-scaled by CDF-matching (Brocca et al., 2011) using different SMAP and SMOS official (SMAP L2 V005, SMOS L3 V300, SMOS NRT V100&amp;amp;V200) and research (SMOS IC V103) SM products as well as the SMAP and SMOS LPRM v6 SM data used by the ESA CCI. The time series re-scaled using L-band remote sensing data were compared to those re-scaled using GLDAS and were evaluated against in situ measurements at several hundred sites retrieved from the International Soil Moisture Network (Dorigo et al., 2011). The results were analyzed as a function of the land cover class and the Koppen-Geiger climate classification.&amp;lt;br&amp;gt;Overall, AMSR-2 time series re-scaled using SMAP L2, SMAP LPRM and SMOS IC data sets as reference gave the best correlations with respect to in situ measurements, similar to those obtained by the time series re-scaled using GLDAS and slightly better than those of the original AMSR-2 time series. These results imply that different SMAP and SMOS products could actually be used to replace GLDAS as reference for the re-scaling of other sensors time series within the ESA CCI. However, one must bear in mind that this study is limited to the re-scaling of AMSR-2 data at a few hundred sites.&amp;lt;br&amp;gt;For a more detailed assessment of the L-band data set to be used for a global re-scaling, it is necessary to investigate other effects such as the spatial coverage or the time series length. SMAP spatial coverage is better than that of SMOS in regions affected by radio frequency interference. In contrast, the length of SMAP time series can be too short to capture the long term SM variability for climate applications in some regions. The CDF of SMOS time series computed from the date of SMAP launch is significantly different to those of the full length SMOS time series in some regions of the Globe. Possible ways of using a coherent SMAP/SMOS L-band data set will be discussed.&amp;lt;/p&amp;gt;

Soil moisture (SM) is a key variable in hydrological processes, bio-ecological processes, and biogeochemical processes. Long-term observations of soil moisture over large areas are critical to research on flooding and drought monitoring, water resource management, and crop yield forecasts. In this paper, Fengyun (FY3B and FY3C) SM products, Japan Aerospace Exploration Agency (JAXA) SM products from the Advanced Microwave Scanning Radiometer 2 (AMSR2), the Land Parameter Retrieval Model (LPRM) AMSR2 L3 SM products, the Version 2 (v2) global land parameter data record (LPDR) of SM products, Soil Moisture Ocean Salinity (SMOS) Centre Aval de Traitement des Donnees SMOS (CATDS) L3 SM products, the Soil Moisture Active Passive (SMAP) passive L3 SM products and the European Space Agency(ESA) Climate Change Initiative (CCI) SM products were evaluated using the ground-based observations in the Little Washita, Fort Cobb and Yanco networks. Long-time series comparison between measured and satellite products was conducted to evaluate the overall performance of FY3 series satellites SM products. Bias (mean bias), R (correlation coefficient), RMSE (root mean square error) and ubRMSE (unbiased root mean square error) were calculated to explore the agreement between satellite products and in-situ measurements. Taylor diagrams were used to compare the performance of various satellite products. The result showed that (1) FY3B, FY3C and LPRM AMSR2 ascending and descending products had an obvious overestimate with in-situ soil moisture in three networks. (2) JAXA AMSR2 ascending and descending products had considerable underestimation in three networks. (3) The validation result of SMOS ascending and descending products over the three networks was satisfactory, with a rather high correlation than other X-band products. (4) The validation result of SMAP ascending and descending products outperformed the other products over the Little Washita and Fort Cobb networks. (5) The ESA CCI product had the lowest values of RMSE and ubRMSE than the other products in the Yanco network, revealing the effectiveness of merging active and passive soil moisture products. (6)The LPDR descending SM products had better performance than the LPDR ascending SM products in the LW and FC networks.

Soil moisture is a significant environmental factor that influences both the water and energy balance at the land-atmosphere interface. Therefore, proper assessment of the spatial and temporal distribution of soil moisture is crucial for many hydrological applications such as weather forecasting, agricultural water resource management and drought monitoring. This study involves the assimilation of Soil Moisture Active Passive (SMAP) soil moisture dataset within a land surface model and the evaluation of its performance in precise estimation of soil moisture by comparing the statistics with respect to standard European Space Agency&amp;#8217;s Climate Change Initiative (ESA-CCI) soil moisture dataset. The Ensemble Kalman Filter technique has been used for assimilating SMAP soil moisture data using Noah-MP land surface model within NASA Land Information System (LIS) framework. The data assimilation (DA) framework includes Cumulative Distribution Function (CDF) matching for bias correction and twenty ensembles per tile. Meteorological forcings for the simulations have been taken from MERRA2 and IMD. Improvement or degradation due to DA has been analyzed in terms of the difference in anomaly correlation between open loop (OL) and DA soil moisture outputs with respect to the ESA-CCI soil moisture dataset over the entire Indian domain. The DA result shows improvement over larger areas in the case of MERRA2 forced simulations than IMD+MERRA2. The seasonal impact of DA in terms of the differences in DA and OL simulated soil moisture shows less variability in summer than winter. The results are validated with in-situ soil moisture datasets. Overall, the study shows that data assimilation is giving better results than open loop LSM simulation, which can be used for improved estimation of other water balance components.

In this study, we used a multilinear regression approach to retrieve surface soil moisture from NASA's Soil Moisture Active Passive (SMAP) satellite data to create a global dataset of surface soil moisture which is consistent with ESA's Soil Moisture and Ocean Salinity (SMOS) satellite retrieved surface soil moisture. This was achieved by calibrating coefficients of the regression model using SMOS soil moisture and horizontal and vertical brightness temperatures (TB), over the 2013 — 2014 period. Next, this model was applied to recent SMAP TB data from 31/03/2015–08/09/2015. The retrieved surface soil moisture from SMAP (referred here to as SMAP-reg) was compared to the operational SMAP L3 surface soil moisture retrieved using the single channel algorithm. Both exhibit comparable temporal dynamics with a good agreement of correlation (correlation coefficient R mostly > 0.8) between the SMAP-reg and the operational SMAP L3 surface soil moisture products.

Accurate soil moisture (SM) estimates with high spatial resolution are highly desirable for agricultural, hydrological, and environmental applications. This study developed a two-step reconstruction approach to obtain a high-quality and high-spatial-resolution (0.05°) SM dataset from microwave and model-based SM products, combining Bayesian three-cornered hat (BTCH) merging and machine/deep learning downscaling algorithms. Firstly, a three-cornered hat (TCH) method was used to analyze the uncertainty of seven SM products on four main land cover types in the Pearl River Basin (PRB). On this basis, the SM products with low uncertainty were merged using the BTCH method. Secondly, two machine/deep learning algorithms (random forest, RF, and long short-term memory, LSTM) were applied to downscale the merged SM data from 0.25° to 0.05° based on the relationship between SM and auxiliary variables. The overall performance of RF and LSTM downscaling models with/without antecedent precipitation were compared. The merged and downscaled SM results were validated against in situ observations and the China Meteorological Administration (CMA) Land Data Assimilation System (CLDAS) SM data. The results indicated the following: (1) The BTCH-based SM estimate outperformed the parent products and the AVE-based SM estimate (the arithmetic average), indicating that BTCH is a fusion approach that can effectively reduce data uncertainties and optimize weights. (2) The optimal time scale for the cumulative effect of precipitation on SM was 35 days during 2015–2020 in the PRB. SM estimations using RF and LSTM downscaling algorithms both had substantial improvement by considering the antecedent precipitation variable, both at the 0.25° and 0.05° spatial scales. Feature importance assessment also revealed the most important role of antecedent precipitation (30.01%). Moreover, the LSTM model with antecedent precipitation performed slightly better than the RF model with antecedent precipitation. (3) The downscaled SM results all mitigated the overestimation inherent in the original SM data, though they were inevitably limited by the performance of the original SM data and difficult to surpass. The developed two-step reconstruction approach was effective in generating an accurate SM dataset at a finer spatial scale for wide regional applications.

Soil moisture is a crucial component of the water cycle and plays an important role in regional weather and climate. However, owing to the lack of In Situ observations, an accurate understanding of the spatiotemporal variations of soil moisture (SM) on the Tibetan Plateau (TP) is still lacking. In this study, we used three gridded SM products to characterize the spatiotemporal features of SM on the TP during the warm season (May to August). We analyzed the fifth-generation European Centre for Medium-Range Weather Forecasts atmospheric reanalysis (ERA5), Global Land Data Assimilation System (GLDAS), and Soil Moisture Active Passive (SMAP) datasets and used station observation data and triple collocation to quantify product accuracy and consistency. Results of the evaluation based on observation data show that both ERA5 and GLDAS overestimate SM, while the accuracy of SMAP is high. In terms of capturing the temporal variations of SM measured at stations, the performance of ERA5 and that of SMAP are superior to that of GLDAS. According to the evaluation based on triple collocation, SMAP exhibits the smallest random error over the TP and the highest temporal correlation with the unknown true SM in eastern TP. For SMAP, SM variability is the largest in the southern TP. For ERA5 and GLDAS, variability in the western TP is substantially larger than that for SMAP. Low-frequency (30&amp;#8211;90 days) variations are the largest contributor to TP SM intraseasonal variability. Relative to SMAP, the contribution of high-frequency variations is low in ERA5 and GLDAS. Land-atmosphere coupling is stronger (weaker) in the western (southeastern) TP, which is relatively dry (wet). Our evaluation of SM product performance over the TP may facilitate the use of these products for disaster monitoring and climate and hydrological studies.

Global-scale evaluation of SMAP, SMOS and ASCAT soil moisture products using triple collocation