Evaluating Calibration and Spectral Variable Selection Methods for Predicting Three Soil Nutrients Using Vis-NIR Spectroscopy

The soil organic matter (SOM) content is a key factor affecting the function and health of soil ecosystems. For measurements of land reclamation and soil fertility, SOM monitoring using visible and near-infrared spectroscopy (Vis-NIR) is one approach to quantifying soil quality, and Vis-NIR is important for monitoring the SOM content in a broad and nondestructive manner. To investigate the influence of environmental factors and Vis-NIR spectroscopy in estimating SOM, 249 soil samples were collected from the Werigan–Kuqa oasis in Xinjiang, China, and their spectral reflectance, SOM content and soil salinity were measured. To classify and improve the prediction accuracy, we also take into account the soil salinity content as a variable indicator. Relevant environmental variables were extracted using remote sensing datasets (land-use/land-cover (LULC), digital elevation model (DEM), World Reference Base for Soil Resources (WRB), and soil texture). On the basis of Savitzky–Golay (S-G) smoothing and first derivative (FD) preprocessing of the original spectrum, three clusters were obtained by K-means clustering through the use of Vis-NIR and used as spectral classification variables. Using Vis-NIR as Model 1, Vis-NIR combined with spectral classification as Model 2, environmental variables as Model 3, and the combination of all the above variables (Vis-NIR, spectral classification, environmental variables, and soil salinity) as Model 4, a SOM content estimation model was constructed using partial least squares regression (PLSR). Using the 249 soil samples, the modeling set contained 166 samples and the validation set contained 83 samples. The results showed that Model 2 (validation r2 = 0.78) was better than Model 1 (validation r2 = 0.76). The prediction accuracy for Model 4 (validation r2 = 0.85) was better than Model 2 (validation r2 = 0.78). Among these, Model 3 was the worst (validation r2 = 0.39). Therefore, the combination of environmental variables with Vis-NIR spectroscopy to estimate SOM content is an important method and has important implications for improving the accuracy of SOM predictions in arid regions.

Sugar content in fruit is one of the most crucial internal quality factors and provides valuable information for predicting maturity and making commercial decisions. The distribution of sugar content in natural fruit is uneven. Conventional spectroscopy methods are not able to effectively solve this problem because they acquire spectral data from a single point or multiple points of the fruit. Therefore, this study explores the potential application of hyperspectral imaging in the wavelength range of 400–1000 nm for fast nondestructive prediction and visualization of sugar content in Dangshan pear. Hyperspectral imagery data and sugar content of pear samples were acquired in the laboratory. A mean normalization step was used to reduce the effect of sample curvature on spectral profiles. Spectra from whole fruit were extracted to build spectrum datasets. Different variable-selection methods, including Monte Carlo uninformative variable elimination (MC-UVE), the successive projection algorithm (SPA), competitive adaptive reweighted sampling (CARS), the genetic algorithm (GA), CARS-SPA, GA-SPA, and various other modeling methods such as linear partial least squares (PLS), nonlinear least squares support vector machine (LS-SVM), and back propagation artificial neural network (BP-ANN), were compared, and the results show that the linear CARS-PLS (correlation coefficient (rpre) = 0.8971 and prediction root mean square error (RMSEP) = 0.3937%) and GA-SPA-PLS (rpre = 0.8969 and RMSEP = 0.3482%) models are the optimal models for predicting sugar content in Dangshan pear. Compared with the GA-SPA-PLS model, CARS-PLS is more stable, whereas the GA-SPA-PLS model might be faster for performing a prediction task. Finally, the sugar content of pears given by hyperspectral imagery data is predicted and visualized by using the developed model. This study shows that the combination of hyperspectral imaging with lighting correction, CARS, GA-SPA variable selection methods, and PLS modeling has a great potential for nondestructive quantitative measurement and visualization of sugar content in Dangshan pear.

The influence of vegetation cover on the spectroscopic estimation of soil properties

Soil organic matter (SOM) content is an important index to measure the level of soil function and soil quality. However, conventional studies on estimation of SOM content concerned about the classic integer derivative of spectral data, while the fractional derivative information was ignored. In this research, a total of 103 soil samples were collected in the Ebinur Lake basin, Xinjiang Uighur Autonomous Region, China. After measuring the Vis-NIR (visible and near-infrared) spectroscopy and SOM content indoor, the raw reflectance and absorbance were treated by fractional derivative from 0 to 2nd order (order interval 0.2). Partial least squares regression (PLSR) was applied for model calibration, and five commonly used precision indices were used to assess the performance of these 22 models. The results showed that with the rise of order, these parameters showed the increasing or decreasing trends with vibration and reached the optimal values at the fractional order. A most robust model was calibrated based on 1.8 order derivative of R, with the lowest RMSEC (3.35 g kg−1) and RMSEP (2.70 g kg−1) and highest Rc2 (0.92), Rp2 (0.91), and RPD (3.42 > 3.0). This model had excellent predictive performance of estimating SOM content in the study area.

As a laboratory proximal sensing technique, the capability of visible and near-infrared (Vis-NIR) diffused reflectance spectroscopy with partial least squares (PLS) regression to determine soil properties has previously been demonstrated. However, the evaluation of the soil phosphorus (P) content—a major nutrient constraint for crop production in the tropics—is still a challenging task. PLS regression with waveband selection can improve the predictive ability of a calibration model, and a genetic algorithm (GA) has been widely applied as a suitable method for selecting wavebands in laboratory calibrations. To develop a laboratory-based proximal sensing method, this study investigated the potential to use GA-PLS regression analyses to estimate oxalate-extractable P in upland and lowland soils from laboratory Vis-NIR reflectance data. In terms of predictive ability, GA-PLS regression was compared with iterative stepwise elimination PLS (ISE-PLS) regression and standard full-spectrum PLS (FS-PLS) regression using soil samples collected in 2015 and 2016 from the surface of upland and lowland rice fields in Madagascar (n = 103). Overall, the GA-PLS model using first derivative reflectance (FDR) had the best predictive accuracy (R2 = 0.796) with a good prediction ability (residual predictive deviation (RPD) = 2.211). Selected wavebands in the GA-PLS model did not perfectly match wavelengths of previously known absorption features of soil nutrients, but in most cases, the selected wavebands were within 20 nm of previously known wavelength regions. Bootstrap procedures (N = 10,000 times) using selected wavebands also confirmed the improvements in accuracy and robustness of the GA-PLS model compared to those of the ISE-PLS and FS-PLS models. These results suggest that soil oxalate-extractable P can be predicted from Vis-NIR spectroscopy and that GA-PLS regression has the advantage of tuning optimum bands for PLS regression, contributing to a better predictive ability.

This paper investigated the feasibility of Fourier transform infrared transmission (FT-IR) spectroscopy to detect talcum powder illegally added in tea based on chemometric methods. Firstly, 210 samples of tea powder with 13 dose levels of talcum powder were prepared for FT-IR spectra acquirement. In order to highlight the slight variations in FT-IR spectra, smoothing, normalize and standard normal variate (SNV) were employed to preprocess the raw spectra. Among them, SNV preprocessing had the best performance with high correlation of prediction (RP = 0.948) and low root mean square error of prediction (RMSEP = 0.108) of partial least squares (PLS) model. Then 18 characteristic wavenumbers were selected based on a hybrid of backward interval partial least squares (biPLS) regression, competitive adaptive reweighted sampling (CARS) algorithm and successive projections algorithm (SPA). These characteristic wavenumbers only accounted for 0.64% of the full wavenumbers. Following that, 18 characteristic wavenumbers were used to build linear and nonlinear determination models by PLS regression and extreme learning machine (ELM), respectively. The optimal model with RP = 0.963 and RMSEP = 0.137 was achieved by ELM algorithm. These results demonstrated that FT-IR spectroscopy with chemometrics could be used successfully to detect talcum powder in tea.

Detection of mango soluble solid content using hyperspectral imaging technology

Variable selection can improve the robustness and prediction accuracy of partial least squares (PLS) regression models and decrease the calculation time by selecting the optimal subset of variables in multivariate calibration. In this study, the performance of two variable selection methods for wavelength interval and individual wavelength coupled with partial least squares regression are investigated by employing the experimental data of asiaticoside (AS) and madecassoside (MS) contents in centella total glucosides (CTG) and a public dataset of corn. The studied variable selection methods include interval partial least squares regression ( iPLS), backward interval partial least squares ( biPLS), synergy interval partial least squares regression ( siPLS), competitive adaptive reweighted sampling (CARS), uninformative variable elimination (UVE) and variable importance in projection (VIP). The results show that the implementation of variable selection methods improved the performance of the model compared with full-spectrum modeling. All variable selection methods improved the prediction of AS or MS contents in CTG. When latent variables for PLS models are less than 10 in the practical application, the RPD value of AS models by iPLS method is 7.5, and the RPD value of MS models by biPLS method is 2.9. The results of wavelength interval selection are better than individual wavelength selection, especially for iPLS and biPLS. The same results were obtained with the public data for moisture in corn, and the RPD value of biPLS model of moisture is 1.6. Therefore, the wavelength interval selection methods, such as iPLS or biPLS, are appropriate for improving the PLS model’s accuracy and robustness to determine the target components’ contents in solid samples.

Accurate determination of Soil Organic Carbon (SOC), which is the foundation of soil health and safeguards ecological and food security, is crucial in local agricultural production. We aimed to investigate the influence of soil texture on hyperspectral models for predicting SOC content and to evaluate the role of different preprocessing methods and feature band selection algorithms in improving modeling efficiency. Laboratory-determined SOC content and hyperspectral reflectance data were obtained using soil samples from daylily cultivation areas in Yunzhou District, Datong City. Mathematical transformations, including Savitzky–Golay smoothing (SG), First Derivative (FD), Second Derivative (SD), Multiplicative Scatter Correction (MSC), and Standard Normal Variate (SNV), were applied to the spectral reflectance data. Feature bands extracted based on the successive projection algorithm (SPA) and Competitive Adaptive Reweighted Sampling (CARS) were used to establish SOC content inversion models employing four algorithms: partial least-squares regression (PLSR), Random Forest (RF), Backpropagation Neural Network (BP), and Convolutional Neural Network (CNN). The results indicate the following: (1) Preprocessing can effectively increase the correlation between the soil spectral reflectance process and SOC content. (2) SPA and CARS effectively screened the characteristic bands of SOC in daylily cultivated soil from the spectral curves. The SPA algorithm and CARS selected 4–11 and 9–122 bands, respectively, and both algorithms facilitated model construction. (3) Among all the constructed models, the FD-CARS-PLSR performed most prominently, with coefficients of determination (R2) for the training and validation sets reaching 0.93 and 0.83, respectively, demonstrating high model stability and reliability. (4) Incorporating soil texture as an auxiliary variable into the PLSR inversion model improved the inversion accuracy, with accuracy gains ranging between 0.01 and 0.05.

Hyperspectral imaging technology was used to assess the soluble solids content (SSC), titrable acidity content (TAC), and firmness of hami melons. The mean spectra were extracted from the regions of interest (ROI) of the hyperspectral images of each hami melon. Spectral data were first pretreated with different preprocessing methods and analyzed using the partial least squares (PLS) method to build calibration models. However, full spectral data contain a great number of redundant and colinear variables that lead to poor predicting ability of calibration models. Three typical variable selection methods, i.e., the genetic algorithm (GA), successive projections algorithm (SPA), and competitive adaptive reweighted sampling (CARS) were used to extract effective wavelength variables from the full spectra for determination of SSC, TAC, and firmness of hami melon. The results showed that the SNV‐CARS‐PLS model based on 51 and 40 wavelength variables achieved the optimal performance for SSC and TAC compared with full spectral SNV‐PLS, SNV‐GA‐PLS, and SNV‐SPA‐PLS models. The RP, RMSEP, and RPD of SNV‐CARS‐PLS model for SSC were 0.9606, 0.3816, and 3.598, respectively, and 0.9125, 0.0263, and 2.445 for TAC, respectively. The best model for predicting firmness was RAW‐CARS‐PLS model based on 57 wavelength variables with the RP RMSEP, and RDP values of 0.8671, 20.05, and 1.996, respectively. Overall, the results indicate that the CARS is a powerful method for selecting effective wavelength variables and demonstrated the feasibility of using hyperspectral imaging technology as a fast and nondestructive method for simultaneous detection of SSC, TAC, and firmness of hami melons.Practical applicationsThe quality of melon has a direct relationship with the soluble solids content, titrable acidity content, and texture of melon. To ensure the best taste and highest nutritional value, it is better for melons to be harvested when completely mature. Nowadays, melons were harvested according to different fruit development times after flowering. Traditional methods for detection of internal qualities and maturity of melon mainly focus on the appearance of melon and depend on the feelings of professionals, which are tedious, time‐consuming, expensive, and greatly influenced by subjective factors. More than anything, Traditional methods are hard to be highly accurate. Hyperspectral imaging technology has the advantages of rapid and nondestructive. The results showed that hyperspectral imaging technology for the simultaneous detection of soluble solids content, titrable acidity content, and firmness of melon is feasible. And the results can provide a reference for detecting large fruits by using hyperspectral imaging technology

Estimating the available potassium (AK) in soil can help improve field management and crop production. Fourier-transform infrared (FTIR) spectroscopy is one of the most promising techniques for the fast and real-time analysis of soil AK content. However, the successful estimation of soil AK content by FTIR depends on the proper selection of appropriate spectral dimensionality reduction techniques. To magnify the subtle spectral signals concerning AK content and improve the understanding of the characteristic FTIR wavelengths of AK content, a total of 145 soil samples were collected in an agricultural site located in the southwest part of Sichuan, China, and three typical spectral dimensionality reduction methods—the successive projections algorithm (SPA), simulated annealing algorithm (SA) and competitive adaptive reweighted sampling (CARS)—were adopted to select the appropriate spectral variable. Then, partial least squares regression (PLSR) was utilized to establish AK inversion models by incorporating the optimal set of spectral variables extracted by different dimensionality reduction algorithms. The accuracy of each inversion model was tested based on the coefficient of determination (R2), root mean square error (RMSE) and mean absolute value error (MAE), and the contribution of the inversion model variables was explored. The results show that: (1) The application of spectral dimensionality reduction is a useful technique for isolating specific components of multicomponent spectra, and as such is a powerful tool to improve and expand the predicted potential of the spectroscopy of soil AK content. Compared with the SA and CARS algorithms, the SPA was more suitable for soil AK content inversion. (2) The inversion model results showed that the characteristic wavelengths were mainly around 777 nm, 1315 nm, 1375 nm, 1635 nm, 1730 nm and 3568–3990 nm. (3) Comparing the performances of different inversion models, the SPA–PLSR model (R2= 0.49, RMSE = 22.80, MAE = 16.82) was superior to the SA–PLSR and CARS–PLSR models, which has certain guiding significance for the rapid detection of soil AK content.

在调查分析长江上游现有坡耕地植物篱的配置方式、生长状况和室内分析植物篱系统内土壤有机质和土壤理化性质分析的基础上，对3种植物篱系统中土壤有机质分布特征及其与土壤理化性质的关系进行了研究，结果表明：坡耕地系统中不同位置土壤有机质含量存在显著差异，3种植物篱带间坡耕地土壤有机质平均含量为6.33 g∙kg−1，植物篱带上、带内和植物篱带下土壤有机质含量比植物篱带间坡耕地土壤有机质平均含量分别增加27.9%，67.3%和31.6%；植物篱系统内土壤有机质与土壤孔隙度、土壤含水量、饱和导水率、土壤抗冲指数、土壤抗蚀指数、水稳性团聚体极显著正相关，而与土壤容重呈显著负相关关系。土壤有机质与土壤全氮、水解氮、土壤全钾、有效钾、全磷、有效磷和阳离子交换量具有极显著正相关关系。 To achieve the soil organic matter distribution characters and the relationship between physicochemical properties of hedgerow systems in the Three Gorges reservoir area, soil organic matter and physicochemical properties of soil collected from different position in hedgerow systems with different species were tested, and the coefficient between soil organic matter and physicochemical properties was analyzed. The result revealed that 1) the mean value of soil organic matter of the steep land between hedgerows was 6.33 g∙kg−1, and the soil organic matter of soil before, within and behind the hedgerows were higher than that of soil in steep land between hedgerows, by 27.9%, 67.3% and 31.6% respectively. 2) The organic matter of soil in hedgerow systems was correlated positively and highly significantly (P < 0. 01) with the soil porosity, water content, soil saturated water conductivity, anti-erodibility, anti-scouribility and soil water stable aggregates content, and correlated negatively with soil bulk density. In addition, the organic matter of soil in hedgerow systems was highly correlated with soil nutrients, containing total and exchangeable nitrogen, total and exchangeable phosphorus, total and exchangeable potassium, and cation exchangeable capacity.

Microbial activity and organic matter composition in Mediterranean humus forms

Soil nitrate content and organic matter are two important parameters that determine the amount of nitrogen available for plant growth. The goal of this study was to develop soil nitrate and organic matter sensing techniques, since traditional methods of determining soil organic matter and nitrate content are tedious, time-consuming, and expensive. Studies conducted at UC Davis have clearly shown that unmistakable nitrate peaks exist in the mid-infrared (MIR) spectra of soil pastes. In particular, the peak located at 1,390 cm−1 wave number is nitrate sensitive. However, this peak is broad and is influenced by carbonate/bicarbonate and organic matter peaks, which exist in the vicinity of the nitrate peak. Using wavelet analysis, we were able to isolate a single wave number, 1,350 cm−1, at which these interferences were minimal. However, if the soil contained a large amount of carbonates, a second wavelength located at 1,500 cm−1 was necessary to obtain a unique calibration equation. Using the absorbance values at these two wave numbers, nitrate-N concentrations of 14 different soils (6 Californian soils and 8 Israeli ones) could be determined using a single calibration curve with a coefficient of multiple determination (R 2 value) of 0.98. When only six Californian field soils were considered, the calibration equation resulted in an R 2 value of 0.95 with a standard error of 8 ppm. Moreover, preliminary tests indicated that soil organic matter can also be determined using MIR spectral response at two wave numbers (1,383 and 1,452 cm−1). The R 2 value for soil organic matter determination was 0.95 and the standard error was 0.25%. These results indicate that both soil nitrate content and organic matter can be determined using just a few wavelengths in the MIR region with sufficient accuracy for use in site-specific nutrient management.

Biodegradability of soluble organic matter in maize-cropped soils