Soil spectroscopy improves mid infrared soil property prediction through optimized preprocessing and variable selection

As an important process analysis tool, near infrared spectroscopy (NIRS) has been widely used in process monitoring. In the present work, the feasibility of NIRS for monitoring the moisture content of human coagulation factor VIII (FVIII) in freeze-drying process was investigated. A partial least squares regression (PLS-R) model for moisture content determination was built with 88 samples. Different pre-processing methods were explored, and the best method found was standard normal variate (SNV) transformation combined with 1st derivation with Savitzky–Golay (SG) 15 point smoothing. Then, four different variable selection methods, including uninformative variable elimination (UVE), interval partial least squares regression (iPLS), competitive adaptive reweighted sampling (CARS) and manual method, were compared for eliminating irrelevant variables, and iPLS was chosen as the best variable selection method. The correlation coefficient (R), correlation coefficient of calibration set (R cal ), correlation coefficient of validation set (R val ), root mean square errors of cross-validation (RMSECV) and root mean square errors of prediction (RMSEP) of PLS model were 0.9284, 0.9463, 0.8890, 0.4986% and 0.4514%, respectively. The results showed that the model for moisture content determination has a wide range, good linearity, accuracy and precision. The developed approach was demonstrated to be a potential for monitoring the moisture content of FVIII in freeze-drying process.

Variable selection in partial least squares with the weighted variable contribution to the first singular value of the covariance matrix

Applying NIR and MIR spectroscopy for C and soil property prediction in northern cold-region ecosystems. Which approach works better?

Variable selection can improve the robustness and prediction accuracy of partial least squares (PLS) regression models and decrease the calculation time by selecting the optimal subset of variables in multivariate calibration. In this study, the performance of two variable selection methods for wavelength interval and individual wavelength coupled with partial least squares regression are investigated by employing the experimental data of asiaticoside (AS) and madecassoside (MS) contents in centella total glucosides (CTG) and a public dataset of corn. The studied variable selection methods include interval partial least squares regression ( iPLS), backward interval partial least squares ( biPLS), synergy interval partial least squares regression ( siPLS), competitive adaptive reweighted sampling (CARS), uninformative variable elimination (UVE) and variable importance in projection (VIP). The results show that the implementation of variable selection methods improved the performance of the model compared with full-spectrum modeling. All variable selection methods improved the prediction of AS or MS contents in CTG. When latent variables for PLS models are less than 10 in the practical application, the RPD value of AS models by iPLS method is 7.5, and the RPD value of MS models by biPLS method is 2.9. The results of wavelength interval selection are better than individual wavelength selection, especially for iPLS and biPLS. The same results were obtained with the public data for moisture in corn, and the RPD value of biPLS model of moisture is 1.6. Therefore, the wavelength interval selection methods, such as iPLS or biPLS, are appropriate for improving the PLS model’s accuracy and robustness to determine the target components’ contents in solid samples.

In this study, a regression model of paddy soil properties using diffuse reflectance spectroscopy was developed to replace chemical soil analysis as a more efficient alternative. Soil samples were collected and analyzed from saltwater paddy fields located in Jeongnam-myeon, Hwaseong-si, Gyeonggi-do in the Republic of Korea, and the spectral data of wet and dry soil were collected. The regression models were compared and analyzed using partial least squares regression (PLSR) with Savitzky–Golay smoothing (SG smoothing) and Standard Normal Variate (SNV) preprocessing to predict the soil properties. Analysis showed that the predictive regression model of wet soil with SG smoothing and an SNV did not meet the evaluation criteria of a fair model. However, the regression model of dry soil with SG smoothing was fair for clay, pH, EC, and TN at RPD = 1.90, 1.87, 1.60, and 1.43 and R2 = 0.79, 0.81, 0.64, and 0.64, respectively, while the regression model of dry soil with an SNV was good for clay, pH, EC, and TN at RPD = 2.21, 1.96, 1.70, and 1.44 and R2 = 0.84, 0.81, 0.76, 0.69, respectively. When developing predictive regression models of soil properties, the accuracy for dry soil was higher than that for wet soil, and when applying a single round of preprocessing, the regression model with SNV preprocessing was more accurate than that with SG smoothing.

A computational approach for the identification and investigation of correlations between a chemical structure and a selected biological property is described. It is based on a set of 132 compounds of known chemical structures, which were tested for their binding affinities to the estrogen receptor. Different multivariate modeling methods, i.e., partial least-squares regression, counterpropagation neural network, and error-back-propagation neural network, were applied, and the prediction ability of each model was tested in order to compare the results of the obtained models. To reduce the extensive set of calculated structural descriptors, two types of variable selection methods were applied, depending on the modeling approach used. In particular, the final partial least-squares regression model was built using the "variable importance in projection" variable selection method, while genetic algorithms were applied in neural network modeling to select the optimal set of descriptors. A thorough statistical study of the variables selected by genetic algorithms is shown. The results were assessed with the aim to get insight to the mechanisms involved in the binding of estrogenic compounds to the receptor. The variable selection on the basis of genetic algorithm was controlled with the test set of compounds, extracted from the data set available. To compare the predictive ability of all the optimized models, a leave-one-out cross-validation procedure was applied, the best model being the nonlinear neural network model based on error back-propagation algorithm, which resulted in R2= 92.2% and Q2= 70.8%.

In this paper, an efficient variable and subset selection approach related to modified partial least squares (PLS) regression are proposed and applied to numerical and industry application to show the superior of the results. The main purpose of the proposed variable selection approach is to offer further useful information to identification, control as well as diagnosis. The significant advantage of the modified PLS-based variable selection method is to keep the superior fitting and prediction performance as well as the significantly reduced number of selected variables. To this aim, the canonical and modified PLS regression methods have been introduced. Then an effective and precise variable selection method which can select the most significant elements among huge redundant information is proposed. To this end, the effectiveness of the proposed variable selection method is demonstrated by the simulation results of a numerical instance and a process industry case study.

Visible/near infrared-partial least-squares analysis of Brix in sugar cane juice: A test field for variable selection methods

The combined optimization of Savitzky-Golay (SG) smoothing and multiplicative scatter correction (MSC) were discussed based on the partial least squares (PLS) models in Fourier transform near-infrared (FT-NIR) spectroscopy analysis. A total of 5 cases of separately (or combined) using SG smoothing and MSC were designed and compared for optimization. For every case, the SG smoothing parameters were optimized with the number of PLS latent variables (F), with an expanded number of smoothing points. Taking the FT-NIR analysis of soil organic matter (SOM) as an example, the joint optimization of SG smoothing and MSC was achieved based on PLS modeling. The results showed that the optimal pretreatment was successively using SG smoothing and MSC, in which the SG smoothing parameters were 4th degree of polynomial, 2nd-order derivative, and 67 smoothing points, the best corresponding F, RMSEP, and RP were 7, 0.3982 (%), and 0.8862, respectively. This result was far better than those without any pretreatment. The combined optimization of SG smoothing and MSC could obviously improve the modeling result for NIR analysis of SOM. In addition, a new method for the classification of calibration and prediction was proposed by normalization principle. The optimizations were done on this basis of this classification.

Variable (wavelength) selection is essential in the multivariate analysis of near-infrared spectra to improve model performance and provide a more straightforward interpretation. This paper proposed a new variable selection method named binning-normalized mutual information (B-NMI) based on information entropy theory. "Data binning" was applied to reduce the effects of minor measurement errors and increase the features of near-infrared spectra. "Normalized mutual information" was employed to calculate the correlation between each wavelength and the reference values. The performance of B-NMI was evaluated by two experimental datasets (ideal ternary solvent mixture dataset, fluidized bed granulation dataset) and two public datasets (gasoline octane dataset, corn protein dataset). Compared with classic methods of backward and interval PLS (BIPLS), variable importance projection (VIP), correlation coefficient (CC), uninformative variables elimination (UVE), and competitive adaptive reweighted sampling (CARS), B-NMI not only selected the most featured wavelengths from the spectra of complex real-world samples but also improved the stability and robustness of variable selection results.

Core Ideas Selection of spectral regions was included in MIR predictions of soil properties. The R software outperformed a commercial chemometric software. SVMR and model averaging performed better than PLSR and artificial neural networks. With decreasing calibration sample size, the usefulness of SVMR over PLSR decreased. MIRS studies may focus more on representativeness, sample sizes, and variabilities. Different algorithms exist in various software programs for the estimation of soil properties using mid‐infrared (MIR) spectroscopy, with recommendations varying between different studies regarding which algorithm should be used. Objectives were to compare the performance of the commercial OPUS Quant 2 software, which uses partial least squares regression (PLSR) and a selection of spectral ranges, with the R software and to study the accuracy of different algorithms as a function of the information provided in the calibration. Contents of soil organic carbon (SOC), nitrogen, and texture for surface soils of an arable field were determined, and MIR were spectra recorded. Partial least squares regression used with either software was useful (ratio of performance to interquartile distance in the validation sample [RPIQ V ] >1.89) for an estimation of SOC, clay, and N contents but not for sand and silt. The wavenumber region selection concept used in OPUS was also implemented in R, and it proved useful for SOC (all algorithms) and total nitrogen (artificial neural networks, support vector machine regression [SVMR]) in the validation. Support vector machine regression generally slightly outperformed the other approaches and resulted in a successful estimation of sand content. The usefulness of SVMR over PLSR generally decreased with decreasing sample size used for the calibration (thus decreasing the information provided), and PLSR partly outperformed SVMR in the validation. Overall, this study indicates that there is no general superiority of a chemometric algorithm over PLSR independent of the information provided in the calibration sample.

ABSTRACTThe quality of green forage is crucial in pasture grazing, influencing both animal welfare, environmental sustainability, and production yield. Traditionally, the evaluation of forage composition requires time‐consuming and costly chemical analysis. In this context, near‐infrared spectroscopy (NIR) emerges as a promising alternative. This study adopted Fourier transform NIR (FT‐NIR) spectroscopy to predict nutritional characteristics of green forages. A total of 324 samples were collected from pastures in central Italy. Partial least squares (PLS) regression models were then developed, applying variable selection methods to improve PLS model accuracy. The interval PLS (iPLS) variable selection method gave the best results for fresh forage, while the genetic algorithm (GA) performed best for dried samples. The best results from the PLS models were obtained for dry matter (DM) and crude protein (CP). The DM model for fresh forage yielded an R2P of 0.96 and an RMSEP of 2.95 g 100 g−1 FW, while the CP model for dried forage yielded an R2P of 0.94 and an RMSEP of 1.84 g 100 g−1 DW, with a normalised root‐mean‐square error of cross‐validation (NRMSECV) of 3.8% and 5.6%, respectively. The results for the neutral detergent fibre (aNDF) were acceptable. NIR spectroscopy has proven to be a useful tool for assessing forage nutritional quality. Variable selection through iPLS also enabled the identification of “core” spectral regions for the development of compact and portable NIR sensors. Future research should further investigate sample preparation and moisture content effects and expand sampling to different geographical areas to enhance model robustness.

Using a mobile real-time soil visible-near infrared sensor for high resolution soil property mapping

Quality assessment of diesel fuel is highly necessary for society, but the costs and time spent are very high while using standard methods. Therefore, this study aimed to develop an analytical method capable of simultaneously determining eight diesel quality parameters (density; flash point; total sulfur content; distillation temperatures at 10% (T10), 50% (T50), and 85% (T85) recovery; cetane index; and biodiesel content) through attenuated total reflection Fourier transform infrared (ATR-FTIR) spectroscopy and the multivariate regression method, partial least square (PLS). For this purpose, the quality parameters of 409 samples were determined using standard methods, and their spectra were acquired in ranges of 4000–650 cm−1. The use of the multivariate filters, generalized least squares weighting (GLSW) and orthogonal signal correction (OSC), was evaluated to improve the signal-to-noise ratio of the models. Likewise, four variable selection approaches were tested: manual exclusion, forward interval PLS (FiPLS), backward interval PLS (BiPLS), and genetic algorithm (GA). The multivariate filters and variables selection algorithms generated more fitted and accurate PLS models. According to the validation, the FTIR/PLS models presented accuracy comparable to the reference methods and, therefore, the proposed method can be applied in the diesel routine monitoring to significantly reduce costs and analysis time.

The inversion of arid-coastal cultivated soil salinity using explainable machine learning and Sentinel-2