Chapter 19 - Statistical (Machine) Learning: Using Complex Models With Large Data Sets

Purpose This study aims to enhance cryptocurrency price and trend prediction by applying advanced machine learning (ML) techniques. Given the market’s high volatility and complexity, the research identifies effective models for different conditions, providing insights for investors and risk management. Design/methodology/approach This study proposes a six-stage framework for cryptocurrency price prediction, integrating advanced ML techniques. Data from ten cryptocurrencies are processed, extracting 37 key features, including return, the Fear and Greed Index and various technical indicators. The model employs Deep Q-Networks (DQN), Long Short-Term Memory (LSTM) and multiple regression methods such as linear regression, support vector regression, ridge, LASSO, decision tree, Random Forest, multi-layer perceptron, stochastic gradient descent, elastic net and Bayesian regression. Model performance is evaluated using trading strategies and metrics like accuracy, sensitivity, recall, MSE, MAE and F1-score. Findings The results indicate that complex models like DQN and LSTM excel in volatile markets due to their ability to capture intricate price patterns, whereas simpler models such as linear regression and ridge regression perform better in stable conditions. The multi-layered parallel design enhances computational efficiency, enabling independent asset evaluation. These findings highlight the potential of artificial intelligence in improving prediction accuracy and supporting informed investment decisions. Originality/value This research introduces a novel six-stage ML framework incorporating diverse predictive models and key features for cryptocurrency forecasting. The multi-layered parallel approach enhances computational efficiency, setting this study apart from existing research. The comparative analysis of models offers valuable guidance for investors, traders and financial analysts navigating volatile cryptocurrency markets.

This study applies machine learning (ML) techniques to model and predict Assam’s agricultural Gross State Domestic Product (GSDP). Three predictive models—multiple linear regression, random forest regression, and gradient boosting—are evaluated. The random forest model achieved the best fit, exhibiting the highest R² and the lowest mean squared error (MSE) and Akaike information criterion (AIC), along with statistically significant coefficients. Ensemble methods (random forest and gradient boosting) markedly improve forecast accuracy of agricultural growth trends compared to traditional regression, yielding more reliable predictions of productivity and GSDP contributions. The findings underscore the vital role of agricultural productivity in driving economic growth, strengthening GSDP, and supporting food security and employment. Integrating advanced ML techniques with statistical analysis provides insights for policymakers to make data-driven decisions that foster sustainable agricultural development and economic prosperity in Assam. Objectives: Predict Assam’s agricultural sector performance using selected machine learning models. Evaluate and compare the effectiveness of these models in assessing the state’s agricultural economy. Methods: Data preprocessing involved handling outliers (using interquartile range and mean-max scaling) and feature selection via correlation heatmaps. Predictive models (multiple linear regression, random forest regression, and gradient boosting) were implemented in Python. Results: The gradient boosting model emerged as the most effective, achieving the highest accuracy and generalization (testing R² = 0.9867). Farm area, labour, maize yield, and autumn rice yield were the most significant positive contributors to GSDP. The random forest model performed similarly well (R² = 0.9867), while the multiple linear regression model was least accurate (R² = 0.9521), likely due to its inability to capture nonlinear relationships. Conclusions: Machine learning models offer transformative potential for Assam’s agricultural sector. Leveraging data-driven insights from these models can empower policymakers to design targeted interventions, promoting inclusive and sustainable economic growth in the region.

Daily scale streamflow forecasting in multiple stream orders of Cauvery River, India: Application of advanced ensemble and deep learning models

The Internet of Things (IoT) has become a foundational element of the digital infrastructure, extending its connectivity across various sectors and embedding intelligence in everyday devices. This article introduces SecureNet, a pioneering approach that integrates Machine Learning (ML), Blockchain, and Federated Learning (FL) to enhance IoT security. To navigate this challenging train, an innovative framework that synergizes Machine Learning (ML), Blockchain technology, and Federated Learning (FL) to fortify IoT security. SecureNet is architected to deliver a robust defense mechanism for IoT ecosystems, providing resilience against increasingly sophisticated cyber threats, and ensuring the preservation of data integrity, privacy, and unwavering system reliability. This study explores the application of advanced ML techniques NSL-KDD dataset, implementing two highly effective classifiers: Random Forest and Logistic Regression. The Random Forest classifier exhibited an exceptional accuracy of 99.85%, while the Logistic Regression model demonstrated a near-perfect accuracy of 99.03%. These compelling results highlight the efficacy of ML in identifying and mitigating activities within network traffic. SecureNet leverages ML’s profound analytical capabilities for intelligent threat discernment, Blockchain’s immutable ledgers for unassailable data verification, and FL’s privacy-centric approach to distribute model training. These outcomes underscore the potential of ML models to enhance IoT security by accurately identifying malicious patterns and anomalies within network traffic.

Early detection of ankylosing spondylitis using texture features and statistical machine learning, and deep learning, with some patient age analysis

Machine learning in computational docking.

The machine learning (ML) and Internet of things (IoT) technologies are increasingly focussed on decision tree classification algorithm. Its use is expanding through numerous fields, solving incredibly complex problems. DTCA is also being used in medical health data using computer-aided diagnosis to identify chronic kidney diseases like cancer and diabetes. Deep learning is a class of machine learning that utilizes neural networks to solve problems and learn unsupervised from unstructured or unlabelled data. The DL used to deep stacked auto-encoder and softmax classifier methods is applied for CKD. In this work, based an R Studio and Python Colab software using random forest, SVM, C5.0, decision tree classification algorithm, C4.5, ANN, neuro-fuzzy systems, classification and clustering, CNN, RNN, MLP is used to predict multiple machine and deep learning techniques, discover an early diagnosis of CKD patients. In this work, classify the chronic kidney disease various stages are identified.KeywordsDecision tree classification algorithmComputational decision support systemChronic kidney diseaseDeep and machine learning algorithmsIoT and CKD stages

Breast cancer is one of the most prevalent and chronic disease that affect women. To overcome this disease, effective medical treatment is required. Early detection of the disease plays an important role for suitable medication and survival of patient. To identify the breast cancer in the patients, standard imaging technique mammography is used. Due to the subtle and varied nature of cancer tissues interpreting mammogram images can be a challenge to doctors. Machine learning (ML) and Deep Learning (DL) techniques offer promising solutions that provide efficient breast cancer detection from mammograms. In this review paper a comprehensive review of ML and DL algorithms and their applications in mammogram image analysis are presented. Various supervised and unsupervised learning techniques, such as convolutional neural networks (CNNs), support vector machines (SVMs), random forests, and other popular ML and DL models are discussed in paper. The integration of these DL methods that are efficiently used in image preprocessing techniques, feature extraction, and classification strategies. The overall survey focusses on various performance metrics, datasets, and benchmarks used in existing studies. Further the strengths and limitations of different approaches used by various researchers are identified. By understating current research trends this paper aims to contribute to the ongoing development of more accurate and reliable breast cancer detection systems using advanced ML techniques.

In an open source software development environment, it is hard to decide the number of group members required for resolving software issues. Developers generally reply to issues based totally on their domain knowledge and interest, and there are no predetermined groups. The developers openly collaborate on resolving the issues based on many factors, such as their interest, domain expertise, and availability. This study compares eight different algorithms employing machine learning and deep learning, namely—Convolutional Neural Network, Multilayer Perceptron, Classification and Regression Trees, Generalized Linear Model, Bayesian Additive Regression Trees, Gaussian Process, Random Forest and Conditional Inference Tree for predicting group size in five open source software projects developed and managed using an open source development framework GitHub. The social information foraging model has also been extended to predict group size in software issues, and its results compared to those obtained using machine learning and deep learning algorithms. The prediction results suggest that deep learning and machine learning models predict better than the extended social information foraging model, while the best-ranked model is a deep multilayer perceptron((R.M.S.E. sequelize—1.21, opencv—1.17, bitcoin—1.05, aseprite—1.01, electron—1.16). Also it was observed that issue labels helped improve the prediction performance of the machine learning and deep learning models. The prediction results of these models have been used to build an Issue Group Recommendation System as an Internet of Things application that recommends and alerts additional developers to help resolve an open issue.

This project focuses on predicting heart, lung, and brain diseases using advanced machine learning (ML) and deep learning (DL) techniques. The system analyzes complex patterns from diverse datasets, including patient demographics, clinical information, and diagnostic imaging. By employing algorithms such as Random Forest, Support Vector Machines (SVM), and Convolutional Neural Networks (CNNs), it effectively identifies disease-specific indicators. Recent research demonstrates AI's remarkable accuracy-over 95 % in heart disease prediction, 96.57 % in lung disease detection (including pneumonia and COVID-19), and promising results in early dementia prediction through eye scans. The proposed framework aims to support healthcare professionals by providing data-driven, early diagnostic insights, thereby enhancing decision-making, enabling timely interventions, and improving overall patient care and outcomes.

Purpose This study aims to examine the potential effect that corporate social responsibility (CSR) practices in general and business ethics, more specifically, can exert on the financial performance of companies using advanced machine learning (ML) techniques. Design/methodology/approach Based on a sample of 523 international firms headquartered in North America and Western Europe between 2005 and 2019, the research design employs advanced machine learning algorithms including LightGBM, CatBoost, XGBoost, NGBoost, Support Vector Machine, and Random Forest to assess the relationship between business ethics, CSR practices, and financial performance. In addition, the SHapley Additive exPlanations (SHAP) technique was used to provide comprehensive insights into feature contributions to individual predictions. Findings Findings indicate a growing interest in business ethics and societal practices over the past decade. The results have revealed that the adoption of an ethical and socially responsible approach is associated with financial performance. Practical implications This study provides actionable insights for multinational companies, emphasizing the role of business ethics and social responsibility practices in enhancing financial performance. It also holds significance for practitioners aiming to improve financial efficiency. By exploring the relationship between social effectiveness and financial efficiency, this research leverages ML techniques to offer a deeper understanding of this critical connection. Originality/value This study contributes to the field by using advanced ML techniques, including SHAP, to explore the relationship between business ethics and CSR practices on Financial Performance. The findings, particularly the superior accuracy of XGBoost among algorithms, provide valuable insights into the implications of business ethics and social practices on firm financial performance.

Abstract. Firn density plays a crucial role in assessing the surface mass balance of the Antarctic ice sheet. However, our understanding of the spatial and temporal variations in firn density is limited due to (i) spatial and temporal limitations of in situ measurements, (ii) potential modelling uncertainties, and (iii) lack of firn density products driven by satellite remote-sensing data. To address this gap, this paper explores the potential of satellite microwave radiometer (Special Sensor Microwave Imager/Sounder (SSMIS)) and scatterometer (Advanced Scatterometer (ASCAT)) observations for assessing spatial and temporal dynamics of dry-firn density over the Antarctic ice sheet. Our analysis demonstrates a clear relation between density anomalies at a depth of 40 cm and fluctuations in satellite observations. However, a linear relationship with individual satellite observations is insufficient to explain the spatial and temporal variation in snow density. Hence, we investigate the potential of a non-linear random forest (RF) machine learning approach trained on radiometer and scatterometer data to derive the spatial and temporal variations in dry-firn density. In the estimation process, 10 years of SSMIS observations (brightness temperature) and ASCAT observations (backscatter intensity) is used as input features to a random forest (RF) regressor. The regressor is first trained on time series of modelled density and satellite observations at randomly sampled pixels and then applied to estimate densities in dry-firn areas across Antarctica. The RF results reveal a strong agreement between the spatial patterns estimated by the RF regressor and the modelled densities. The estimated densities exhibit an error of ±10 kg m−3 in the interior of the ice sheet and ±35 kg m−3 towards the ocean. However, the temporal patterns show some discrepancies, as the RF regressor tends to overestimate summer densities, except for high-elevation regions in East Antarctica and specific areas in West Antarctica. These errors may be attributed to underestimations of short-term or seasonal variations in the modelled density and the limitations of RF in extrapolating values outside the training data. Overall, our study presents a potential method for estimating unknown Antarctic firn densities using known densities and satellite parameters.

Integration of metabolomics and machine learning for precise management and prevention of cardiometabolic risk in Asians.

Cell identification and phenotyping using classical machine learning and deep learning Recent image analysis and machine learning technologies have enabled the characterization of biology and human health in multiple dimensions, even at the single cell level. By revealing unique characteristics of any cells of interest, single cell analysis offers significant advantages in various fields of science from cancer or brain research through drug development to personalized and targeted treatments. Usually the first step in most microscopy-based single cell analyses task is the identification of nuclei. Precise localization and identification are fundamental for the accurate assessment of important cell functions. Previously the leading approaches were based on classical image processing methods which usually require prior knowledge to accurately fine-tune the parameters assessed. Recently, the revolutionary approach of deep learning has been incorporated into image analysis, and has fundamentally contributed to achieve ground-breaking results in single cell analysis, including image classification, image registration, object detection and recognition, and its applications such as computer-aided diagnosistics. This thesis consists of two parts: phenotypic analysis (with machine learning) and single cell analysis (with deep learning). In the first part, I have focused on answering major questions in phenotypic data analysis, including how to discover the whole dataset effectively or how to increase the accuracy of classification. In the second part the primary focus is on deep learning based approaches. During my research I used and developed various software tools and created algorithms to support in-depth single cell analysis. All of these are aimed to support a higher level of single cell detection and segmentation. This part covers some major research tasks in which I have participated, thus a full automation of a patch-clamping system previously driven manually and a standalone tool to help the user analyze object occurrences on specific regions of an image are presented here. As a result, we have successfully demonstrated how to apply machine learning methods effectively by implementing flexible software tools appropriate for phenotyping. Thereby, we have successfully accomplished the main aim of our research to improve time-efficiency of biological image-based data analysis for human experts. We have also demonstrated that deep learning, a more advanced machine learning technique, utilized in various approaches is capable of finding cells at high precision. Thereby, it has the potential to support automatizing and replacing human tasks in single cell research.

Background/aimsTo investigate the feasibility and accuracy of using machine learning (ML) techniques on self-reported questionnaire data to predict the 10-year risk of cataract surgery, and to identify meaningful predictors of...