Big Data Analysis Using Unsupervised Machine Learning: K-means Clustering and Isolation Forest Models for Efficient Anomaly Detection and Removal in Complex Lithologies

Accurate energy demand prediction is crucial for efficient grid management and resource optimization, particularly across multiple countries with varying consumption patterns. However, real-world energy demand data often contains outliers that can distort forecasting accuracy. This study evaluates the impact of five outlier detection techniques—Z-Score, density- based spatial clustering of applications with noise (DBSCAN), isolation forest (IF), local outlier factor (LOF), and one-class support vector machine (SVM)—on the performance of three time-series forecasting models: long short-term memory (LSTM) networks, convolutional neural network (CNN) Autoencoders, and LSTM with attention mechanisms. The models are tested using energy demand data from four European countries— Germany, France, Spain, and Italy—derived from real-time consumption records. A comparative analysis based on root mean squared error (RMSE) demonstrates that incorporating outlier detection significantly enhances model robustness, reducing forecasting errors caused by anomalous data. The findings emphasize the importance of selecting appropriate outlier detection strategies to improve the accuracy and reliability of energy demand forecasting. This research provides valuable insights into the trade-offs involved in outlier removal, with implications for policy and operational practices in energy management.

IntroductionIn swine disease surveillance, obtaining labeled data for supervised learning models can be challenging because many farms lack standardized diagnostic routines and consistent health monitoring systems. Unsupervised learning is particularly suitable in such scenarios because it does not require labeled data, allowing for detecting anomalies without predefined labels. This study evaluates the effectiveness of unsupervised machine learning models in detecting anomalies in productivity indicators in swine breeding herds.MethodsAnomalies, defined as deviations from expected patterns, were identified in indicators such as abortions per 1000 sows, prenatal losses, preweaning mortality, total born, liveborn, culled sows per 1000 sows, and dead sows per 1000 sows. Three unsupervised models - Isolation Forest, Autoencoder, and K-Nearest Neighbors (KNN) - were applied to data from two swine production systems. The herd-week was used as the unit of analysis, and anomaly scores above the 75th percentile were used to flag anomalous weeks. A permutation test assessed differences between anomalous and non-anomalous weeks. Performance was evaluated using F1-score, precision, and recall, with true anomalous weeks defined as those coinciding with reported health challenges, including porcine reproductive and respiratory syndrome (PRRS) and Seneca Valley virus outbreaks. A total of 8,044 weeks were analyzed.ResultsThe models identified 336 anomalous weeks and 1,008 non-anomalous weeks in Production System 1, and 1,675 anomalous weeks and 5,025 non-anomalous weeks in Production System 2. The results from the permutation test revealed significant differences in productivity indicators between anomalous and non-anomalous weeks, especially during PRRS outbreaks, with more subtle changes observed during Seneca Valley virus outbreaks. The models performed well in detecting the PRRSV anomaly, achieving perfect precision (100%) across all models for both production systems. For anomalies like SVV the models showed lower performance compared to PRRSV.DiscussionThese findings suggest that unsupervised machine learning models are promising tools for early disease detection in swine herds, as they can identify anomalies in productivity data that may signal health challenges.

The increase of the classification accuracy level has become an important problem in machine learning especially in diverse data-set that contain the outlier data. In the data stream or the data from sensor readings that produce large data, it allows a lot of noise to occur. It makes the performance of the machine learning model is disrupted or even decreased. Therefore, clean data from noise is needed to obtain good accuracy and to improve the performance of the machine learning model. This research proposes a two-stages for detecting and removing outlier data by using the covariance estimator and isolation forest methods as pre-processing in the classification process to determine fine motor skill (FMS). The dataset was generated from the process of recording data directly during cursive writing by using a digitizer. The data included the relative position of the stylus on the digitizer board. x position, y position, z position, and pressure values are then used as features in the classification process. In the process of observation and recording, the generated data was very huge so some of them produce the outlier data. From the experimental results that have been implemented, the level of accuracy in the FMS classification process increases between 0.5-1% by using the Random Forest classifier after the detection and outlier removal by using covariance estimator and isolation forest. The highest accuracy rate achieves 98.05% compared to the accuracy without outlier removal, which is only about 97.3%.

Fuzzy anomaly scores for Isolation Forest

Anomaly detection is essential in applications such as financial fraud detection, network security, and system health monitoring. This study presents a hybrid anomaly detection model for financial transactions, combining the strengths of both unsupervised and supervised learning techniques—Isolation Forest and Random Forest, respectively. the Isolation Forest algorithm, performs unsupervised detection of outliers, effectively identifying rare or irregular data points without requiring labeled input. The detected anomalies are then used to construct a pseudo-labeled dataset, which is subsequently used to train a Random Forest classifier. The Random Forest model learns discriminative patterns between normal and anomalous data, enhancing overall detection capabilities. The system is implemented in Python using Scikit-learn for model development and evaluation. Experiments on benchmark financial transaction "Credit Card Fraud Detection Dataset from Kaggle" The hybrid model achieved a precision of 99.7%, recall of near 100%, and an F1-score of 99.98%, outperforming standalone algorithm The model also improved robustness to noise and better scalability for large datasets. Keywords: Anomaly Detection, Financial Fraud, Hybrid Machine Learning, Supervised Learning, Security, Unsupervised Learning. CISDI Journal Reference Format Azubuike Nnendah Daisy; Anireh, Vincent Ike-Emeka & Sako, D.J.S. (2025): A Model for Anomaly Detection in Financial Transactions using Hybrid Machine Learning Technique. Computing, Information Systems, Development Informatics and Allied Research Journal. Vol 16 No 3, Pp 37-50 Available online at www.isteams.net/cisdijournal. dx.doi.org/10.22624/AIMS/CISDI/V16N3P3

Tremendous research has been done and is still in progress in the domain of earth science. With the advent of Big Data and availability of datasets on Earth science, the study of Earth sciences has reached new dimensions. The diversity and high dimensional remote sensing data have provided with complex data sets capable of giving insights and intelligence that was not possible in last decades. With Computing progress made in ingesting and inferring data from myriad of sources including high resolution cameras mounted on satellites and sensors giving access to unconventional Big Data and also with the GPU computing and Data science advances we are today able to leverage machine learning and deep learning in extensively complex datasets derived from remote sensing about Earth Sciences. Our focus is to analyze what exactly does big data mean in earth science applications and how can big data provide added value in this context. Furthermore, this chapter demonstrates various big data tools which can be mapped with various techniques to be used for experimenting earth science datasets, processed, and exploited for different earth science applications. In order to illustrate the aforementioned aspects, instances are presented in order to demonstrate the use of big data in remote sensing. Firstly, this chapter presents earth science studies, application areas/research fields and a brief insight on earth science data. Then various techniques implemented in this domain are elaborated viz. classification, clustering, regression, deep learning, pattern recognition, machine learning, earth data analysis and processing. Later this chapter introduces big data analytics and various tools/platforms in big data viz. BigInsights, GIS, Jupyter notebook, Matlab, Python. Finally, it is shown how these tools are mapped to Earth science datasets using ArcGIS to illustrate with experimental instances the inferences and patterns generated.

BACKGROUND Patient heterogeneity is a key challenge in immunotherapy for IDH-wildtype glioblastoma (GBM), where hypermutated tumors or highly immunosuppressive tumor microenvironments (TME) act as clinical outliers affecting treatment response and trial stratification. We developed Alert-GBM, a radiogenomic AI model designed to flag such outliers based on radiomic signatures from diagnostic imaging. MATERIAL AND METHODS We used multicenter open-access radiogenomic datasets (n = 176) to train unsupervised outlier detection models. Principal Component Analysis (PCA) served as a first-pass filter to harmonize data before applying Isolation Forest (IF) modeling. Three IF models were trained using inter-dataset leave-one-out cross-validation (LOOCV) to simulate real-world performance and were benchmarked against each other. The final model was selected using stratified 10-fold cross-validation (CV). Outliers were validated through visual inspection, statistical profiling, comparison with other outlier detection methods (Local Outlier Factor, One-Class SVM, Mahalanobis distance), and correlation with biological and immune profiles from imaging and omics data. RESULTS The PCA filter identified 3 initial outliers based on MRI metadata. The final IF model, following stratified 10-fold CV, detected 6 outliers across the datasets, with anomaly scores plotted. Flagged cases included large, highly enhancing tumors, minimally enhancing tumors, and post-operative scans. Visualizations using PCA, t-SNE, and UMAP were generated to illustrate separability. False positives and negatives were assessed using immune-based scores, and the distribution of anomaly scores was analyzed with KDE plots. Cross-validation against alternative outlier models confirmed consistency of the flagged cases. CONCLUSION Alert-GBM is a proof-of-concept AI architecture for outlier detection in GBM MRI radiogenomics, capable of flagging cases with atypical immune TME profiles. Integrating such models with radiogenomic immune predictors at the time of diagnosis may support multidisciplinary decision-making and improve patient stratification for immunotherapy trials. Incorporating a human-in-the-loop framework ensures that AI outputs remain clinically interpretable and actionable.

Most working professionals as well as school and college students require very less physical activity and they are mostly seated during most part of the day. Maintaining a good posture while sitting is a step towards avoiding several lifestyle diseases that arise due to decreased physical activity. In this work, posture detection by placing multiple accelerometers on the back of user is presented. Novel use of one-class classification for detecting poor posture is proposed. One-class support vector machine and isolation forest are employed for posture detection. The classifiers are provided unlabeled data containing mostly samples of good posture. During testing, the poor postures, which we be different postures other than that considered as the good posture, are predicted as outliers. The performance of the models is compared quantitatively. The performance of one-class classifiers is also compared with commonly used unsupervised learning algorithm, k-mean clustering. Results are presented for tuning of the considered models and to study the effect of sensor placement. The isolation forest provides the highest accuracy of 99.3% and F1-score of 0.99.

Geomechanical analysis is fundamental to well planning and field development, ensuring operational safety and efficiency in the entire life cycle of the well. Traditional methods for constructing one-dimensional mechanical earth models (1D MEM) are time-intensive, requiring weeks of manual data handling and expert interpretation, with accuracy heavily dependent on complete and high-quality datasets. This study presents an integrated, machine learning (ML)-driven approach that automates the entire geomechanical workflow—from data import, quality control to generation of 1D MEM properties and wellbore stability analysis, reducing analysis time by over 90% and improving model accuracy through intelligent ML algorithms, ultimately enabling cleaner, safer, and more profitable drilling operations. The process starts with quality control and the detection of outliers via interquartile range (IQR) and isolation forest (IF) algorithms on multiple wells. The ML model fills in missing log segments and predicts any missing logs in the key wells employing the random forest regression algorithm based on training from the available wells. Rock mechanical properties are calculated by applying lithology-based industry-used formulations, and lithology is predicted with prediction errors of less than 10% using both supervised and unsupervised ML techniques. Automated normal compaction trend (NCT) generation based on shale formation is used for precise pore pressure prediction for target wells, calibrated against modular formation dynamics tester (MDT) / repeat formation tester (RFT) / drill stem test (DST) data. The subsequent 1D MEM properties are calculated using industry-standard equations that offer mud window planning and wellbore stability analysis. This ML-based solution is a paradigm shift in geomechanical analysis, cutting weeks of traditional workflows down to hours while preserving high accuracy through systematic calibration against field and core data, ultimately optimizing drilling operations and reducing non-productive time (NPT) in a range of geological settings.

The rapid growth of network data has increased the complexity of detecting anomalies, which are crucial for ensuring the security and integrity of information systems. This study investigates the use of the Isolation Forest algorithm for anomaly detection in network traffic, utilizing the Luflow Network Intrusion Detection dataset, which contains 590,086 records with 16 features related to network activities. The methodology encompasses data preprocessing (cleaning, normalization, and feature scaling), feature selection (bytes in, bytes out, entropy, and duration), model training, and performance evaluation. The results demonstrate that Isolation Forest can effectively identify anomalies based on feature patterns, isolating suspicious data points without the need for labeled datasets. However, performance metrics, such as accuracy (42.92%), precision (14.37%), recall (2.87%), and F1-score (4.79%), reveal challenges such as high false-positive rates and low sensitivity to true anomalies. These findings highlight the potential of the algorithm for dynamic, high-dimensional datasets but also indicate the need for further improvements through hyperparameter tuning, feature engineering, and alternative approaches. This study contributes to the development of adaptive anomaly detection frameworks for network security and suggests future integration into real-time systems for proactive threat mitigation. The study's findings are particularly relevant for enhancing network security in environments such as corporate and governmental networks, where real-time anomaly detection is crucial.

Bayesian active learning isolation forest (B-ALIF): A weakly supervised strategy for anomaly detection

Casing deformation during hydraulic fracturing in unconventional reservoirs presents significant challenges, reducing well productivity, increasing costs, and restricting lateral access. While multi-finger calipers offer reliable diagnostics, they are often costly and typically used after issues arise. In contrast, the Casing Collar Locator (CCL) is routinely deployed, offering a cost-effective and scalable diagnostic alternative. This study evaluates the use of high-frequency CCL data for real-time anomaly detection, aiming to identify potential casing deformation earlier and proactively reduce the unstimulated lateral length with preventive actions. The goal is to give completion engineers timely insights that lower intervention costs and improve operational efficiency. The study proposes an end-to-end workflow including data ingestion, data processing, machine learning model deployment, and real-time visualizations. High-frequency CCL data (750Hz) is analyzed using an unsupervised anomaly detection algorithm based on isolation forests, eliminating the need for labeled datasets. The approach includes (1) validating the predictions with actual deformation events, (2) testing wireline data processing methods, and (3) applying downsampling algorithms. Techniques like frequency tuning, normalization, and downsampling (to 30–75Hz) improve efficiency without compromising resolution. A collar tagging method helps distinguish true deformation from collar-induced signals. CCL data is combined with stage and well context for accurate depth matching and real-time anomaly scoring. The developed Anomaly Detection Tool (ADT) has successfully identified casing deformations in multiple wells in a high-impact region. The model flagged deformation risks early, preventing operational disruptions. Notably, collar tagging enhances model precision by filtering collar-related signal noise. Higher resolution (0.01m sampling) sharpened anomaly resolution and improved detection. Overlaying the runs and anomalies enabled confirmation of casing degradation trends. Field testing included two phases: a shadow mode deployment where ADT findings supported contingency planning (e.g., slim plugs, longer stages) that enabled full lateral stimulation; and a live field test, where near real-time results informed operational decisions as deformation signals evolved, offering operators critical lead time for intervention planning. This study presents a scalable, real-time casing deformation detection system leveraging machine learning and ubiquitous CCL data without requiring additional tools or expensive sensors. The novelty lies in the use of an unsupervised learning approach, which performs well in the absence of labeled data and adapts across geologies and operational styles. Signal processing advancements, including collar tagging, further elevate the tool's reliability. This methodology was first developed in the Vaca Muerta formation and has since been validated across U.S. and Canadian basins, confirming its generalizability. Future work will focus on integrating supervised learning as labels mature to further improve precision and usability.

Traditional anomaly detection is mostly based on single-dimensional data for identification and analysis. The isolated forest algorithm is not a description of normal samples, but a field is divided by isolated anomalies. This paper introduces a multi-dimensional segmentation plane anomaly detection algorithm for isolated forest. By using multi-dimensional plane segmentation in the process of constructing hyperplane segmentation, and controlling the generation process of the isolated tree, the generation process of the isolated forest is optimized. By adding some common datasets for comparative experiments, good results have been achieved

In the rapidly emerging field of transferring healthcare from hospitals to patients' homes, it is essential that signs of remote deterioration at home should be detected early to timely upscale care when needed. As patient deterioration is a rare event, datasets have insufficient event data for supervised machine learning. Personalized anomaly detection (AD) may be a good alternative. Therefore, this study aimed to obtain insight into the use of personalized AD models in detecting early signs of patient deterioration at home using wearable sensor data. To address this, Isolation Forest and Local Outlier Factor models were applied to detect signs of deterioration in terms of mortality or unplanned readmission in the 24 preceding hours in two datasets: one of heterogeneous patients, the other of postoperative patients. A pipeline was developed for continuously updating personalized AD models over time and applying them to detection windows twice per day. Results were compared with the Remote Early Warning Score. Isolation Forest (AUROC: 0.69) and Local Outlier Factor (AUROC: 0.67) models were able to find some early signs of deterioration in heterogeneous patients (n = 113). However, in postoperative patients (n = 193), Isolation Forest (AUROC: 0.41) and Local Outlier Factor (AUROC 0.44) performed badly. The Remote Early Warning Score was able to find some early signs of deterioration for both groups (AUROC: 0.63-0.76). Based on these findings, three requirements were formulated that should be fulfilled for a potentially successful application of personalized AD. First, the training set should be normal and representative of non-deteriorating patients. Second, signs of deterioration should exhibit abnormal characteristics. Third, non-deteriorating patients should not exhibit abnormal characteristics.

Anomaly detection is a cornerstone of modern intelligent systems, with critical applications in healthcare, cybersecurity, smart grids, and IoT environments. While traditional machine learning and deep learning models have shown promise in identifying outliers, they often face challenges such as dependence on large labeled datasets, high computational costs, and limited scalability to edge devices or high-dimensional data streams. This study introduces D2H-AD, a novel anomaly detection framework built upon Hyperdimensional Computing (HDC), a brain-inspired paradigm that encodes information using high-dimensional distributed vectors. Unlike prior HDC-based approaches, D2H-AD fuses distance-based similarity and density-aware encoding in a hybrid design, significantly improving anomaly characterization and detection accuracy. Ablation experiments confirm that hyperdimensional encoding alone contributes up to 5.4% higher ROC-AUC compared to applying the same density-distance scoring in the original Euclidean feature space, and D2H-AD consistently outperforms all five published baselines (HDAD, ODHD, One-Class SVM, Isolation Forest, and Autoencoders) across every evaluated dataset. The method is designed to be lightweight, interpretable, and computationally efficient, suggesting strong suitability for deployment in resource-constrained and real-time environments. To validate its effectiveness, we evaluate D2H-AD on five diverse benchmark datasets, comparing its performance against five leading techniques: HDAD, ODHD, One-Class SVM, Isolation Forest, and Autoencoders. Our results show that D2H-AD consistently achieves superior F1 scores and ROC-AUC metrics, demonstrating robustness against class imbalance, noise, and data complexity. Beyond accuracy, D2H-AD offers practical advantages including scalability, minimal memory footprint, and an expected low-latency profile derived from its binary operations and lightweight design. These characteristics are crucial for TinyML and edge AI applications. This work highlights the untapped potential of HDC for high-performance anomaly detection and opens new avenues for secure, interpretable, and energy-efficient AI solutions in dynamic environments such as IoT, embedded systems, and beyond. Additionally, D2H-AD provides feature-level interpretability through hypervector decoding, enabling transparent explanations for safety-critical applications.