Dynamic Record Linking Using Multi-Agent Machine Learning: An Architecture for Noisy and Variable Datasets

Seismic data denoising is a critical component of seismic data processing, yet effectively removing erratic noise, characterized by its non-Gaussian distribution and high amplitude, remains a substantial challenge for conventional methods and deep-learning (DL) algorithms. Supervised learning frameworks typically outperform others, but they require pairs of noisy data sets alongside corresponding clean ground truth, which is impractical for real-world seismic data sets. In contrast, unsupervised learning (UL) methods, which do not rely on ground truth during training, often fall short in performance when compared with their supervised or traditional denoising counterparts. Moreover, current unsupervised DL methods fail to address the specific challenges posed by erratic seismic noise adequately. This paper introduces a novel zero-shot unsupervised DL framework designed specifically to mitigate random and erratic noise, with a particular emphasis on blended noise. Drawing inspiration from Noise2Noise (N2N) and data augmentation principles, we develop a robust self-supervised denoising network called robust Noiser2Noiser. Our approach eliminates the need for paired noisy and clean data sets as required by supervised methods or paired noisy data sets as in N2N. Instead, our framework relies solely on the original noisy seismic data set. Our methodology generates two independent recorrupted data sets from the original noisy data set, using one as the input and the other as the training target. Subsequently, we use a DL-based denoiser, denoising convolutional neural network, for training purposes. To address various types of random and erratic noise, the original noisy data set is recorrupted with the same noise type. Detailed explanations for generating training input and target data for blended data are provided. We apply our network to synthetic and real marine data examples, demonstrating significantly improved noise attenuation performance compared with traditional denoising methods and state-of-the-art UL Codes are available on https://github.com/Ji-seismic/N2N_deblending.

We investigated the estimation of the ensemble channelized Hotelling observer (CHO) signal-to-noise ratio (SNR) for ordered-subset (OS) image reconstruction using noisy projection data. Previously, we computed the ensemble CHO SNR using a method for approximating the channelized covariance of OS reconstruction, which requires knowledge of the noise-free projection data. Here, we use a “plug-in” approach, in which noisy data is used in place of the noise-free data in the aforementioned channelized covariance approximation. Additionally, we evaluated the use of smoothing of the noisy projections before use in the covariance approximation. Additionally, we evaluated the use of smoothing of the noisy projections before use in the covariance calculation. The task was detection of a 10% contrast Gaussian signal within a slice of the MCAT phantom. Simulated projections of the MCAT phantom were scaled and Poisson noise was added to create 100 noisy signal-absent data sets. Simulated projections of the scaled signal were then added to the noisy background projections to create 100 noisy signal-present data set. These noisy data sets were then used to generate 100 estimates of the ensemble CHO SNR for reconstructions at various iterates. For comparison purposes, the same calculation was repeated with the noise-free data. The results, reported as plots of the average CHO SNR generated in this fashion, along with 95% confidence intervals, demonstrate that this approach works very well, and would allow optimization of imaging systems and reconstruction methods using a more accurate object model (i.e., real patient data).

Accurate blood glucose level prediction is crucial for effective diabetes management, enabling timely interventions and reducing complications. The limitations of traditional machine learning models like Random Forest and XGBOOST include the inability to deal with noisy data and datasets with imbalances. This study proposes a hybrid ensemble model that combines Random Forest for robust feature selection and initial predictions with XGBOOST to improve accuracy by focusing on instances that have been misclassified. The hybrid approach improves the handling of imbalanced and noisy datasets, achieving better accuracy and generalization. The model's superiority over standalone machine learning models is demonstrated by experimental results on publicly accessible datasets, highlighting its potential as a reliable method for predicting blood glucose levels and supporting diabetes management.

In cluster analysis, distinguishing abnormal behavior from noise is an important problem that has remained unresolved for many years. This paper presents a general approach for distinguishing abnormal behavior from noise and for identifying abnormal behavior in noisy data sets. The approach is an application of a new, complete linkage hierarchical clustering method for n·(n−1) over 2 + 1-level hierarchical sequences and a means for finding meaningful levels of such hierarchical sequences prior to performing a cluster analysis. These technologies were designed with small-n, large-m data sets in mind. Based on four broadly applicable assumptions, the approach uses higher dimensionalities to reveal inherent structure in noisy data sets and find meaningful levels in the corresponding hierarchical sequences. The new clustering method is used to construct only the cluster sets that correspond to these levels. Results from a first experiment show how the effects of noise are attenuated as the dimensionality of the data points increases. Results from a second experiment show how meaningful cluster sets can have real world meanings that are useful for identifying abnormal behavior.

Visual analysis has witnessed a growing acceptance as a method of scientific inquiry in the research community. It is used in qualitative and mixed research methods. Even so, visual data analysis is likely to produce biased results when used in analysing a large and noisy dataset. This can be evident when a data analyst is not able to holistically explore, all the values associated with the objects of interest in a dataset. Consequently, the data analyst may assess inconsistent data as consistent when contradiction associated with the data is not visualised. This work identifies incomplete analysis as a challenge in the visual data analysis of a large and noisy dataset. It considers Formal Concept Analysis (FCA) tools and techniques and prescribes the mining and visualisation of Incomplete or Inconsistent Data (IID) when dealing with a large and noisy dataset. It presents an automated approach for transforming IID from a noisy context whose objects are associated with mutually exclusive many-valued attributes, to a formal context.

Chemical heuristics have been fundamental to the advancement of chemistry and materials science. These heuristics are typically established by scientists using knowledge and creativity to extract patterns from limited datasets. Machine learning offers opportunities to perfect this approach using computers and larger datasets. Here, we discuss the relationships between traditional heuristics and machine learning approaches. We show how traditional rules can be challenged by large-scale statistical assessment and how traditional concepts commonly used as features are feeding the machine learning techniques. We stress the waste involved in relearning chemical rules and the challenges in terms of data size requirements for purely data-driven approaches. Our view is that heuristic and machine learning approaches are at their best when they work together.

Machine learning is an artificial intelligence division that utilizes various computational, probabilistic, and optimization techniques that enable computers to “read” from previous examples and to identify trends that cannot be discerned from large, noisy, or complex datasets. This is particularly suitable for medical applications, particularly those dependent on complex proteomic and genomic measurements. The primary tumor is the deadliest disease, with a high mortality rate. Machine learning is therefore frequently used in the diagnosis and detection of primary tumors. Machine learning was more recently used to forecast and predict primary tumor diagnosis. The classification of the primary tumor may be defined with different machine learning algorithms, given the extreme impact of the disorder. Logistic regression, SVM, random forest, AdaBoost node, Naive Bayes, K-neighbor grouping, decision tree, and the Gaussian system classifiers are discussed. The dimensional reduction is implemented to simplify the dataset to reduce the measurement time. The objective of this paper is to propose an ensemble model for the prediction of the primary tumor.

Machine learning is a branch of artificial intelligence that employs a variety of statistical, probabilistic and optimization techniques that allows computers to “learn” from past examples and to detect hard-to-discern patterns from large, noisy or complex data sets. As a result, machine learning is frequently used in cancer diagnosis and detection. In this paper, support vector machines, K-nearest neighbours and probabilistic neural networks classifiers are combined with signal-to-noise ratio feature ranking, sequential forward selection-based feature selection and principal component analysis feature extraction to distinguish between the benign and malignant tumours of breast. The best overall accuracy for breast cancer diagnosis is achieved equal to 98.80% and 96.33% respectively using support vector machines classifier models against two widely used breast cancer benchmark datasets.

The proliferation of DNA sequence data has generated a concern about the effects of “noise” on phylogeny reconstruction. This concern has led to various recommendations for weighting schemes and for separating data types prior to analysis. A new technique is explored to examine directly how noise influences the stability of parsimony reconstruction. By appending purely random characters onto a matrix of pure signal, or by replacing characters in a matrix of signal by random states, one can measure the degree to which a matrix is robust against noise. Reconstructions were sensitive to tree topology and clade size when noise was added, but were less so when character states were replaced with noise. When a signal matrix is complemented with a noise matrix of equal size, parsimony will trace the original signal about half the time when there is only one synapomorphy per node, and about 90% of the time when there are three synapomorphies per node. Similar results obtain when 20% of a matrix is replaced by noise. Successive weighting does not improve performance. Adding noise to only some taxa is more damaging, but replacing characters in only some taxa is less so. The bootstrap and g1 (tree skewness) statistics are shown to be uninterpretable measures of noise or departures from randomness. Empirical data sets illustrate that commonly recommended schemes of differential weighting (e.g. downweighting third positions) are not well supported from the point of view of reducing the influence of noise nor are more noisy data sets likely to degrade signal found in less noisy data sets.

Sales trend forecasting is a critical function within modern business intelligence (BI) systems, enabling organizations to optimize inventory management, allocate resources effectively, and make strategic decisions in an increasingly volatile market. Traditionally, classical time series models such as ARIMA and Exponential Smoothing have been widely used due to their interpretability and robust theoretical foundations. However, the emergence of machine learning (ML) methods such as Random Forests, Gradient Boosting Machines, and Long Short-Term Memory (LSTM) networks has introduced new opportunities for capturing complex, non-linear patterns in sales data. This review provides a comprehensive comparison between classical and machine learning approaches for sales trend prediction, examining their strengths, limitations, and practical applications. Classical models offer simplicity, computational efficiency, and strong performance with stationary and linear data, making them suitable for well-structured datasets. Conversely, machine learning models excel in handling large, noisy, and multi-dimensional datasets, offering superior accuracy at the cost of higher computational demands and reduced interpretability. Real-world applications across retail, finance, supply chain management, and healthcare are explored, highlighting the transformative impact of time series forecasting in business operations. Key challenges including data quality, model maintenance, and the need for explainable AI are discussed alongside future directions such as real-time forecasting, transfer learning, and federated learning. By synthesizing insights from both classical and contemporary forecasting paradigms, this review aims to guide researchers, data scientists, and business leaders in selecting appropriate methodologies for enhancing predictive capabilities within business intelligence ecosystems.

Chapter 19 - Statistical (Machine) Learning: Using Complex Models With Large Data Sets

Hand joints-based gesture recognition using a neural network provides excellent performance in hand gesture recognition. However, during the collection of sequential skeletal datasets, joints identification through hand pose estimations usually includes noise and even errors, which often diminish the accuracy of gesture recognition. To promote the availability of hand gesture recognition for such noisy datasets, this paper presents a nested interval unscented Kalman filter (UKF) with long short-term memory (NIUKF-LSTM) network to improve the accuracy of hand gesture recognition from noisy datasets. This nested interval method with the UKF changes the distribution of the sigma points based on two sampling intervals. By considering the information of previous frames, the nested interval method helps the NIUKF-LSTM network revise the noise in the sequential hand skeletal data and improve the recognition accuracy. The experimental results showing the removal of noisy skeletal data from the dynamic hand gesture dataset demonstrate the effectiveness of our NIUKF-LSTM network, which achieves better performance than do other state-of-the-art methods.

Two decades of U.S. government legislative outcomes, as well as the policy preferences of high-income people, the general population, and diverse interest groups, were captured in a detailed dataset curated and analyzed by Gilens, Page et al. (2014). They found that the preferences of high-income earners correlated strongly with policy outcomes, while the preferences of the general population did not, except via a linkage with the preferences of high earners. Their analysis applied the tools of classical statistical inference, in particular logistic regression. In this paper we analyze the Gilens dataset using the complementary tools of Random Forest classifiers (RFs), from Machine Learning. We present two primary findings, concerning respectively prediction and inference: (i) Holdout test sets can be predicted with approximately 70% balanced accuracy by models that consult only the preferences of those in the 90th income percentile and a small number of powerful interest groups, as well as policy area labels. These results include retrodiction, where models trained on pre-1997 cases predicted “future” (post-1997) cases. The 20% gain in accuracy over baseline (chance), in this detailed but noisy dataset, indicates the high importance of a few distinct players in U.S. policy outcomes, and aligns with a body of research indicating that the U.S. government has significant plutocratic tendencies. (ii) The feature selection methods of RF models identify especially salient subsets of interest groups (economic players). These can be used to further investigate the dynamics of governmental policy making, and also offer an example of the potential value of RF feature selection methods for inference on datasets such as this one.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

This paper explores the application of active learning to enhance machine learning classifiers for spoofing detection in automatic speaker verification (ASV) systems. Leveraging the ASVspoof 2019 database, we integrate an active learning framework with traditional machine learning workflows, specifically focusing on Random Forest (RF) and Multilayer Perceptron (MLP) classifiers. The active learning approach was implemented by initially training models on a small subset of data and iteratively selecting the most uncertain samples for further training, which allowed the classifiers to refine their predictions effectively. Experimental results demonstrate that while the MLP initially outperformed RF with an accuracy of 95.83% compared to 91%, the incorporation of active learning significantly improved RF's performance to 94%, narrowing the performance gap between the two models. After applying active learning, both classifiers showed enhanced precision, recall, and F1-scores, with improvements ranging from 3% to 5%. This study provides valuable insights into the role of active learning in boosting the efficiency of machine learning models for dynamic spoofing scenarios in ASV systems. Future research should focus on designing advanced active learning techniques and exploring their integration with other machine learning paradigms to further enhance ASV security.