Neural Network Approach to Program Synthesis for Tabular Transformation by Example

Runoff predictions in new-gauged basins using two transformer-based models

In the realm of water resource management, meticulous monitoring and control methodologies are quintessential to the refinement of wastewater treatment processes. This research elucidates an avant-garde methodology for forecasting the Chemical Oxygen Demand (COD), an instrumental indicator of water quality, by harnessing the capabilities of long short-term memory (LSTM) neural networks in conjunction with Internet of Things (IoT) paradigms. The efficacy of the LSTM model is juxtaposed with that of an advanced Deep Belief Network (DBN), as well as contemporary models like a Convolutional Neural Network–Long Short-Term Memory (CNN-LSTM) hybrid model and a Transformer-based model, employing data sourced from a wastewater treatment facility located in Changsha. The empirical findings show that notwithstanding the comparable training durations used, the LSTM model exhibits a preeminent error rate of merely 7%, thus surpassing the DBN model (which has an error rate of 35%), the CNN-LSTM model (registering a 22% error rate), and the Transformer-based model (with a 17% error rate) in its predictive precision. This research underscores the potential of integrating an astute wastewater control system with IoT and LSTM models, thereby hinting at prospective enhancements in the sustainability and operational efficacy of wastewater treatment installations.

In this study, we propose a programming-by-example (PBE)-based data transformation method for feature engineering in machine learning. Data transformation by PBE is not new. However, we utilized the one proposed herein to improve the performance of machine learning in synthesizing a transformation rule from examples. Herein, the system first generates candidate rules, and then chooses the rule that achieves the highest performance in a target machine learning task. We tested this system with the Titanic dataset, and the result shows that the proposed method can avoid worst-case performance compared to the original PBE method.

To evaluate the diagnostic performance of a Transformer-based deep learning model that integrates real-world laboratory test indicators for differential diagnosis of ovarian cancer. The clinical data and 99 laboratory test indicators were retrospectively collected from patients with ovarian cancer and benign ovarian lesions admitted to Department of Obstetrics and Gynecology of Tongji Hospital between January 1, 2012 and April 4, 2021. A feature selection algorithm based on ANOVA F-test was used on the training set to identify 20 key features. Each case was then converted into a unified embedded vector using a tabular data Transformer. An improved stacked Transformer model was then trained to encode these feature vectors. The proposed model was compared with multiple traditional machine learning methods. The evaluation metrics included the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, and specificity. Five-fold cross-validation was performed to assess the generalization ability and robustness of the model. Five-fold cross-validation showed that the Transformer-based deep learning model achieved the best performance in predicting ovarian cancer with an AUC of 0.931, an accuracy of 0.813, a sensitivity of 0.833, and a specificity of 0.865. The proposed Transformer-based model demonstrates high accuracy and generalization capability in predicting ovarian cancer, and may thus offer a assistance in clinical diagnosis of ovarian tumors.

Lip reading, the process of interpreting speech by visually observing the movements of the lips, has emerged as a critical area of research with applications spanning communication aids for the hearing impaired, silent speech interfaces, and enhanced human-computer interaction. This paper reviews recent advancements in lip reading technologies, focusing on the integration of machine learning and computer vision techniques. We explore state-of-the-art methods including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer-based models that have significantly improved the accuracy and robustness of lip reading systems. The study highlights the importance of large annotated datasets, such as LipNet and LRW, which have facilitated the training of deep learning models. Additionally, we examine multimodal approaches that combine visual information with audio signals to enhance performance, especially in noisy environments. Despite substantial progress, challenges remain in addressing speaker variability, low resolution, and real-time processing. Future research directions are discussed, emphasizing the need for more diverse datasets, improved model generalization, and real-world application testing. This comprehensive review underscores the potential of advanced lip reading technologies to revolutionize communication accessibility and human-computer interaction. This paper presents the method for Vision based Lip Reading system that uses convolutional neural network (CNN) with attention-based Long Short-Term Memory (LSTM). The dataset includes video clips pronouncing words sentence. The pretrained CNN is used for extracting features from pre- processed video frames which then are processed for learning temporal characteristics by LSTM. The SoftMax layer of architecture provides the result of lip reading. In the present work experiments are performed with two pre-trained models. The system provides 80% accuracy using Tensorflow and ensemble learning. Keywords— CNN; RNN; LSTM; Tensorflow; lip reading; deep learning

Detection of breast cancer remains an important global health issue, particularly in patients with dense breast tissue. Despite wide use of machine learning for breast cancer diagnosis, many existing studies rely on single-model evaluations and lack rigorous statistical validation, while transformer-based models remain insufficiently explored for structured tabular biomedical data. Using the Wisconsin Breast Cancer Diagnostic (WBCD) dataset, this study presents a comparative analysis of machine learning, deep learning, and transformer-based models for binary classification of breast masses as benign or malignant. A robust preprocessing and feature engineering pipeline was applied, and Support Vector Machine (SVM), Random Forest (RF), Multi-Layer Perceptron (MLP), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and BERT-based models were evaluated using a stratified 66–34% train–test split and stratified 5-fold cross-validation. Performance was assessed using standard classification metrics, Jaccard Index, AUC-ROC, false negative rate, and statistical significance testing (paired t-tests and Bayesian analysis). Three-layer RNN and LSTM models achieved the highest accuracy of 98.25%, showing statistically significant improvement over SVM (p = 0.042), while the optimized SVM demonstrated strong performance with superior computational efficiency. BERT-based models yielded lower accuracy (92.98%), reflecting domain mismatch between language-pretrained transformers and numerical tabular data. These findings indicate that deeper recurrent architectures provide superior diagnostic performance, while SVM remains a practical and efficient alternative for clinical deployment.

Depression detection in low-resource languages, such as Bangla, is an emerging challenge in mental health research due to the limited availability of labeled datasets and NLP resources. Bangla, the seventh most spoken language globally, is used by over 230 million people, yet it remains underrepresented in natural language processing (NLP) research compared to high-resource languages like English. In this study, we present a Bangla depression detection dataset collected from social media comprising 18,612 text samples, with 8556 labeled as depressive and the remaining as non-depressive. To effectively classify depressive content, we explore multiple deep learning architectures, including Long Short-Term Memory (LSTM), Bidirectional LSTM (BiLSTM), Gated Recurrent Unit (GRU), and Bidirectional GRU (BiGRU). Additionally, we evaluate state-of-the-art transformer-based models such as BanglaBERT, BanglaBERT-Base, ELECTRA, RoBERTa, and BERT. Machine learning models are also employed as baselines for comparison. Our experimental results show that transformer-based models outperform both deep learning and machine learning approaches, with BanglaBERT achieving the highest accuracy of 89.23%, followed closely by BiLSTM with 88%. These findings highlight the effectiveness of transformer-based architectures in depression detection for Bangla text, contributing to improved mental health analysis in low-resource language settings. This study emphasizes the need for further NLP advancements in Bangla to develop AI-driven solutions for early depression detection and intervention.

In time series stock forecasting tasks, Long Short-Term Memory (LSTM) networks have been extensively utilized due to their proficiency in modeling sequential data. However, LSTM-based methods often struggle to capture long-range dependencies inherent in complex datasets. To address this limitation, Transformer-based models, such as Informer, have been introduced, leveraging attention mechanisms to better capture these long-term dependencies. Nevertheless, these models introduce increased computational complexity and encounter challenges in effectively modeling local dependencies. Additionally, stock market dynamics are heavily influenced by global financial news, necessitating the integration of sentiment data to enhance prediction accuracy. In this paper, we approach stock price prediction as a time series forecasting task by developing a hybrid model that integrates traditional time series forecasting methods (LSTM), attention mechanisms (Informer), and natural language processing techniques (FinBERT). Utilizing preprocessed historical stock price data from the Dow Jones Industrial Average and sentiment data from The Wall Street Journal, we conducted comparative experiments to demonstrate the superior performance of our hybrid model in predicting stock prices.

The research study is to predict human activity recognition and analyze their strength in machine learning (ML) processes to achieve precision and also to determine performance of human activities. The framework to classify the Novel Long Short Term Memory (LSTM) and Decision Tree (DT) to perform all measures. This research study used the Novel Long Short Term Memory and Decision Tree to perform the required actions to accurately identify body movements in adjacent film sequence. Recent study findings and a threshold of 0.05%, as well as 95% confidence interval and standard parameter estimates, were castoff in a study 47 samples divided into two groups calculated using the G-power of 80%. Since the novel Long Short Term Memory (LSTM) algorithm and local invariant methods have achieved an accuracy of 93.83% in predicting the activity analysis, this research needs to obtain improved accuracy to perform activity prediction by using the Decision Tree algorithm. When applied to human activity analysis, the Novel Long Short Term Memory technique was shown to have an accuracy of 90.12 percent, with a 95% confidence interval including the value 0.045 (p 0.05). Researchers found that when comparing two algorithms for analyzing human behaviour, Long Short Term Memory (LSTM) performed much better than Decision Tree (DT).

BackgroundDiabetic retinopathy (DR) is a leading cause of blindness in American adults. If detected, DR can be treated to prevent further damage causing blindness. There is an increasing interest in developing artificial intelligence (AI) technologies to help detect DR using electronic health records. The lesion-related information documented in fundus image reports is a valuable resource that could help diagnoses of DR in clinical decision support systems. However, most studies for AI-based DR diagnoses are mainly based on medical images; there is limited studies to explore the lesion-related information captured in the free text image reports.MethodsIn this study, we examined two state-of-the-art transformer-based natural language processing (NLP) models, including BERT and RoBERTa, compared them with a recurrent neural network implemented using Long short-term memory (LSTM) to extract DR-related concepts from clinical narratives. We identified four different categories of DR-related clinical concepts including lesions, eye parts, laterality, and severity, developed annotation guidelines, annotated a DR-corpus of 536 image reports, and developed transformer-based NLP models for clinical concept extraction and relation extraction. We also examined the relation extraction under two settings including ‘gold-standard’ setting—where gold-standard concepts were used–and end-to-end setting.ResultsFor concept extraction, the BERT model pretrained with the MIMIC III dataset achieve the best performance (0.9503 and 0.9645 for strict/lenient evaluation). For relation extraction, BERT model pretrained using general English text achieved the best strict/lenient F1-score of 0.9316. The end-to-end system, BERT_general_e2e, achieved the best strict/lenient F1-score of 0.8578 and 0.8881, respectively. Another end-to-end system based on the RoBERTa architecture, RoBERTa_general_e2e, also achieved the same performance as BERT_general_e2e in strict scores.ConclusionsThis study demonstrated the efficiency of transformer-based NLP models for clinical concept extraction and relation extraction. Our results show that it’s necessary to pretrain transformer models using clinical text to optimize the performance for clinical concept extraction. Whereas, for relation extraction, transformers pretrained using general English text perform better.

Financial markets are greatly affected by how investors feel, big global events, overall economic trends, and how people behave. As digital content has grown quickly, the emotions expressed in news, social media, and financial forums have become a key part of predicting how stock prices will move. This research introduces a combined system that analyzes financial market sentiment and predicts prices, using Natural Language Processing (NLP), Machine Learning (ML), and Time-Series Analysis. The system gathers historical market data and text-based sentiment information.It uses tools like VADER, Text Blob, and Transformer-based sentiment models to process this data. To predict market direction, it applies algorithms such as Long Short-Term Memory (LSTM), Random Forest Regression, and Autoregressive Integrated Moving Average (ARIMA). Testing this system on stock market data shows that including sentiment features with numerical price data greatly improves prediction accuracy compared to methods that only use price data. This study shows that models that take sentiment into account better understand market psychology, making financial forecasting systems more reliable, consistent, and effective for short-term predictions

Battery Energy Storage Systems (BESS) are critical to the global transition away from fossil fuels, particularly in the transportation and energy sectors. However, maximizing their potential requires accurate forecasting of battery behavior under diverse and dynamic operating conditions. Traditional Battery Management Systems (BMS) estimate the State of Charge (SoC) based on real-time sensor readings but are primarily designed for monitoring and protection rather than forecasting or planning purposes. Using a BMS alone for predictive tasks or long-term energy planning would demand significant modeling effort and computational resources, especially under complex and variable load scenarios. To address this challenge, this study investigates the use of deep learning frameworks specifically Long Short-Term Memory (LSTM) networks and Transformer architectures for SoC estimation based on real-world electric vehicle (EV) charging and discharging data. While most existing models rely on instantaneous measurements of voltage, current, temperature, and time, this work introduces Coulomb counting values as an additional input. By integrating the accumulated charge over time, Coulomb counting offers a physically grounded perspective on battery usage, enabling the models to capture long-term dependencies more effectively. Two deep learning models were developed and evaluated, an LSTM-based model and a Transformer-based model, each tested with two configurations: (1) with standard inputs only, serving as the benchmark; and (2) with standard inputs plus Coulomb counting included as an additional feature.

Detecting multi-word expressions (MWEs) is essential in natural language processing (NLP) for machine translation, sentiment analysis, and information retrieval. We evaluated the performance of different deep learning models on MWE detection using samples from the major consumer electronics product retailer corpus. We explored transformer-based models such as bidirectional encoder representations from transformers (BERT), BERT-Conditional Random Field (CRF), and robustly optimized BERT variant (RoBERTa)-CRF, alongside a recurrent neural network-based model such as long short-term memory (LSTM)-CRF, using a sample of 10,000 rows, with the CRF layer incorporated in model variations. The transformer-based model with a CRF layer, RoBERTa-CRF, showed the best performance by leveraging the contextual embeddings and sequential dependency modeling. The study results underscore the importance of integrating contextual and sequential deep learning techniques to improve MWE detection and provide a basis for developing robust and scalable systems for diverse linguistic tasks.

When analyzing a sequence of customer interactions, it is important for firms to understand how these interactions align with key objectives, such as generating qualified customer leads, driving conversion events, or reducing churn. The authors introduce a transformer-based framework that models customer interactions in a sequence similar to how a sentence is modeled as a sequence of words by large language models. They propose a heterogeneous-mixture multihead self-attention mechanism that captures individual heterogeneity in touchpoint effects. The model identifies self-attention patterns that reflect both population-level trends and the unique relationships between touchpoints within each customer journey. By assigning varying weights to each attention head, the model accounts for the distinctive aspects of the journey of each user. This results in more accurate predictions, enabling precise targeting and outperforming existing approaches such as hidden Markov models, point process models, and long short-term memory (LSTM) models. This empirical application in a multichannel marketing context demonstrates how managers can leverage the model's features to identify high-potential customers for targeting. Extensive simulations further establish the model's superiority over competing approaches. Beyond multichannel marketing, the transformer-based model also has broad applicability in customer journeys across other domains.

Abstract—Alzheimer disease (AD) is a progressive neurodegen- erative disorder, treatment and management of which needs to be timely. Conventional diagnosis techniques such as MRI, PET, and cerebrospinal fluid test are mostly invasive, costly, or not available at large scale. On the other hand, electroencephalography (EEG) represents a non-invasive, low-cost, and portable method of brain monitoring, hence, making it a favorable modality to use to screen AD at an early onset. The article contains a deep neural network framework of automatic EEG-based Alzheimer disease classification. Raw EEG data were processed and transformed to time-frequency representation and connection matrices. The different architectures (convolutional neural networks (CNNs), hybrid CNN-LSTM models as well as transformer-based models) were trained and compared with the conventional machine learning-based models (support vector machines and random forests). The deep neural models were found to be more effective than the traditional methods with improved accuracy, sensitiv- ity and specificity in distinguishing between the patients with AD and mild cognitive impairment (MCI) and those without mental illnesses. On top of highlighting frequency bands and areas that are correlated with known AD biomarkers, which is important to validate the clinical relevance of the finding, model interpretability methods, such as Grad-CAM, also focused on frequency bands and areas that are correlated with known AD biomarkers. These findings indicate that EEG-based deep neural models can be used practically to scalably and early diagnose the presence of Alzheimer as not only deep learning using EEG signals improves diagnostic accuracy but also provides biologically relevant outcomes. Index Terms—Alzheimer’s disease, Electroencephalography (EEG), Deep learning, Neural networks, Convolutional neural networks (CNN), Long short-term memory (LSTM), Trans- formers, Mild cognitive impairment (MCI), Early diagnosis, Brain–computer interface (BCI), Medical signal processing.