Device-Cloud Collaborative Correction for On-Device Recommendation

Federated learning (FL) is gradually gaining traction as the de facto standard for distributed recommendation model training that takes advantage of on-device user data while reducing server costs. However, the computation resources of user devices in FL are usually much more limited compared to servers in a datacenter, which hinders the application of some advanced recommendation models (e.g., Transformer-based models) in FL. In addition, models with better recommendation performance tend to have more parameters, which increases the cost of communication between servers and user devices. Therefore, it is difficult for existing federated recommendation methods to achieve a good trade-off between recommendation accuracy and computation and communication costs. As a response, we propose a novel federated recommendation framework for efficient recommendations. First, we propose an all-MLP model by replacing the self-attention sublayer in a Transformer encoder with a Fourier sublayer, in which the noise information in the user interaction data is effectively attenuated using Fast Fourier Transform and learnable filters. Second, we adopt an adaptive model pruning technique in the FL framework, which can significantly reduce the model size without affecting the recommendation performance. Extensive experiments on four real-world datasets demonstrate that our method outperforms existing federated recommendation methods and strikes a good trade-off between recommendation performance and model size.KeywordsRecommender systemSequential recommendationFederated recommendationFederated learningAll-MLP model

Tropical cyclones (TCs) are destructive meteorological phenomena, necessitating accurate predictions of TC track and intensity to reduce risks to human life. This study evaluates three Transformer-based models – vanilla Transformer (Transformer), inverted Transformer (iTransformer), and temporal-variate Transformer (TVFormer) – which are trained, validated, and tested on best track data from 1980 to 2021 from the China Meteorological Administration for TC prediction, integrating temporal, variate, and hybrid token embeddings to analyze temporal and variate correlations. Comparative analysis with four recurrent neural network (RNN) models demonstrates the superiority of the refined Transformer models over RNNs: iTransformer reduces mean absolute error (MAE) and root mean square error (RMSE) by 29.55% and 25.80% (latitude), 50.31% and 46.18% (longitude), 8.71% and 9.98% (pressure), and 8.68% and 9.45% (wind speed), while TVFormer achieves MAE and RMSE reductions of 13.98% and 13.84% (latitude), 39.11% and 38.02% (longitude), 13.69% and 14.02% (pressure), and 12.84% and 12.94% (wind speed) on average. Among Transformer variants, iTransformer excels in track prediction, outperforming Transformer with 21.74% lower MAE and 18.26% lower RMSE for latitude, and 32.73% lower MAE and 24.01% lower RMSE for longitude. TVFormer dominates intensity prediction, reducing pressure errors by 4.42% (MAE) and 3.92% (RMSE) and wind speed errors by 19.21% (MAE) and 14.79% (RMSE) compared to Transformer, while outperforming iTransformer with 4.59% lower MAE and 3.68% lower RMSE for pressure and 3.83% lower MAE and 3.18% lower RMSE for wind speed. Notably, TVFormer also enhances track prediction, with 7.10% reduction in MAE and 7.02% reduction in RMSE for latitude, and 22.84% reduction in MAE and 17.09% reduction in RMSE for longitude compared to Transformer. These results highlight the superiority of iTransformer in track prediction and the efficacy of TVFormer in intensity prediction, thanks to their ability to exploit temporal and variate dependencies, offering potential for TC disaster preparedness systems.

Question answering (QA) Transformer-based models might become efficient in inclusive education. For example, one can test and tune such models with small closed-domain datasets before the implementation of a new system in an inclusive organization. However, studies in the sociomedical domain show that such models can be unpredictable. They can mislead a user or evoke aversive emotional states. The paper addresses the problem of investigating safety-first QA models that would generate user-friendly outputs. The study aims to analyze the performance of SOTA Transformer-based QA models on a custom dataset collected by the author of the paper. The dataset contains 1 134 question-answer pairs about autism spectrum disorders (ASD) in Russian. The study presents the validation and evaluation of extractive and generative QA models. The author used transfer learning techniques to investigate domain-specific QA properties and suggest solutions that might provide higher QA efficiency in the inclusion. The study shows that although generative QA models can misrepresent facts and generate false tokens, they might bring diversity in the system outputs and make the automated QA more user-friendly for younger people. Although extractive QA is more reliable, according to the metric scores presented in this study, such models might be less efficient than generative ones. The principal conclusion of the study is that a combination of generative and extractive approaches might lead to higher efficiency in building QA systems for inclusion. However, the performance of such combined systems in the inclusion is yet to be investigated.KeywordsQuestion answeringDialogue systemTransformer

Abstract— Audio data carries rich information in the form of speech, music, and environmental sounds, but its raw waveform is often complex and high-dimensional, making direct analysis difficult. Neural network architectures have emerged as powerful tools for extracting meaningful representations from audio signals, enabling efficient analysis and interpretation. Recurrent Neural Networks (RNNs), and Transformer-based models— for learning robust and discriminative features from audio. By automatically capturing temporal, spectral, and contextual patterns, these architectures significantly improve performance in tasks such as speech recognition, speaker identification, music classification, and environmental sound detection. The findings highlight the potential of neural networks to replace traditional handcrafted features, thereby advancing the development of scalable, accurate, and realtime audio processing applications. The rapid growth of audio data across domains such as speech, music, healthcare, and environmental monitoring has created a strong need for effective methods to extract meaningful representations from complex audio signals. Traditional approaches rely on handcrafted features like MFCCs and spectrogram descriptors, which often fail to capture the full temporal and spectral dynamics present in raw audio. Keywords: Neural Networks, Deep Learning, Audio Representation Learning, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformer Models, Feature Extraction, Speech Recognition, Speaker Identification, Sound Event Detection, Spectrogram Analysis, Audio Signal Processing

Encouraged by the success of pretrained Transformer models in many natural language processing tasks, their use for International Classification of Diseases (ICD) coding tasks is now actively being explored. In this study, we investigated two existing Transformer-based models (PLM-ICD and XR-Transformer) and proposed a novel Transformer-based model (XR-LAT), aiming to address the extreme label set and long text classification challenges that are posed by automated ICD coding tasks. The Transformer-based model PLM-ICD, which currently holds the state-of-the-art (SOTA) performance on the ICD coding benchmark datasets MIMIC-III and MIMIC-II, was selected as our baseline model for further optimisation on both datasets. In addition, we extended the capabilities of the leading model in the general extreme multi-label text classification domain, XR-Transformer, to support longer sequences and trained it on both datasets. Moreover, we proposed a novel model, XR-LAT, which was also trained on both datasets. XR-LAT is a recursively trained model chain on a predefined hierarchical code tree with label-wise attention, knowledge transferring and dynamic negative sampling mechanisms. Our optimised PLM-ICD models, which were trained with longer total and chunk sequence lengths, significantly outperformed the current SOTA PLM-ICD models, and achieved the highest micro-F1 scores of 60.8 % and 50.9 % on MIMIC-III and MIMIC-II, respectively. The XR-Transformer model, although SOTA in the general domain, did not perform well across all metrics. The best XR-LAT based models obtained results that were competitive with the current SOTA PLM-ICD models, including improving the macro-AUC by 2.1 % and 5.1 % on MIMIC-III and MIMIC-II, respectively. Our optimised PLM-ICD models are the new SOTA models for automated ICD coding on both datasets, while our novel XR-LAT models perform competitively with the previous SOTA PLM-ICD models.

When long-term user proles are not available, session-based recommendation methods are used to predict the user's next actions from anonymous sessions-based data. Recent advances in session-based recommendation highlight the necessity of modeling not only user sequential behaviors but also the user's main interest in a session, while avoiding the eect of unintended clicks causing interest drift of the user. In this work, we propose a Dual Transformer Encoder Recommendation model (DTER) as a solution to address this requirement. The idea is to combine the following recipes: (1) a Transformer-based model with dual encoders capable of modeling both sequential patterns and the main interest of the user in a session; (2) a new recommendation model that is designed for learning richer session contexts by conditioning on all permutations of the session prex. This approach provides a unied framework for leveraging the ability of the Transformer's self-attention mechanism in modeling session sequences while taking into account the user's main interest in the session. We empirically evaluate the proposed method on two benchmark datasets. The results show that DTER outperforms state-of-the-art session-based recommendation methods on common evaluation metrics.

Spam detection is an evolving issue in line with the increasing volume of data and the evolution of spam techniques. In recent years, the application of machine learning (ML) algorithms has become an effective solution to enhance the accuracy and efficiency of spam detection systems. This study aims to analyze various machine learning algorithms applied in spam detection systems through a literature review. Several popular algorithms used in spam detection include Naive Bayes, Support Vector Machine (SVM), Neural Network, Recurrent Neural Network (RNN), and Transformer-based models. Each algorithm has its strengths and weaknesses that affect its performance in handling spam detection issues, depending on the characteristics of the data and the application requirements. Based on the data obtained, the Naive Bayes algorithm achieved 88% accuracy, 85% precision, 90% recall, and 87% F1-score. In contrast, SVM showed higher results with 93% accuracy, 92% precision, 94% recall, and 93% F1-score. Neural Network reached 96% accuracy, 95% precision, 97% recall, and 96% F1-score, while Recurrent Neural Network (RNN) achieved 95% accuracy, 94% precision, 96% recall, and 95% F1-score. Transformer-based models provided the best results with 97% accuracy, 96% precision, 98% recall, and 97% F1-score. This study adopts a literature analysis method by reviewing various articles and research that discuss the application of these algorithms in spam detection. In conclusion, the selection of the appropriate algorithm should be adjusted to the characteristics of the dataset, the complexity of the problem, and the availability of computational resources, as each algorithm has its own strengths and weaknesses in the context of spam detection.

Skin cancer is the most common type of cancer, a life-threatening condition that leads to serious health problems if not detected early, and its incidence is increasing worldwide. In recent years, computer vision and decision support systems have been used for disease detection in dermoscopic images. Furthermore, it has been observed that data representation methods affect the detection performance of these models, and the effect of color information on transformer-based models has not been sufficiently investigated. This study used the International Skin Imaging Collaboration (ISIC) 2017 dataset consisting of RGB images, and these images were converted into the HSV, LAB, and YCbCr color spaces. Transformer-based models, including visual transformer (ViT), swin transformer, data efficient image transformer (DeiT), and label-free self-distillation (DINO), were used for benign and malignant classification. According to the classification performance results, RGB and HSV color spaces particularly in the DeiT and Swin models, stable and high accuracy values ​​were obtained. It was observed that the ViT and DINO models were more sensitive to color space transformations and achieved lower classification performance compared to other models. The highest performance in skin cancer classification was achieved with the DeiT model trained in the RGB color space, with the highest accuracy (0.7668). Furthermore, the explainability-based gradient-weighted class activation mapping (Grad-CAM) method was used to analyze where the models focused in image regions when making classification decisions. This study shows the effect and usability of color space transformations in transformer-based models for skin cancer classification and offers a comparative contribution to the literature.

Lip reading, the process of interpreting speech by visually observing the movements of the lips, has emerged as a critical area of research with applications spanning communication aids for the hearing impaired, silent speech interfaces, and enhanced human-computer interaction. This paper reviews recent advancements in lip reading technologies, focusing on the integration of machine learning and computer vision techniques. We explore state-of-the-art methods including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer-based models that have significantly improved the accuracy and robustness of lip reading systems. The study highlights the importance of large annotated datasets, such as LipNet and LRW, which have facilitated the training of deep learning models. Additionally, we examine multimodal approaches that combine visual information with audio signals to enhance performance, especially in noisy environments. Despite substantial progress, challenges remain in addressing speaker variability, low resolution, and real-time processing. Future research directions are discussed, emphasizing the need for more diverse datasets, improved model generalization, and real-world application testing. This comprehensive review underscores the potential of advanced lip reading technologies to revolutionize communication accessibility and human-computer interaction. This paper presents the method for Vision based Lip Reading system that uses convolutional neural network (CNN) with attention-based Long Short-Term Memory (LSTM). The dataset includes video clips pronouncing words sentence. The pretrained CNN is used for extracting features from pre- processed video frames which then are processed for learning temporal characteristics by LSTM. The SoftMax layer of architecture provides the result of lip reading. In the present work experiments are performed with two pre-trained models. The system provides 80% accuracy using Tensorflow and ensemble learning. Keywords— CNN; RNN; LSTM; Tensorflow; lip reading; deep learning

In this paper, we address the challenge of accurate full-chip power mapping for commercial off-the-shelf CPU and GPU processors, explicitly considering temperature dependence. It is well known that both dynamic and leakage power are strongly temperature-dependent; however, existing power estimation methods for real chips often neglect this critical factor. To mitigate this, we characterize temperature-dependent spatial power maps for the first time on commercial processors, including the AMD Radeon RX 6400 GPU and Qualcomm Snapdragon 680 (SM6225) CPU. Using a back-side cooling infrared (IR) thermal imaging system, we capture full-chip thermal maps under different cooling conditions while running identical workloads. These thermal maps are converted into power maps using first-principles-based methods. By repeating this process across varying cooling environments, we collect power maps corresponding to different average chip temperatures. Our experimental results confirm that both total power and spatial power distributions vary significantly with cooling conditions, even under the same workload. We then train machine learning models using real-time performance and utilization metrics—collected via AMD Adrenalin Edition and Qualcomm Snapdragon Profiler—to capture these thermal effects. Two deep neural network architectures are explored: a transformer-based model, ChipPowerMap, and a CNN-based decoder model. We compare their performance in accurately predicting temperature-aware full-chip power maps. Numerical results highlight the effectiveness of ChipPowerMap in achieving highly accurate thermal map predictions, boasting an RMSE of only 67.88mW/mm<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> or 0.97% of the full-scale error. It also outperforms the CNN-based method by 1.62x in terms of accuracy on average. Besides, the proposed model offers real-time estimation with a rapid speed of 25ms on the target chip.

Detection of breast cancer remains an important global health issue, particularly in patients with dense breast tissue. Despite wide use of machine learning for breast cancer diagnosis, many existing studies rely on single-model evaluations and lack rigorous statistical validation, while transformer-based models remain insufficiently explored for structured tabular biomedical data. Using the Wisconsin Breast Cancer Diagnostic (WBCD) dataset, this study presents a comparative analysis of machine learning, deep learning, and transformer-based models for binary classification of breast masses as benign or malignant. A robust preprocessing and feature engineering pipeline was applied, and Support Vector Machine (SVM), Random Forest (RF), Multi-Layer Perceptron (MLP), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and BERT-based models were evaluated using a stratified 66–34% train–test split and stratified 5-fold cross-validation. Performance was assessed using standard classification metrics, Jaccard Index, AUC-ROC, false negative rate, and statistical significance testing (paired t-tests and Bayesian analysis). Three-layer RNN and LSTM models achieved the highest accuracy of 98.25%, showing statistically significant improvement over SVM (p = 0.042), while the optimized SVM demonstrated strong performance with superior computational efficiency. BERT-based models yielded lower accuracy (92.98%), reflecting domain mismatch between language-pretrained transformers and numerical tabular data. These findings indicate that deeper recurrent architectures provide superior diagnostic performance, while SVM remains a practical and efficient alternative for clinical deployment.

Recent advancements in transformers exploited computer vision problems which results in state-of-the-art models. Transformer-based models in various sequence prediction tasks such as language translation, sentiment classification, and caption generation have shown remarkable performance. Auto report generation scenarios in medical imaging through caption generation models is one of the applied scenarios for language models and have strong social impact. In these models, convolution neural networks have been used as encoder to gain spatial information and recurrent neural networks are used as decoder to generate caption or medical report. However, using transformer architecture as encoder and decoder in caption or report writing task is still unexplored. In this research, we explored the effect of losing spatial biasness information in encoder by using pre-trained vanilla image transformer architecture and combine it with different pre-trained language transformers as decoder. In order to evaluate the proposed methodology, the Indiana University Chest X-Rays dataset is used where ablation study is also conducted with respect to different evaluations. The comparative analysis shows that the proposed methodology has represented remarkable performance when compared with existing techniques in terms of different performance parameters.

Artificial intelligence and machine learning heuristics for discovery of ncRNAs.

Deep Learning Applications in Primary Education: A Systematic Literature Review of Emerging Trends, Challenges, and Opportunities. Objectives: Despite significant advances in healthcare and finance, Deep Learning (DL) remains underutilized in primary education, with only about 12% of AI-related studies focusing on K–6. Given the formative role of early schooling in cognitive and social development, this systematic review analyzed recent empirical studies to identify key trends, challenges, and opportunities in applying DL to primary education. Methods: Following the PRISMA 2020 guidelines, 21 studies published between 2021 and 2025 across 11 countries were analyzed using a structured coding sheet and further examined with bibliometric mapping, Microsoft Excel, and Python-based visualization tools. The review included diverse sources focusing on DL use cases across global primary education systems. Findings: The findings reveal a global increase in DL adoption, led by China and South Korea, with growing contributions from Indonesia, India, and the Philippines. Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), and transformer-based models are among the most commonly applied architectures to tasks such as personalized learning, early detection of learning difficulties, assessment automation, and curriculum development. However, there are still problems in the context of ethical concerns, including data privacy, algorithmic bias, and equity of access. Technical barriers involve dataset complexity, model generalization, and resource limitations, while pedagogical issues center on aligning DL applications with developmental needs and classroom realities. Despite these obstacles, DL demonstrates significant potential to enhance personalization, foster engagement, and support holistic educational outcomes. Conclusion: This review contributes a strategic roadmap for integrating DL in primary education by balancing innovation with pedagogy and ethics. Future research should prioritize cross-disciplinary collaboration, greater geographic diversity, and improvements in scalability and interpretability to ensure DL supports equitable, future-ready learning. Keywords: deep learning, primary education, personalized learning, systematic literature review, educational technology.

This review presents a comprehensive technical analysis of deep learning (DL) methodologies in biomedical signal processing, focusing on architectural innovations, experimental validation, and evaluation frameworks. We systematically evaluate key deep learning architectures including convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformer-based models, and hybrid systems across critical tasks such as arrhythmia classification, seizure detection, and anomaly segmentation. The study dissects preprocessing techniques (e.g., wavelet denoising, spectral normalization) and feature extraction strategies (time-frequency analysis, attention mechanisms), demonstrating their impact on model accuracy, noise robustness, and computational efficiency. Experimental results underscore the superiority of deep learning over traditional methods, particularly in automated feature extraction, real-time processing, cross-modal generalization, and achieving up to a 15% increase in classification accuracy and enhanced noise resilience across electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG) signals. Performance is rigorously benchmarked using precision, recall, F1-scores, area under the receiver operating characteristic curve (AUC-ROC), and computational complexity metrics, providing a unified framework for comparing model efficacy. The survey addresses persistent challenges: synthetic data generation mitigates limited training samples, interpretability tools (e.g., Gradient-weighted Class Activation Mapping (Grad-CAM), Shapley values) resolve model opacity, and federated learning ensures privacy-compliant deployments. Distinguished from prior reviews, this work offers a structured taxonomy of deep learning architectures, integrates emerging paradigms like transformers and domain-specific attention mechanisms, and evaluates preprocessing pipelines for spectral-temporal trade-offs. It advances the field by bridging technical advancements with clinical needs, such as scalability in real-world settings (e.g., wearable devices) and regulatory alignment with the Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR). By synthesizing technical rigor, ethical considerations, and actionable guidelines for model selection, this survey establishes a holistic reference for developing robust, interpretable biomedical artificial intelligence (AI) systems, accelerating their translation into personalized and equitable healthcare solutions.