A vessel trajectory prediction method based on an improved transformer architecture

The learning speed of the feed forward neural network takes a lot of time to be trained which is a major drawback in their applications since the past decades. The key reasons behind may be due to the slow gradient-based learning algorithms which are extensively used to train the neural networks or due to the parameters in the networks which are tuned iteratively using some learning algorithms. Thus, in order to eradicate the above pitfalls, a new learning algorithm was proposed known as Extreme Learning Machines (ELM). This algorithm tries to compute Hidden-layer-output matrix that is made of randomly assigned input layer and hidden layer weights and randomly assigned biases. Unlike the other feedforward networks, ELM has the access of the whole training dataset before going into the computation part. Here, we have devised a new two-layer-feedforward network (TFFN) for ELM in a new manner with randomly assigning the weights and biases in both the hidden layers, which then calculates the output-hidden layer weights using the Moore-Penrose generalized inverse. TFFN doesn't restricts the algorithm to fix the number of hidden neurons that the algorithm should have. Rather it searches the space which gives an optimized result in the neurons combination in both the hidden layers. This algorithm provides a good generalization capability than the parent Extreme Learning Machines at an extremely fast learning speed. Here, we have experimented the algorithm on various types of datasets and various popular algorithm to find the performances and report a comparison.

Evaluation metrics play an important role in accessing the performance of a regression method. In practice, these multiple evaluation metrics can be used in two ways. The first way defines a loss function by aggregating multiple metrics, while the second way defines a multiobjective loss function by considering each metric as an objective function. In this paper, we propose a new way to use multiple evaluation metrics, which is different from the aggregating method and the mutliobjective method. Our method is based on genetic programming. The idea is to randomly use one metric in each iteration of the selection operator. Therefore, multiple metrics can be used alternatively in the running process. To validate the effectiveness of our new approach, we conduct experiments on ten benchmark datasets. The experimental results show that the new approach can improve the population diversity, and can achieve the performance better than or similar to that of the traditional symbolic regression algorithms.

On computing decision regions with neural nets

Inter-areal information transmission in the brain cortex relates to cognitive functions. Researches used to pay attention to activity pattern transmission, signals gating, or routing in neuronal networks. However, the underlying mechanism of simultaneous transmission of multiple neural signals in the same channel across networks remains unclear. In this work, we construct a two-layer feedforward neuronal network (sender-receiver) with each layer's intrinsic rhythms consisting of slow- (low-frequency) and fast- gamma rhythms (high-frequency), investigating how to realize simultaneous transmission of multiple signals in neuronal systems. With the aid of resonance and frequency analysis, it is shown that low- and high-frequency signals can be transmitted simultaneously in such a feedforward network through frequency division multiplexing (FDM) communication. The transmission performance is related to the local resonance, connectivity, as well as background noise. Moreover, low- and high-frequency signals can also be gated or selected with appropriate adjustments of recurrent connection strength and delay, and background noise. Our model might provide a novel insight into the underlying mechanism of complex signals communication between different cortex areas.

This study proposes a power load forecasting method that combines the Informer model with the Ant Colony Optimization (ACO) algorithm. ACO optimizes the hyperparameters of the Informer model, significantly improving the model's accuracy and stability in power load forecasting. The Informer model utilizes its ProbSparse Attention technology to efficiently process long-term time series data and capture long-term dependencies in power load variations. ACO optimizes hyperparameter combinations through global search, avoiding the limitations of manual parameter tuning. Experimental results demonstrate that the proposed model outperforms traditional LSTM and TCN models across multiple evaluation metrics, demonstrating greater stability and prediction accuracy, particularly during peak load periods. This method provides effective technical support for smart grid scheduling and resource optimization.

Three-dimensional tooth segmentation plays a vital role in the diagnosis and treatment of dental diseases. However, due to the complex structure, significant regional heterogeneity, fuzzy boundaries, and the prevalence of small objects in CBCT images, existing methods face challenges in accuracy and robustness. This paper proposes a transformer-based approach, ToothFormer, designed to improve the segmentation accuracy of different tooth regions, fuzzy boundaries, and small objects. ToothFormer includes three key modules: the RAAM module (Region-Adaptive Attention Module), which implements region-adaptive attention allocation to improve segmentation consistency across different tooth regions; the DDRFM module (Dual-Domain Residual Fusion Module), which enhances boundary and small structure modeling through a detailed reconstruction path; and the DASM module (Dynamic Axis-aware Scale-fusion Module), which combines axial perception with a multi-scale attention mechanism to adapt to the anisotropic characteristics of CBCT images. Furthermore, this paper introduces the HAS module (Hybrid Attention Synergy) in the decoder for more effective fusion of local and global information. Experimental results on a public dental dataset demonstrate that ToothFormer significantly outperforms existing mainstream methods on multiple evaluation metrics, including Dice, Intersection over Union (IoU), and Average Segmented Distance (ASSD), validating the proposed method’s segmentation accuracy and robustness in complex dentition scenes.

To optimize the complex nonlinear temporal relationships in wind power forecasting, we proposed an improved BAS-BiLSTM-Attention model for enhancing the accuracy and robustness of wind power predictions. The Beetle Antennae Search algorithm was utilized for obtaining hidden features across different frequency bands. The Bidirectional Long Short-Term Memory algorithm captured the bidirectional temporal associations of hidden features. Finally, an Attention mechanism was added to ensure that the weight selection of the feature components remains unaffected. Results demonstrated that our method performed excellently in wind power forecasting tasks. Compared to individual models and other common algorithms, it exhibited higher prediction accuracy and robustness across multiple evaluation metrics.

Global climate change is one of the most pressing environmental challenges of the 21st century. Traditional climate prediction methods have significant limitations in capturing complex temperature variation patterns, necessitating the development of innovative approaches. This study proposes a multimodal temperature anomaly detection and climate trend prediction method based on deep learning. By integrating linear regression with a long short-term memory (LSTM) network, the approach enhances the accuracy and reliability of global temperature predictions. The study employs the Berkeley Earth Surface Temperature dataset, which contains global historical temperature data from 1750 to 2015. The proposed multimodal deep learning model consists of two branches: linear regression, which captures long-term trends, and LSTM, which models nonlinear temperature variations. A systematic data preprocessing pipeline was developed, incorporating missing value handling, time series normalization, and advanced feature engineering, significantly improving data utilization efficiency. Experimental results demonstrate that the proposed method outperforms traditional statistical approaches and existing deep learning models across multiple evaluation metrics. Compared with the Transformer model, it achieves reductions of 7.1% in mean square error (MSE), 3.6% in root mean square error (RMSE), and 4.2% in mean absolute error (MAE). This research not only introduces an innovative framework to academia but also establishes a new technical paradigm for climate change studies, contributing to climate response strategy formulation.

Medical organ segmentation is essential for accurately identifying the underlying causes of diseases and developing appropriate treatment plans. Current distribution methods primarily focus on the segmentation task, often neglecting the relationship between image distribution and segmentation. Effective image distribution aligns features and can also serve as a boundary constraint in the segmentation task. The high cost of annotating medical images limits the number of available labeled samples. Furthermore, effective methods for extracting organ positional information between image slices are lacking. This paper introduces a distribution-constrained semi-supervised segmentation method (DCSSM). We first design a Category Center of Mass Calculation Module to capture the positional correlation between image slices. Next, we use a public encoder and distribution network for pretraining, enabling segmentation of a large number of unlabeled samples with only a few labeled samples. We also design a lightweight multiscale feature extraction module and use it to build the segmentation network, achieving rapid feature extraction without compromising performance. Finally, the pretrained public encoder and distribution network align the features of the input images, with the aligned boundaries acting as constraints to guide mask generation in the segmentation network. The proposed method is evaluated through comparative and ablation experiments on the publicly available Synapse and skin lesion datasets, against eight baseline methods. We further validate the generalization ability of the proposed method by applying it to skin lesion segmentation datasets. Experimental results show that the proposed method achieves optimal performance across multiple evaluation metrics.

Model editing techniques modify a minor proportion of knowledge in Large Language Models (LLMs) at a relatively low cost, which have demonstrated notable success. Existing methods assume Transformer Layer (TL) hidden states are values of key-value memories of the Feed-Forward Network (FFN). They usually optimize the TL hidden states to memorize target knowledge and use it to update the weights of the FFN in LLMs. However, the information flow of TL hidden states comes from three parts: Multi-Head Self-Attention (MHSA), FFN, and residual connections. Existing methods neglect the fact that the TL hidden states contains information not specifically required for FFN. Consequently, the performance of model editing decreases. To achieve more precise model editing, we analyze hidden states of MHSA and FFN, finding that MHSA encodes certain general knowledge extraction patterns. This implies that MHSA weights do not require updating when new knowledge is introduced. Based on above findings, we introduce PMET, which simultaneously optimizes Transformer Component (TC, namely MHSA and FFN) hidden states, while only using the optimized TC hidden states of FFN to precisely update FFN weights. Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the \textsc{counterfact} and zsRE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. Our code is available at \url{https://github.com/xpq-tech/PMET}.

We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) a two-layer feed-forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. Finally, by adapting our model to machine translation we achieve surprisingly good results. We share pre-trained models and our code based on the Timm library.

There are more than 2,000 listed companies on the UK’s London Stock Exchange, divided into 11 sectors who are required to communicate their financial results at least twice in a single financial year. UK annual reports are very lengthy documents with around 80 pages on average. In this study, we aim to benchmark a variety of summarisation methods on a set of different pre-trained transformers with different extraction techniques. In addition, we considered multiple evaluation metrics in order to investigate their differing behaviour and applicability on a dataset from the Financial Narrative Summarisation (FNS 2020) shared task, which is composed of annual reports published by firms listed on the London Stock Exchange and their corresponding summaries. We hypothesise that some evaluation metrics do not reflect true summarisation ability and propose a novel BRUGEscore metric, as the harmonic mean of ROUGE-2 and BERTscore. Finally, we perform a statistical significance test on our results to verify whether they are statistically robust, alongside an adversarial analysis task with three different corruption methods.

Credit risk refers to the possibility that a borrower fails to meet contractual repayment obligations, posing potential losses to lenders. This study aims to enhance post-loan default prediction in credit risk management by constructing a time-series modeling framework based on repayment behavior data, enabling the capture of repayment risks that emerge after loan issuance. To achieve this objective, a Residual Enhanced Encoder Bidirectional Long Short-Term Memory (ResE-BiLSTM) model is proposed, in which the attention mechanism is responsible for discovering long-range correlations, while the residual connections ensure the preservation of distant information. This design mitigates the tendency of conventional recurrent architectures to overemphasize recent inputs while underrepresenting distant temporal information in long-term dependency modeling. Using the real-world large-scale Freddie Mac Single-Family Loan-Level Dataset, the model is evaluated on 44 independent cohorts and compared with five baseline models, including Long Short-Term Memory (LSTM), Bidirectional LSTM (BiLSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) across multiple evaluation metrics. The experimental results demonstrate that ResE-BiLSTM achieves superior performance on key indicators such as F1 and AUC, with average values of 0.92 and 0.97, respectively, and demonstrates robust performance across different feature window lengths and resampling settings. Ablation experiments and SHapley Additive exPlanations (SHAP)-based interpretability analyses further reveal that the model captures non-monotonic temporal importance patterns across key financial features. This study advances time-series–based anomaly detection for credit risk prediction by integrating global and local temporal learning. The findings offer practical value for financial institutions and risk management practitioners, while also providing methodological insights and a transferable modeling paradigm for future research on credit risk assessment.

An indoor positioning optimization method based on dimensionality reduction and clustering fusion

Biomedical knowledge graphs (KGs), which can help with the understanding of complex biological systems and pathologies, have begun to play a critical role in medical practice and research. However, challenges remain in their embedding and use due to their complex nature and the specific demands of their construction. Existing studies often suffer from problems such as sparse and noisy datasets, insufficient modeling methods and non-uniform evaluation metrics. In this work, we established a comprehensive KG system for the biomedical field in an attempt to bridge the gap. Here, we introduced PharmKG, a multi-relational, attributed biomedical KG, composed of more than 500000 individual interconnections between genes, drugs and diseases, with 29 relation types over a vocabulary of ~8000 disambiguated entities. Each entity in PharmKG is attached with heterogeneous, domain-specific information obtained from multi-omics data, i.e. gene expression, chemical structure and disease word embedding, while preserving the semantic and biomedical features. For baselines, we offered nine state-of-the-art KG embedding (KGE) approaches and a new biological, intuitive, graph neural network-based KGE method that uses a combination of both global network structure and heterogeneous domain features. Based on the proposed benchmark, we conducted extensive experiments to assess these KGE models using multiple evaluation metrics. Finally, we discussed our observations across various downstream biological tasks and provide insights and guidelines for how to use a KG in biomedicine. We hope that the unprecedented quality and diversity of PharmKG will lead to advances in biomedical KG construction, embedding and application.