Fusion-Based Hybrid Model for SMS Spam Detection Integrating Local, Sequential, and Contextual Features

For the traditional model based on the deep learning method most used CNN(convolutional neural networks) or RNN(Recurrent neural Network) model and is based on the dynamic character-level embedding or word-level embedding as input, so there is a problem that the text feature extraction is not comprehensive. In the development environment of the Internet of Things, A method of Automatic text classification based on BERT(Bidirectional Encoder Representations from Transformers) and Feature Fusion was proposed in this paper. Firstly, the text-to-dynamic character-level embedding is transformed by the BERT model, and the BiLSTM(Bi-directional Long-Short Term Memory) and CNN output features are combined and merged to make full use of CNN to extract the advantages of local features and to use BiLSTM to have the advantage of memory to link the extracted context features to better represent the text, so as to improve the accuracy of text classification task. A comparative study with state-of-the-art approaches manifests the proposed method outperforms the state-of-the-art methods in accuracy. It can effectively improve the accuracy of tag prediction for text data with sequence features and obvious local features.

This study proposed a medicine auxiliary diagnosis model based on neural network. The model combines a bidirectional long short-term memory（Bi-LSTM）network and bidirectional encoder representations from transformers (BERT), which can well complete the extraction of local features of Chinese medicine texts. BERT can learn the global information of the text, so use BERT to get the global representation of medical text and then use Bi-LSTM to extract local features. We conducted a large number of comparative experiments on datasets. The results show that the proposed model has significant advantages over the state-of-the-art baseline model. The accuracy of the proposed model is 0.75.

In the rapidly evolving Internet of Things (IoT) landscape, accurate indoor positioning is increasingly vital. The proposed algorithm synergizes an ultrawideband (UWB) sensor with an inertial measurement unit (IMU) and artificial intelligence to obtain precise positioning in nonline-of-sight (NLOS) scenarios. In the proposed UWB module, a large language model (LLM) such as bidirectional encoder representations from transformers (BERT) algorithm is designed to utilize the channel impulse response (CIR) for effective NLOS identification and UWB ranging trustworthiness evaluation. Concurrently, the IMU module is also designed with BERT to recognize various pedestrian activity states, thereby optimizing positioning. BERT’s self-attention mechanism and deep learning (DL) bidirectional training efficiently extract essential features from sequential data, capturing both local and global information. The integration of both UWB and IMU through a proposed tightly coupled algorithm significantly boosts positioning performance. Experimental campaigns demonstrate an average NLOS identification accuracy, line-of-sight (LOS), and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${F}2$ </tex-math></inline-formula> of 98.8%, 99.4%, and 0.9926, respectively. These performances surpass the state-of-the-art least-squares support vector machine (LS-SVM), convolutional neural network (CNN), and CNN with long short-term memory (CNN-LSTM) up to 17.66% in NLOS identification. In terms of pedestrian activity recognition using BERT, the BERT algorithm achieves a precision (recall) of 99.3% (99.4%), notably outperforming the CNN and CNN-LSTM by 17.9% (16.2%) and 11.9% (10.9%), respectively. Finally, the UWB-IMU algorithm significantly enhances positioning accuracy by 80.5%, outperforming Kalman, LSTM-EKF, and particle filter (PF) methods by 68.1%, 48.3%, and 45.0%, respectively. The proposed approach presents a robust solution for indoor positioning for IoT applications, particularly in challenging NLOS environments.

Driven by the rapid development of Internet, more e-commerce product reviews are available on e-commerce platforms, which can help enterprises make business decisions. Currently, bidirectional encoder representations from transformers (BERT) applied in the embedding layer contributes to achieve promising results in English text sentiment analysis (SA). This paper proposes a novel model Chinese BERT with fused deep neural networks (CBERT-FDNN), extracting richer and more accurate semantic and grammatical information in Chinese text. First, Chinese BERT with whole word masking (Chinese-BERT-wwm) is used in the embedding layer to generate dynamic sentence representation vectors. It is a Chinese pre-training model based on the whole word masking (WWM) technology, which is more effective for Chinese text contextual embedding. Second, multi-channel and multi-scale convolutional neural networks (CNN) and bidirectional long short-term memory (BiLSTM) are designed to capture further crucial features in the feature extraction layer. To obtain more comprehensive sentence attributes, these features are concatenated together. Last, the model is evaluated on 100,000 sentence-level Chinese e-commerce product reviews for sentiment binary classification. The accuracy and F1 score can achieve 94.37% and 94.34%, respectively. Compared with the baseline models, the experiments show that our proposed model has higher accuracy and better prediction performance.

Sentiment analysis of netizens’ comments can accurately grasp the psychology of netizens and reduce the risks brought by online public opinion. However, there is currently no effective method to solve the problems of short text, open word range, and sometimes reversed word order in comments. To better solve the above problems, this article proposes a hybrid model of sentiment classification, which is based on bidirectional encoder representations from transformers (BERT), bidirectional long short-term memory (BiLSTM) and a text convolution neural network (TextCNN) (BERT-BiLSTM-TextCNN). The experimental results show that (1) the hybrid model proposed in this article can better combine the advantages of BiLSTM and TextCNN; it not only captures local correlation while retaining context information but also has high accuracy and stability. (2) The BERT-BiLSTM-TextCNN model can extract important emotional information more flexibly in text and achieve multiclass classification tasks of emotions more accurately. The innovations of this study are as follows: (1) the use of BERT to generate word vectors has the advantages of more prior information and a full combination of contextual semantics; (2) the BiLSTM model, as a bidirectional context mechanism model, can obtain contextual information well; and (3) the TextCNN model can obtain important features well in the problem of text classification, and the combined effect of the three modules can significantly improve the accuracy of emotional multilabel classification.

With the continuous evolution of natural language processing (NLP), conversational agents have become pivotal in enhancing user engagement and satisfaction in the e-commerce domain. This paper presents a comprehensive study and implementation of intent recognition and slot filling techniques tailored for an e-commerce chatbot. Leveraging a dataset of 8,000 samples, we employed a Convolutional Neural Network (CNN) combined with Long Short-Term Memory (LSTM) networks for intent recognition, achieving an accuracy of 87%. For slot filling, we utilized the Bidirectional Encoder Representations from Transformers (BERT) model, also attaining an accuracy of 87%. The chatbot seamlessly integrates with an e-commerce database, using OpenAI's GPT-3 to generate natural language responses from query results. Our system demonstrates significant advancements in processing user queries, generating precise database queries, and providing coherent and relevant responses, thereby enhancing the overall user experience. This work aims to equip researchers and practitioners with insights into the methodologies and challenges in developing sophisticated e-commerce chatbots, fostering further innovation in this field. Key Words: Intent Recognition, Slot Filling, Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), Bidirectional Encoder Representations from Transformers (BERT), GPT-3, Natural Language Understanding (NLU), Chatbot

Emails are the simplest way to communicate between users. But increasing rate of cybercrime activities require shrewd use of email. Phishing email is one of the most perilous threats nowadays. Through phishing email, attackers want to steal the sensitive information of the email users. This sensitive information includes username, password, credit card details etc. These may cause excessive financial loss. Many anti-phishing methods exist in literature to detect phishing emails. But criminals use new methods day by day, which causes challenge to build anti-phishing methods to prevent phishing emails. In this paper the performance of two deep learning models for detection of phishing email have been compared. The first model uses convolution neural network with Global Vector (GloVe) word embedding where Bidirectional Encoder Representations from Transformers (BERT) model with fine tuning has been used in second model. The proposed method detects the phishing email by analyzing the content of the email. Some widely used datasets (lingSpam, enronSpamSubset, completeSpamAssassin, Jose Nazario's phishing dataset and Enron email dataset) are merged together and applied to measure the performance of the model. It is found that the GloVe word embedding achieves better accuracy (98%) than BERT model (96%) in detection of phishing emails in the present context.

Fenomena bullying di media sosial, khususnya di Twitter, telah menjadi isu yang semakin memprihatinkan dengan dampak signifikan terhadap kesehatan mental pengguna. Dalam rangka mengatasi masalah ini, deteksi otomatis tweet yang mengandung konten bullying menjadi sangat penting. Penelitian ini bertujuan untuk membandingkan performa dua model pemrosesan bahasa alami terbaru, yaitu BERT (Bidirectional Encoder Representations from Transformers) dan XLNet, dalam klasifikasi tweet yang mengandung bullying. Metodologi penelitian ini melibatkan pengumpulan dataset tweet yang telah dilabeli sebagai bullying atau non-bullying. Proses preprocessing teks dilakukan untuk membersihkan dan menyiapkan data sebelum digunakan dalam pelatihan model. Kedua model, BERT dan XLNet, dilatih dan diuji menggunakan dataset yang sama. Evaluasi performa dilakukan dengan menggunakan metrik akurasi, presisi, recall, dan F1-score. Hasil penelitian menunjukkan bahwa kedua model memiliki kemampuan yang baik dalam mengidentifikasi tweet bullying, akan tetapi XLNet menunjukkan performa yang lebih unggul dibandingkan BERT dengan tingkat akurasi sebesar 95%. Dengan nilai presisi = 100%, recall = 0,87%, dan F1-score = 0,88%. XLNet mampu menangkap konteks dan nuansa bahasa yang lebih kompleks dalam tweet, yang berkontribusi pada akurasi klasifikasi yang lebih tinggi. Penelitian ini memberikan kontribusi penting dalam bidang deteksi bullying di media sosial dengan menunjukkan bahwa penggunaan model XLNet lebih efektif dibandingkan BERT. Temuan ini dapat membantu platform seperti Twitter dalam mengidentifikasi dan mencegah konten bullying, sehingga menciptakan lingkungan online yang lebih aman bagi pengguna, serta dapat digunakan sebagai dasar untuk pengembangan sistem deteksi bullying yang lebih canggih dan efisien di masa depan. Abstract The phenomenon of bullying on social media, particularly on Twitter, has become an increasingly concerning issue with significant impacts on users' mental health. In order to address this issue, automatic detection of tweets containing bullying content is crucial. This study aims to compare the performance of two recent natural language processing models, namely BERT (Bidirectional Encoder Representations from Transformers) and XLNet, in the classification of tweets containing bullying. The research methodology involves collecting a dataset of tweets that have been labelled as bullying or non-bullying. Text preprocessing is done to clean and prepare the data before it is used in model training. Both models, BERT and XLNet, were trained and tested using the same dataset. Performance evaluation was conducted using accuracy, precision, recall, and F1-score metrics. The results show that both models have a good ability to identify bullying tweets, but XLNet shows superior performance compared to BERT with an accuracy rate of 95%. With precision = 100%, recall = 0.87%, and F1-score = 0.88%. XLNet is able to capture more complex context and language nuances in tweets, which contributes to higher classification accuracy. This research makes an important contribution to the field of bullying detection on social media by showing that the use of the XLNet model is more effective than BERT. These findings can help platforms like Twitter identify and prevent bullying content, thereby creating a safer online environment for users, and can be used as a basis for the development of more sophisticated and efficient bullying detection systems in the future.

Short text classification is a research focus for natural language processing (NLP), which is widely used in news classification, sentiment analysis, mail filtering and other fields. In recent years, deep learning techniques are applied to text classification and has made some progress. Different from ordinary text classification, short text has the problem of less vocabulary and feature sparsity, which raise higher request for text semantic feature representation. To address this issue, this paper propose a feature fusion framework based on the Bidirectional Encoder Representations from Transformers (BERT). In this hybrid method, BERT is used to train word vector representation. Convolutional neural network (CNN) capture static features. As a supplement, a bi-gated recurrent neural network (BiGRU) is adopted to capture contextual features. Furthermore, an attention mechanism is introduced to assign the weight of salient words. The experimental results confirmed that the proposed model significantly outperforms the other state-of-the-art baseline methods.

As the online shopping industry continues to grow at a staggering rate, e-commerce channels have also begun the use of automated systems for sentiment analysis so as to be able to assess comprehension of user reviews. Over the past few years, machine learning (ML) has come of age with the introduction of deep learning models such as Convolutional Neural Networks (CNNs) and Bidirectional Encoder Representations from Transformers (BERT) which have greatly improved the levels of accuracy achieved in sentiment analysis. This paper aims to address this problem by proposing a hybrid model, based on networks of convolutional neural networks (CNNs) and BERT, for the purposes of e-commerce sentiment classification. Posting more plausible situations, academic findings, and perspectives for future research, we focus more on practical issues. When we compared traditional approaches with our hybrid, it became clear that the model performs better in many dimensions: accuracy, precision, and progressively recall

In this paper, the named entity recognition (NER) technology is applied to the field of Qu Tan temple murals to identify religious entities in the text. In order to obtain a machine-readable corpus, the relevant books are scanned and optical character recognition is performed, and then the data is manually labeled to form the named entity recognition dataset needed in this study. In this paper, the bidirectional encoder representations from transformers (BERT) model is used to encode the input data, and the obtained word vector can effectively extract the features of the text by making use of the relationship between words. In order to obtain more accurate context information, the bidirectional long short term memory (BiLSTM) is used to extract features and output the prediction label corresponding to each word. After the BiLSTM model, the conditional random field (CRF) algorithm is joined, which will correct the score of the output, and the output is the optimal label sequence. In this study, the performance of the model is also improved by data enhancement.

This paper proposes an approach for aspect-based sentiment analysis of Arabic social data, especially the considerable text corpus generated through communications on X (formerly known as Twitter) for expressing opinions in Arabic-language tweets during the COVID-19 pandemic. The proposed approach examines the performance of several pre-trained predictive and autoregressive language models; namely, Bidirectional Encoder Representations from Transformers (BERT) and XLNet, along with topic modeling algorithms; namely, Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF), for aspect-based sentiment analysis of online Arabic text. In addition, Bidirectional Long Short Term Memory (Bi-LSTM) deep learning model is used to classify the extracted aspects from online reviews. Obtained experimental results indicate that the combined XLNet-NMF model outperforms other implemented state-of-the-art methods through improving the feature extraction of unstructured social media text with achieving values of 0.946 and 0.938, for average sentiment classification accuracy and F-measure, respectively.

With the increasing number of web texts, the classification of web texts has become an important task. In this paper, the text word vector representation method is first analyzed, and bidirectional encoder representations from transformers (BERT) are selected to extract the word vector. The bidirectional gated recurrent unit (BiGRU), convolutional neural network (CNN), and attention mechanism are combined to obtain the context and local features of the text, respectively. Experiments were carried out using the THUCNews dataset. The results showed that in the comparison between word-to-vector (Word2vec), Glove, and BERT, the BERT obtained the best classification result. In the classification of different types of text, the average accuracy and F1 value of the BERT-BGCA method reached 0.9521 and 0.9436, respectively, which were superior to other deep learning methods such as TextCNN. The results suggest that the BERT-BGCA method is effective in classifying web texts and can be applied in practice.

Traditional Chinese medicine diagnostic prediction model for holistic syndrome differentiation based on deep learning

In this article we present an extended version of PolEmo – a corpus of consumer reviews from 4 domains: medicine, hotels, products and school. Current version (PolEmo 2.0) contains 8,216 reviews having 57,466 sentences. Each text and sentence was manually annotated with sentiment in 2+1 scheme, which gives a total of 197,046 annotations. We obtained a high value of Positive Specific Agreement, which is 0.91 for texts and 0.88 for sentences. PolEmo 2.0 is publicly available under a Creative Commons copyright license. We explored recent deep learning approaches for the recognition of sentiment, such as Bi-directional Long Short-Term Memory (BiLSTM) and Bidirectional Encoder Representations from Transformers (BERT).