Text segmentation for patent claim simplification via Bidirectional Long‐Short Term Memory and Conditional Random Field

BackgroundIn biomedical text mining, named entity recognition (NER) is an important task used to extract information from biomedical articles. Previously proposed methods for NER are dictionary- or rule-based methods and machine learning approaches. However, these traditional approaches are heavily reliant on large-scale dictionaries, target-specific rules, or well-constructed corpora. These methods to NER have been superseded by the deep learning-based approach that is independent of hand-crafted features. However, although such methods of NER employ additional conditional random fields (CRF) to capture important correlations between neighboring labels, they often do not incorporate all the contextual information from text into the deep learning layers.ResultsWe propose herein an NER system for biomedical entities by incorporating n-grams with bi-directional long short-term memory (BiLSTM) and CRF; this system is referred to as a contextual long short-term memory networks with CRF (CLSTM). We assess the CLSTM model on three corpora: the disease corpus of the National Center for Biotechnology Information (NCBI), the BioCreative II Gene Mention corpus (GM), and the BioCreative V Chemical Disease Relation corpus (CDR). Our framework was compared with several deep learning approaches, such as BiLSTM, BiLSTM with CRF, GRAM-CNN, and BERT. On the NCBI corpus, our model recorded an F-score of 85.68% for the NER of diseases, showing an improvement of 1.50% over previous methods. Moreover, although BERT used transfer learning by incorporating more than 2.5 billion words, our system showed similar performance with BERT with an F-scores of 81.44% for gene NER on the GM corpus and a outperformed F-score of 86.44% for the NER of chemicals and diseases on the CDR corpus. We conclude that our method significantly improves performance on biomedical NER tasks.ConclusionThe proposed approach is robust in recognizing biological entities in text.

Text mining in the biomedical domain is increasingly important as the volume of biomedical documents increases. Thanks to advances in natural language processing (NLP), extracting valuable information from the biomedical literature is gaining popularity among researchers, and deep learning has enabled the development of effective biomedical text mining models. However, directly applying advancements in NLP to biomedical sources often yields unsatisfactory results, due to a word distribution drift from the general language domain corpora to specific biomedical corpora, and this drift introduces linguistic ambiguities. To overcome these challenges, this paper presents a novel method for biomedical named entity-recognition (BioNER) through hierarchically fusing representations from BioBERT, which is trained on biomedical corpora and Deep contextual-level word embeddings to handle the linguistic challenges within biomedical literature. Proposed text representation is then fed to attention-based Bi-directional Long Short Term Memory (BiLSTM) with Conditional random field (CRF) for the BioNER task. The experimental analysis shows that our proposed end-to-end methodology outperforms existing state-of-the-art methods for the BioNER task.

Human activity recognition has become an important research topic within the field of pervasive computing, ambient assistive living (AAL), robotics, health-care monitoring, and many more. Techniques for recognizing simple and single activities are typical for now, but recognizing complex activities such as concurrent and interleaving activity is still a major challenging issue. In this paper, we propose a two-phase hybrid deep machine learning approach using bi-directional Long-Short Term Memory (BiLSTM) and Skip-Chain Conditional random field (SCCRF) to recognize the complex activity. BiLSTM is a sequential generative deep learning inherited from Recurrent Neural Network (RNN). SCCRFs is a distinctive feature of conditional random field (CRF) that can represent long term dependencies. In the first phase of the proposed approach, we recognized the concurrent activities using the BiLSTM technique, and in the second phase, SCCRF identifies the interleaved activity. Accuracy of the proposed framework against the counterpart state-of-art methods using the publicly available datasets in a smart home environment is analyzed. Our experiment’s result surpasses the previously proposed approaches with an average accuracy of more than 93%.

Lexis Advance is a legal research service provided by LexisNexis that can respond to natural language queries. It includes a module called Lexis Answers which implements advanced Natural Language Processing (NLP) capabilities to improve understanding of the intent of the user’s queries. Lexis Answers can respond to natural language questions concerning legal question types such as statute of limitations, elements of a claim, definition of legal terms, and others. Herein, we report on the successful use of advanced NLP approaches for detecting not only named entities, but entire legal phrases, a skill previously requiring domain knowledge and human expertise. We have utilized the Conditional Random Fields (CRFs) approach that employs hand-engineered features combined with word2vec embeddings trained on legal corpus. Furthermore, to reduce our dependency on hand-engineered features, we have also implemented deep learning architecture comprising of bidirectional Long Short-Term Memory (BiLSTM) and linear chain CRF. Both approaches were benchmarked against a rule-based approach for different types of legal questions. We find that both CRF and BiLSTM-CRF can identify query intents and legal concepts with comparable precision but much higher recall and F-score than the baseline. The resulting models have been employed in Lexis Answers as critical improvement in our natural language query understanding.

Aspect-based totally Opinion Mining is a type of sentiment evaluation that is more advanced in that it models the semantic hyperlink between thing keywords and context terms in a sentence. The lifestyles of a range of surrounding words have a widespread impact on a commentary’s sentiment polarity. Though previous systems considered the position of an aspect in a phrase, few researches have investigated the merging of Sentiment Lexicons (SL) using Deep Learning (DL) techniques. The data for this investigation originated from online item audits compiled from several information repositories. The findings of tests for sentence-level and survey-level characterization are encouraging. The passionate investigation will be used to the survey dataset to analyze the audits and enable the client in determining whether customers are satisfied with the company’s products, prices, and customer service. Unlike traditional NLP (Natural Language Processing) approaches, the proposed framework employs the Bi-LTSM (Bidirectional Long-Short Term Memory) computation for precise results. Finally, some research insights are provided to record future feelings on examination attempts. The emotional analysis was used to the Amazon product review dataset to get the required conclusion. The framework includes fundamental data processing procedures such as data preparation, keyword extraction, and so on, and eventually the Bi-LSTM LSTM (Bidirectional Long-Short Term Memory) algorithm is applied to the data and product ranking is given. In addition, the system compares several algorithms such as CNN (Convolutional Neural Network), RNN (Recurrent Neutral Network), LSTM (Long-Short Term Memory), and Bi-LSTM (Bidirectional Long-Short Term. Memory). According to the data, enhanced Bi-LSTM (Bidirectional Long-Short Term Memory) exceeds all other algorithms in terms of accuracy.

Semantic Role Labeling (SRL) is one of the most challenging tasks in Natural Language Processing (NLP). SRL is a task that consists of predicate identification, argument identification and argument classification. In this article we present novel approach for argument classification that is based on deep neural network architecture. Traditional discrete based SRL relies heavily on feature engineering that uses syntactic structures in contrast to deep learning approaches that encode whole sentences without taking into account syntactic features. We present an approach that uses combination of syntactic features and external word representations from FastText. The advantages of using FastText embeddings is the generation of better vector representations for rare words and FastText gives better results for words that are not within the dictionary. These attributes for vector representations give good results for morphologically rich languages. Most of the SRL approaches today are trained on resource rich languages. In this article we present novel neural architecture for SRL that is suitable for resource poor morphology rich languages. Experiments on hr500k corpus shows that our syntax-aware approach shows competitive results for argument classification. We present architecture for argument classification that is based on Bidirectional Long-Short Term Memory (Bi-LSTM) and Conditional Random Field (CRF) decoding for finding optimal sequence. Our approach showed results that are very close to benchmark results with F1 score of 72%.

Part of speech (POS) tagging is an initial task for many natural language applications. POS tagging for Amharic is in its infancy. This study contributes towards the improvement of Amharic POS tagging by experimenting using Deep Learning and Conditional Random Fields (CRF) approaches. Word embedding is integrated into the system to enhance performance. The model was applied to an Amharic news corpus tagged into 11 major part of speeches and achieved accuracies of 91.12% and 90% for the Bidirectional LSTM and CRF methods respectively. The result shows that the Bidirectional LSTM approach performance is better than the CRF method. More enhancement is expected in the future by increasing the size and diversity of Amharic corpus.

Automatic Parts of Speech(POS) tagging is one of the most fundamental tasks for a language in Natural Language Processing(NLP), which acts as a feature for solving advanced NLP tasks. Named Entity Recognition(NER) is another essential task of NLP for information retrieval. Researchers could not find up to the mark solution yet on these two tasks for Bangla language compared to other languages, for instance, English, Ger-man. Moreover, many solutions heavily depend on handcrafted features that require strong linguistic expertise. As these two sequence labeling tasks are similar, In this work, two different datasets of POS tagging and NER were prepared, and different deep neural network approaches studied for solving these two tasks separately. All of the approaches were end to end and did not need any handcrafted feature like word suffixes or affixes, gazetteers, dictionary. This study came up with an end to end solution using deep neural network-based model consisting of Bi-directional Long short-term memory(BLSTM), Convolutional Neural Network(CNN) and Conditional Random Field(CRF). The proposed model trained on respected datasets achieved an accuracy of 93.86% on POS tagging and a strict f1 score of 0.6285 on NER on prepared datasets, respectively.

Person names often need to be represented in a consistent format in an application, for example, in format in library catalogs. Obtaining a normalized representation automatically from an input name requires precise labeling of its components. The process is difficult owing to numerous cultural conventions in writing personal names. In this paper, we propose deep learning-based techniques to achieve this using sequence-to-sequence learning. We design several architectures using a bidirectional long short-term memory (BiLSTM)-based recurrent neural network (RNN). We compare these methods with one based on the hidden Markov model. We perform experiments on a large collection of author names drawn from the National Digital Library of India. The best accuracy of \(94\%\) is achieved by the character-level BiLSTM with a conditional random field at the output layer. We also show visualizations of the vectors (representing person names) learned by a BiLSTM and how these vectors are clustered according to name structures. Our study shows that deep learning is a promising approach to automatic name segmentation.

Parts of Speech (pos) tagging corresponds to techniques of tagging accurate pos tags of each word of a sentence. It is a pivotal and indispensable part of sentiment analysis, synthesizing text to speech, analyzing textual data and many more Natural Language Processing (NLP) tasks. Though there are many researches done in the field of languages like English, Chinese etc., it is still comparatively unexplored for Bangla. An excellent pos tagger in Bangla language can accelerate NLP of Bangla textual data. In this study, we have developed a novel hybrid Deep Learning (DL) model with two parallel input networks with Convolutional Neural Network (CNN), and Bidirectional Long Short Term Memory (BiLSTM), and an output network of CNN. CNN works well regarding the local dependencies of words in sentences, and BiLSTM works well with the global dependencies of words regarding the context in a sentence. No other work has been done with local features of textual data along with long term word dependencies in Bangla pos tagging studies. The model can work with any external Word Embeddings. In this study, it is implemented with both Embedding Layer and Word2Vec layer. In the evaluation process, Mathews Correlation Coefficient (MCC), Precision-Recall curve are used along with accuracy and F1-score due to the dataset's imbalanced characteristic. Our proposed model accompanied by Word2Vec layer has outperformed Vanilla Recurrent Neural Network (RNN), Gated Recurrent Units (GRU), Long Short Term Memory (LSTM), BiLSTM networks, and also one recent study using the same dataset with the highest accuracy of 0.974, F1-score of 0.883, and MCC value of 0.936.

Failing to identify multi-word expression (MWE) may cause serious problems for many Natural Language Processing (NLP) tasks. Previous approaches heavily depend on language specific knowledge and pre-existing natural language processing (NLP) tools. However, many languages (including Chinese language) have less such resources and tools compared to English. An automatically learn effective features from corpus, without relying on language specific resources is needed. In this paper, we develop a hybrid approach that combines Bidirectional long short-term memory (Bi-LSTM), word correlation degree calculation and weakly supervised K-means cluster to capture both sequence information and correlation degree of phrase from specific contexts, and use them to train a multi-word expression detector for multiple languages without any manually encoded features. Experiment result shows that the extraction results of Chinese and English multi-word expression using this hybrid approach is better than that of baseline algorithm, which verified that the hybrid approach is effective.

The COVID-19 pandemic has had a devastating effect on many people, creating severe anxiety, fear, and complicated feelings or emotions. After the initiation of vaccinations against coronavirus, people's feelings have become more diverse and complex. Our aim is to understand and unravel their sentiments in this research using deep learning techniques. Social media is currently the best way to express feelings and emotions, and with the help of Twitter, one can have a better idea of what is trending and going on in people's minds. Our motivation for this research was to understand the diverse sentiments of people regarding the vaccination process. In this research, the timeline of the collected tweets was from December 21 to July21. The tweets contained information about the most common vaccines available recently from across the world. The sentiments of people regarding vaccines of all sorts were assessed using the natural language processing (NLP) tool, Valence Aware Dictionary for sEntiment Reasoner (VADER). Initializing the polarities of the obtained sentiments into three groups (positive, negative, and neutral) helped us visualize the overall scenario; our findings included 33.96% positive, 17.55% negative, and 48.49% neutral responses. In addition, we included our analysis of the timeline of the tweets in this research, as sentiments fluctuated over time. A recurrent neural network- (RNN-) oriented architecture, including long short-term memory (LSTM) and bidirectional LSTM (Bi-LSTM), was used to assess the performance of the predictive models, with LSTM achieving an accuracy of 90.59% and Bi-LSTM achieving 90.83%. Other performance metrics such as precision,, F1-score, and a confusion matrix were also used to validate our models and findings more effectively. This study improves understanding of the public's opinion on COVID-19 vaccines and supports the aim of eradicating coronavirus from the world.

Weaponry equipment names belong to an important military naming entity that is difficult to identify because of features, such as complex components, miscellaneous, and scarce annotation corpus. Here, the automatic recognition of weaponry equipment names is specifically explored, a NER (Named Entity Recognition) algorithm is proposed based on BI-LSTM-CRF (Bi-directional Long Short Term Memory Conditional Random Field), thereby demonstrating the effectiveness of domain features in domain-specific entity recognition. Firstly, Chinese characters are represented by word embedding and input into the model. Then, the input feature vector sequence is processed by BI-LSTM (Bi-directional Long Short Term Memory) NN (Neural Network) to extract context semantic learning features. Finally, the learned features are connected to the linear CRF (Conditional Random Field), the NEs (Named Entities) in the equipment support field are labeled, and the NER results are obtained and output. The experimental results show that the accuracy of the NER algorithm based on the BI-LSTM-CRF model is 92.02%, the recall rate is 93.21%, and the F1 value reaches 93.88%. The effect of this model is better than the BI-LSTM NN model and LSTM-CRF (Long Short Term Memory Conditional Random Field) NN model. The proposed model provides some references for entity recognition in the field of equipment support.

PurposeOne of the paramount challenges in the field of structural health monitoring (SHM) is the precise assessment of structural damage. Recently, many methods have been proposed to diagnose damage in structures using time series data. Nevertheless, these techniques are restricted in their ability to capture the bidirectional temporal and spatial relationships among the data. To remedy these shortcomings, this study proposes an advanced deep learning (DL) technique that integrates one-dimensional convolutional neural networks (1DCNN) with bidirectional long short-term memory (BiLSTM) networks, termed BiLSTM-1DCNN.Design/methodology/approachThis study proposes an advanced deep learning (DL) technique that integrates one-dimensional convolutional neural networks (1DCNN) with bidirectional long short-term memory (BiLSTM) networks, termed BiLSTM-1DCNN. The novelty of BiLSTM-1DCNN comes in its effective fusion of 1DCNN’s local feature extraction and BiLSTM’s capacity to capture long-term information from both directions (forward and backward).FindingsValidation using a dataset from the Nam O steel truss bridge demonstrates that the proposed method surpasses conventional models, including 1DCNN (87.9%), LSTM (72.5%), BiLSTM (76.1%) and LSTM-1DCNN (91.1%), achieving an accuracy rate of 96.4%. This approach provides a significant advancement in SHM, paving the way for more accurate and reliable damage detection.Research limitations/implicationsThe study’s limitations include potential overfitting due to the model’s complexity and the high computational resources required. The BiLSTM-1DCNN model’s performance is also influenced by the quality and quantity of the training data as well as the specific characteristics of the dataset. Additionally, while the model performs well in detecting damage in the Nam O steel truss bridge, its generalizability to other types of structures and environmental conditions remains untested.Practical implicationsThe BiLSTM-1DCNN model offers significant practical benefits for structural health monitoring by enhancing damage detection accuracy. Its ability to effectively capture both local features and long-term temporal dependencies makes it a valuable tool for real-time monitoring and early detection of structural issues. This can lead to more timely maintenance actions, potentially reducing repair costs and improving safety. The model’s high accuracy in detecting damage can also support infrastructure management decisions, contributing to better resource allocation and enhanced overall safety of critical structures.Social implicationsThe BiLSTM-1DCNN model’s advanced damage detection capabilities have positive social implications, such as improving the safety and reliability of public infrastructure like bridges. Enhanced structural health monitoring can reduce the risk of catastrophic failures, protecting lives and reducing accidents. By providing more accurate and timely assessments, the model can also lead to more efficient use of maintenance and repairs, ensuring that resources are allocated effectively.Originality/valueThe originality of the BiLSTM-1DCNN model lies in its innovative integration of one-dimensional convolutional neural networks (1DCNN) with bidirectional long short-term memory (BiLSTM) networks. This unique combination leverages 1DCNN’s strength in local feature extraction and BiLSTM’s capacity to capture temporal dependencies in both directions, addressing the limitations of existing methods. The model’s high accuracy in detecting structural damage, as demonstrated by its performance on the Nam O steel truss bridge dataset, highlights its value in advancing structural health monitoring technologies and offers a significant improvement over traditional approaches.

Applications in the World Wide Web aggregate vast amounts of information from different data sources. The aggregation process is often implemented with Extract, Transform and Load (ETL) processes. Usually ETL processes require information for aggregation available in structured formats, e. g. XML or JSON. In many cases the information is provided in natural language text which makes the application of ETL processes impractical. Due to the fact that information is provided in natural language, Information Extraction (IE) systems have been evolved. They make use of Natural Language Processing (NLP) tools to derive meaning from natural language text. State-of-the-art NLP tools apply Machine Learning methods. These NLP tools perform on newspapers with good quality, but they drop accuracy in other domains. However, to improve the quality for IE systems in specific domains often NLP tools are trained on domain specific text which is a time consuming process. This paper introduces an approach using a Continuous Integration pipeline for organizing and monitoring the annotation process on domain specific corpora.