Enhancing Bowtie Hazard Analysis

To develop and evaluate domain-specific and pretrained bidirectional encoder representations from transformers (BERT) models in a transfer learning task on varying training dataset sizes to annotate a larger overall dataset. The authors retrospectively reviewed 69 095 anonymized adult chest radiograph reports (reports dated April 2020-March 2021). From the overall cohort, 1004 reports were randomly selected and labeled for the presence or absence of each of the following devices: endotracheal tube (ETT), enterogastric tube (NGT, or Dobhoff tube), central venous catheter (CVC), and Swan-Ganz catheter (SGC). Pretrained transformer models (BERT, PubMedBERT, DistilBERT, RoBERTa, and DeBERTa) were trained, validated, and tested on 60%, 20%, and 20%, respectively, of these reports through fivefold cross-validation. Additional training involved varying dataset sizes with 5%, 10%, 15%, 20%, and 40% of the 1004 reports. The best-performing epochs were used to assess area under the receiver operating characteristic curve (AUC) and determine run time on the overall dataset. The highest average AUCs from fivefold cross-validation were 0.996 for ETT (RoBERTa), 0.994 for NGT (RoBERTa), 0.991 for CVC (PubMedBERT), and 0.98 for SGC (PubMedBERT). DeBERTa demonstrated the highest AUC for each support device trained on 5% of the training set. PubMedBERT showed a higher AUC with a decreasing training set size compared with BERT. Training and validation time was shortest for DistilBERT at 3 minutes 39 seconds on the annotated cohort. Pretrained and domain-specific transformer models required small training datasets and short training times to create a highly accurate final model that expedites autonomous annotation of large datasets.Keywords: Informatics, Named Entity Recognition, Transfer Learning Supplemental material is available for this article. ©RSNA, 2022See also the commentary by Zech in this issue.

Natural Language Processing (NLP) has been profoundly transformed by Artificial Intelligence (AI), particularly in developing academic chatbot systems. This study optimizes NLP for academic chatbots by implementing a domain-specific fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model. The primary objective is to enhance the chatbot's comprehension of conversational context, its accuracy in delivering academic information, and its pertinence in responding to inquiries from students and parents. The methodology involved collecting a corpus of 6,000 academic queries, followed by text pre-processing and fine-tuning the BERT model, with performance evaluated against traditional TF-IDF-SVM and LSTM baselines using accuracy and macro F1-score. The results demonstrate that the fine-tuned BERT model (Model C) achieved a superior accuracy of 95.8% and a macro F1-score of 0.95, significantly outperforming the LSTM (84.2% accuracy, 0.82 F1-score) and TF-IDF-SVM (61.5% accuracy, 0.59 F1-score) models. Furthermore, the model exhibited remarkable robustness, maintaining 93.5% accuracy on a challenging subset of queries with spelling irregularities, informal language, and grammatical errors. These findings indicate that the application of a domain-optimized BERT architecture effectively handles diverse and imperfect linguistic patterns, bridging the gap between generic language models and the specialized needs of the academic domain. The novelty of this study lies in its domain-specific fine-tuning of BERT for academic chatbot intent recognition, providing a replicable framework that can enhance institutional information systems and improve the quality of student and parent engagement.

This study aims to explore and develop tools for early identification of depression concerns among cancer patients by leveraging the novel data source of messages sent through a secure patient portal. We developed classifiers based on logistic regression (LR), support vector machines (SVMs), and 2 Bidirectional Encoder Representations from Transformers (BERT) models (original and Reddit-pretrained) on 6600 patient messages from a cancer center (2009-2022), annotated by a panel of healthcare professionals. Performance was compared using AUROC scores, and model fairness and explainability were examined. We also examined correlations between model predictions and depression diagnosis and treatment. BERT and RedditBERT attained AUROC scores of 0.88 and 0.86, respectively, compared to 0.79 for LR and 0.83 for SVM. BERT showed bigger differences in performance across sex, race, and ethnicity than RedditBERT. Patients who sent messages classified as concerning had a higher chance of receiving a depression diagnosis, a prescription for antidepressants, or a referral to the psycho-oncologist. Explanations from BERT and RedditBERT differed, with no clear preference from annotators. We show the potential of BERT and RedditBERT in identifying depression concerns in messages from cancer patients. Performance disparities across demographic groups highlight the need for careful consideration of potential biases. Further research is needed to address biases, evaluate real-world impacts, and ensure responsible integration into clinical settings. This work represents a significant methodological advancement in the early identification of depression concerns among cancer patients. Our work contributes to a route to reduce clinical burden while enhancing overall patient care, leveraging BERT-based models.

Deep learning approaches, utilizing Bidirectional Encoder Representation from Transformers (BERT) and advanced fine-tuning techniques, have achieved state-of-the-art accuracies in the domain of term extraction from texts. However, BERT presents some limitations in that it primarily captures the semantic context relative to the surrounding text without considering how relevant or central a token is to the overall document content. There has also been research on the application of sequence labeling on contextualized embeddings; however, the existing methods often rely solely on local context for extracting key phrases from texts. To address these limitations, this study proposes a centrality-weighted BERT model for key phrase extraction from text using sequence labelling (CenBERT-SEQ). The proposed CenBERT-SEQ model utilizes BERT to represent terms with various contextual embedding architectures, and introduces a centrality-weighting layer that integrates document-level context into BERT. This layer leverages document embeddings to influence the importance of each term based on its relevance to the entire document. Finally, a linear classifier layer is employed to model the dependencies between the outputs, thereby enhancing the accuracy of the CenBERT-SEQ model. The proposed CenBERT-SEQ model was evaluated against the standard BERT base-uncased model using three Computer Science article datasets, namely, SemEval-2010, WWW, and KDD. The experimental results show that, although the CenBERT-SEQ and BERT-base models achieved higher and close comparable accuracy, the proposed CenBERT-SEQ model achieved higher precision, recall, and F1-score than the BERT-base model. Furthermore, a comparison of the proposed CenBERT-SEQ model to that of related studies revealed that the proposed CenBERT-SEQ model achieved a higher accuracy, precision, recall, and F1-score of 95%, 97%, 91%, and 94%, respectively, than related studies, showing the superior capabilities of the CenBERT-SEQ model in keyphrase extraction from scientific documents.

With the development of Internet technology and the transformation of news media to informatization, news images, texts, and other information have exploded. The news image and text classification can effectively solve the disorder problem of news information. The early news image text classification is to establish artificial classifiers according to specific classification rules, but the classification error rate is high and the classification speed is slow. Later, machine learning technology replaced manual classification of news image texts. Although the classification efficiency has been greatly improved, the classification time is still a bit long. The bidirectional encoder representations from transformers (BERT) model uses transformer and encoder to pretrain news image text to improve classification efficiency. By comparing the differences between machine learning and BERT models in news image text classification, the experiments showed that the average precision, recall, and F1 values of the news image text classification algorithm using the BERT model were 96.6%, 95.7%, and 96.1%, respectively. All three evaluation criteria were about 5% more than the classification algorithm of the machine learning model. The classification speed of the news image text classification algorithm using the BERT model was 1.8 times that of the news image text classification algorithm based on support vector machine. Therefore, the news image text classification algorithm using the BERT model can improve the classification accuracy and efficiency of news image text.

The fast move to digital for HR tasks has led to the creation of a lot of unstructured documents, especially offer letters, which provide important information about employees. Using usual approaches, this has not been processed well for extracting information. This study presents a comprehensive system for the automatic extraction of information from offer letters utilizing optical character recognition (OCR) and named entity recognition. A comparison investigation has been conducted about the performance of the conventional Conditional Random Field (CRF) model and the advanced Bidirectional Encoder Representations from Transformers (BERT) model. This paper presents a system for generating synthetic data to address the challenges of data scarcity and privacy. This framework may create fake data for the process of getting information from offer letters, such as alternative layouts, semantic content, and noise. This paper evaluated the efficacy of the models on a demanding dataset of 500 documents. The findings indicate that, while the traditional Conditional Random Field model yielded satisfactory outcomes, the advanced Bidirectional Encoder Representations from Transformers model produced significantly superior results in comparison to the conventional model. This experimental evaluation also proves the efficiency of the proposed system in extracting key entities such as candidate name, designation, salary, and date of joining accurately from the scanned offer letters. The comparative analysis proves that, although the proposed system using the CRF model works accurately in structured data, the BERT model has a better capacity for understanding the contextual relationship within the document text. Thus, the proposed transformer model proves to be effective in dealing with the variability in the document layouts, linguistic expressions, and document conditions.

BackgroundIt is essential for radiologists to communicate actionable findings to the referring clinicians reliably. Natural language processing (NLP) has been shown to help identify free-text radiology reports including actionable findings. However, the application of recent deep learning techniques to radiology reports, which can improve the detection performance, has not been thoroughly examined. Moreover, free-text that clinicians input in the ordering form (order information) has seldom been used to identify actionable reports. This study aims to evaluate the benefits of two new approaches: (1) bidirectional encoder representations from transformers (BERT), a recent deep learning architecture in NLP, and (2) using order information in addition to radiology reports.MethodsWe performed a binary classification to distinguish actionable reports (i.e., radiology reports tagged as actionable in actual radiological practice) from non-actionable ones (those without an actionable tag). 90,923 Japanese radiology reports in our hospital were used, of which 788 (0.87%) were actionable. We evaluated four methods, statistical machine learning with logistic regression (LR) and with gradient boosting decision tree (GBDT), and deep learning with a bidirectional long short-term memory (LSTM) model and a publicly available Japanese BERT model. Each method was used with two different inputs, radiology reports alone and pairs of order information and radiology reports. Thus, eight experiments were conducted to examine the performance.ResultsWithout order information, BERT achieved the highest area under the precision-recall curve (AUPRC) of 0.5138, which showed a statistically significant improvement over LR, GBDT, and LSTM, and the highest area under the receiver operating characteristic curve (AUROC) of 0.9516. Simply coupling the order information with the radiology reports slightly increased the AUPRC of BERT but did not lead to a statistically significant improvement. This may be due to the complexity of clinical decisions made by radiologists.ConclusionsBERT was assumed to be useful to detect actionable reports. More sophisticated methods are required to use order information effectively.

Bidirectional Encoder Representations from Transformers (BERT) models achieve state-of-the-art performance on a number of Natural Language Processing tasks. However, their model size on disk often exceeds 1 GB and the process of fine-tuning them and using them to run inference consumes significant hardware resources and runtime. This makes them hard to deploy to production environments. This paper fine-tunes DistilBERT, a lightweight deep learning model, on medical text for the named entity recognition task of Protected Health Information (PHI) and medical concepts. This work provides a full assessment of the performance of DistilBERT in comparison with BERT models that were pre-trained on medical text. For Named Entity Recognition task of PHI, DistilBERT achieved almost the same results as medical versions of BERT in terms of F1 score at almost half the runtime and consuming approximately half the disk space. On the other hand, for the detection of medical concepts, DistilBERT's F1 score was lower by 4 points on average than medical BERT variants.

Natural Language Processing (NLP) is a pivotal area in artificial intelligence, aiming to make computers capable of understanding and generating human language. This study evaluates and compares three prominent NLP models—the Generative Pre-trained Transformer (GPT) model, Bidirectional Encoder Representations from Transformers (BERT) model, and Generalized Autoregressive Pretraining for Language Understanding (XLNet)—to determine their strengths, limitations, and suitability for various tasks. The research involves a comprehensive analysis of these models, utilizing well-established datasets such as the Stanford Question Answering Dataset (SQuAD), General Language Understanding Evaluation (GLUE), Reading Comprehension from Examinations (RACE), and the Situations with Adversarial Generations (SWAG). The study explores each model's architecture, pre-training, and fine-tuning processes: GPT’s unidirectional approach is assessed for its language generation and handling of long-range dependencies; Bidirectional encoding is examined for its effectiveness in context understanding, and XLNet permutation-based training is analyzed for its robust contextual comprehension. The experimental results reveal that GPT excels in generative tasks but is constrained by its unidirectional nature. BERT achieves superior accuracy in comprehension tasks but is computationally demanding and susceptible to pre-training bias. XLNet outperforms both GPT and BERT in accuracy and contextual understanding, though at the cost of increased complexity. The results offer a significant understanding of the effectiveness and applicability of these models, suggesting future research directions such as hybrid models and improvements in efficiency.

Multitask learning (MTL) and data augmentation are becoming increasingly popular in natural language processing (NLP). These techniques are particularly useful when data are scarce. In MTL, knowledge learned from one task is applied to another. To address data scarcity, data augmentation facilitates by providing additional synthetic data during model training. In NLP, the bidirectional encoder representations from transformers (BERT) model is the default candidate for various tasks. MTL and data augmentation using BERT have yielded promising results. However, a detailed study regarding the effect of using MTL in different layers of BERT and the benefit of data augmentation in these configurations has not been conducted. In this study, we investigate the use of MTL and data augmentation from generative models, specifically for category classification, sentiment classification, and aspect-opinion sequence-labeling using BERT. The layers of BERT are categorized into top, middle, and bottom layers, which are frozen, shared, or unshared. Experiments are conducted to identify the optimal layer configuration for improved performance compared with that of single-task learning. Generative models are used to generate augmented data, and experiments are performed to reveal their effectiveness. The results indicate the effectiveness of the MTL configuration compared with single-task learning as well as the effectiveness of data augmentation using generative models for classification tasks.

Artificial Intelligence (AI) functionalities have helped improve the working lives of the human population in sectors like the health sector, education sector, industries and tourism. Emerging economies, in particular, have been exploring the potential of AI to transform their tourism and hospitality sectors. This research paper aimed to examine the use of pre-trained AI models in the tourism and hospitality industries in emerging economies. The study presented a literature review of the current state of AI in the tourism and hospitality industries, including its opportunities and challenges. It then proposed the use of a pre-trained AI model, specifically the Bidirectional Encoder Representations from Transformers (BERT) model, to help address some of the challenges facing the tourism and hospitality sectors in emerging economies. BERT is a language model that can be fine-tuned for a range of natural language processing (NLP) tasks such as sentiment analysis, entity recognition and text classification. The paper discussed the potential applications of BERT in the tourism and hospitality sectors, such as analysing customer feedback, enhancing chatbot interactions and improving search engine optimization (SEO) for tourism websites. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) was used to evaluate the accuracy of the chatbot model that was used in this study. In conclusion, this research paper suggested that pre-trained AI models, such as BERT, can offer valuable insights and solutions in improving customer satisfaction in the tourism and hospitality industries in emerging economies.

News currently spreads rapidly through the internet. Because fake news stories are designed to attract readers, they tend to spread faster. For most readers, detecting fake news can be challenging and such readers usually end up believing that the fake news story is fact. Because fake news can be socially problematic, a model that automatically detects such fake news is required. In this paper, we focus on data-driven automatic fake news detection methods. We first apply the Bidirectional Encoder Representations from Transformers model (BERT) model to detect fake news by analyzing the relationship between the headline and the body text of news. To further improve performance, additional news data are gathered and used to pre-train this model. We determine that the deep-contextualizing nature of BERT is best suited for this task and improves the 0.14 F-score over older state-of-the-art models.

To solve the problem that it is difficult to identify the key Chinese entity information in the hazard and operability analysis text, a deep neural network model based on bidirectional long short-term memory and conditional random field (BiLSTM-CRF) is proposed to identify key named entities in the text. In the word vector pre-training process, bidirectional encoder representation from transformers (BERT) model is used to pre-train word vectors instead of the static word vectors in the traditional word2vec model, and then obtain context-related dynamic word vectors, to improve the representational ability of word vectors, and solve the problem of word boundary division when word vectors are used in Chinese corpus training. This model has the ability of complete the task of Chinese named entity recognition. The F1 value on the test corpus reaches 93.31%, which is 21.82% higher than conditional random field (CRF) Baseline, and 4.49% higher than the traditional BiLSTM-CRF model. The experimental results show that the BERT-BiLSTM-CRF model is effective for the named entity recognition (NER) task of the hazard and operability analysis text, and it is helpful to automatically extract the relationship between the entities in the hazard and operability analysis text and build safety analysis knowledge graph.

With the rate at which the data is being generated, it is vital to use it and get some insights from it. When reporting on events, news expresses the opinions of news entities like people, locations, and things. In this paper, we obtain the sentiment of the news headlines using a new technique called transformers, in particular simple transformers that have been a significant advancement in the natural language processing field. Sentiment Analysis has been performed using the four transformers model. These models are pre-trained on extensive data, and we have fine-tuned them by training them on our own news headlines dataset. For our sentiment analysis task, classification models (specific simple transformer model) are used to classify news headlines as negative, neutral, positive. The idea behind taking four different models that are Bidirectional Encoder Representations from Transformers (BERT) base-cased [1], Robustly optimized BERT approach (RoBERTa) base [1], Distilled BERT (DistilBERT) base-cased [1], XLNet base-cased [1], is the different dataset on which they are pre-trained, parameters used by them, and different method used by them, which improve their performance significantly in comparison to different machine learning classifiers and prior deep learning models. The model that performed the best is bert-base-cased with Matthews Correlation Coefficient (MCC) score of 90.1%, an F1 score of 93.6%, and an Accuracy score of 93.6%.

Public opinion is important for decision-making on numerous occasions for national growth in democratic countries like Bangladesh, the USA, and India. Sentiment analysis is a technique used to determine the polarity of opinions expressed in a text. The more complex stage of sentiment analysis is known as Aspect-Based Sentiment Analysis (ABSA), where it is possible to ascertain both the actual topics being discussed by the speakers as well as the polarity of each opinion. Nowadays, people leave comments on a variety of websites, including social networking sites, online news sources, and even YouTube video comment sections, on a wide range of topics. ABSA can play a significant role in utilizing these comments for a variety of objectives, including academic, commercial, and socioeconomic development. In English and many other popular European languages, there are many datasets for ABSA, but the Bengali language has very few of them. As a result, ABSA research on Bengali is relatively rare. In this paper, we present a Bengali dataset that has been manually annotated with five aspects and their corresponding sentiment. A baseline evaluation was also carried out using the Bidirectional Encoder Representations from Transformers (BERT) model, with 97% aspect detection accuracy and 77% sentiment classification accuracy. For aspect detection, the F1-score was 0.97 and for sentiment classification, it was 0.77.