Question Answering-based Socio-economic Indicator Extraction

Machine Reading Comprehension(MRC), including a series of tasks that test the ability of models to understand natural language, has received quite a few attention in Natural Language Processing(NLP). Most existing works deal with MRC tasks by exploiting the expression capability of neural networks. Some of them have achieved impressive performance. Despite the rapid iteration of the models used, few work have focused on output layer and prediction method of answer span - also known as span extraction. In this paper, we focus on span extraction in the Question Answering(QA) task. A cross-sectional comparison of widely used span extraction methods is presented, with their strengths and weaknesses noted in detail. Furthermore, inspired by Faster R-CNN, we propose a brand new span extraction method. Experiment results show that our proposed method outperforms existing span extraction methods on both English and Chinese MRC tasks.

In the era of digitalization, electronic gadgets such as Google Translate, Siri, and Alexa have at least one characteristic: They are all the products of natural language processing (NLP). “Natural Language” refers to a human language used for daily communication, such as English, Hindi, Bengali, etc. Natural languages, as opposed to artificial languages such as computer languages and mathematical nomenclature, have evolved as they have been transmitted from generation to generation and are challenging to explain with clear limits in the first instance. In natural language processing, artificial intelligence (Singh et al., 2021), linguistics, information processing, and cognitive science are all related fields (NLP). NLP aims to use intelligent computer techniques to process human language. However, NLP technologies such as voice recognition, language comprehension, and machine translation exist. With such limited obvious exclusions, machine learning algorithms in NLP sometimes lacked sufficient capacity to consume massive amounts of training data. In addition, the algorithms, techniques, and infrastructural facilities lack enough strength. Humans design features in traditional machine learning, and feature engineering is a limitation that requires significant human expertise. Simultaneously, the accompanying superficial algorithms lack depiction capability and, as a result, the ability to generate layers of duplicatable concepts that would naturally separate intricate aspects in forming visible linguistic data. Deep learning overcomes the challenges mentioned earlier by using deep, layered modelling architectures, often using neural networks and the corresponding full-stack learning methods. Deep learning has recently enhanced natural language processing by using artificial neural networks based on biological brain systems and Backpropagation. Deep learning approaches that use several processing layers to develop hierarchy data representations have produced cutting-edge results in various areas. This chapter introduces natural language processing (NLP) as an AI component. The history of NLP is next. Distributed language representations are the core of NLP's profound learning revolution. After the survey, the boundaries of deep learning for NLP are investigated. The paper proposes five NLP scientific fields.

Learning a similarity function between pairs of objects is at the core of learning to rank approaches. In information retrieval tasks we typically deal with query-document pairs, in question answering -- question-answer pairs. However, before learning can take place, such pairs needs to be mapped from the original space of symbolic words into some feature space encoding various aspects of their relatedness, e.g. lexical, syntactic and semantic. Feature engineering is often a laborious task and may require external knowledge sources that are not always available or difficult to obtain. Recently, deep learning approaches have gained a lot of attention from the research community and industry for their ability to automatically learn optimal feature representation for a given task, while claiming state-of-the-art performance in many tasks in computer vision, speech recognition and natural language processing. In this paper, we present a convolutional neural network architecture for reranking pairs of short texts, where we learn the optimal representation of text pairs and a similarity function to relate them in a supervised way from the available training data. Our network takes only words in the input, thus requiring minimal preprocessing. In particular, we consider the task of reranking short text pairs where elements of the pair are sentences. We test our deep learning system on two popular retrieval tasks from TREC: Question Answering and Microblog Retrieval. Our model demonstrates strong performance on the first task beating previous state-of-the-art systems by about 3\% absolute points in both MAP and MRR and shows comparable results on tweet reranking, while enjoying the benefits of no manual feature engineering and no additional syntactic parsers.

Machine Reading Comprehension (MRC) is the process where computers or, machines are taught to understand a paragraph or more technically called a context. Like humans, machines also need to be evaluated for their understanding on question answering. MRC is one of the formidable sub-domains in the Natural Language Processing (NLP) domain, which has seen considerable progress over the years. In recent years, many novel datasets have tried to challenge the Machine Reading Comprehension (MRC) models with inference based question answering. With the advancement in NLP, many models have surpassed human-level performance on these datasets, albeit ignoring the obvious disparity between genuine human-level performance and state-of-the-art performance. This highlights the need for attention on the collective improvement of existing datasets, metrics, and models towards “real” prehension. Addressing the lack of sanity in the domain, this paper performs a comparative study on various transformer based models and tries to highlight the success factors of each model. Subsequently, we discuss an MRC model that performs comparatively better, if not the best, on question answering and give directions for future research.KeywordsTransformerNatural language processingReading comprehensionQuestion answeringDeep learning

In this paper, we detail our submission to the 7th year BioASQ competition. We present our approach for Task-7b, Phase B, Exact Answering Task. These Question Answering (QA) tasks include Factoid, Yes/No, List Type Question answering. Our system is based on a contextual word embedding model. We have used a Bidirectional Encoder Representations from Transformers (BERT) based system, fined tuned for biomedical question answering task using BioBERT. In the third test batch set, our system achieved the highest ‘MRR’ score for Factoid Question Answering task. Also, for List type question answering task our system achieved the highest recall score in the fourth test batch set. Along with our detailed approach, we present the results for our submissions, and also highlight identified downsides for our current approach and ways to improve them in our future experiments.

BackgroundExtractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared with an average open-domain context. This causes the MRC model to make less accurate and slower predictions. A potential solution to this problem is to reduce the input context of the MRC model by extracting only the necessary parts from the original context.ObjectiveThis study aims to develop a method for extracting useful contexts from long articles as an additional component to the question answering task, enabling the MRC model to work more efficiently and accurately.MethodsExisting approaches to context extraction in MRC are based on sentence selection strategies, in which the models are trained to find the sentences containing the answer. We found that using only the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relationship between the usefulness of the context and the confidence score output of the MRC model. Our investigation showed that a precise input context can boost the prediction correctness of the MRC and greatly reduce inference time. We proposed a method to estimate the utility of each sentence in a context in answering the question and then extract a new, shorter context according to these estimations. We generated a data set to train 2 models for estimating sentence utility, based on which we selected more precise contexts that improved the MRC model’s performance.ResultsWe demonstrated our approach on the Question Answering Data Set for COVID-19 and Biomedical Semantic Indexing and Question Answering data sets and showed that the approach benefits the downstream MRC model. First, the method substantially reduced the inference time of the entire question answering system by 6 to 7 times. Second, our approach helped the MRC model predict the answer more correctly compared with using the original context (F1-score increased from 0.724 to 0.744 for the Question Answering Data Set for COVID-19 and from 0.651 to 0.704 for the Biomedical Semantic Indexing and Question Answering). We also found a potential problem where extractive transformer MRC models predict poorly despite being given a more precise context in some cases.ConclusionsThe proposed context extraction method allows the MRC model to achieve improved prediction correctness and a significantly reduced MRC inference time. This approach works technically with any MRC model and has potential in tasks involving processing long texts.

The rising complexity of cyberattacks poses significant challenges to traditional intrusion detection system (IDS). Approaches that rely mainly on feature engineering and rule matching often fail to address new and complex attack patterns effectively. To address these challenges, this paper proposes a network IDS based on bidirectional encoder representations from transformers (BERT), which aims to leverage recent advances in the field of natural language processing (NLP) to enhance the performance of IDS. The core of our approach lies in converting network traffic data into textual representations that can be processed by applying advanced NLP techniques. Specifically, we first convert various types of information in network traffic into a natural-language-like sequence representation, which allows us to utilize a pre-trained BERT model for feature extraction. The BERT model, with its bidirectional encoder structure, captures complex contextual information in sequences, producing high-quality feature representations. After feature extraction is completed, we further perform classification using a classifier based on a bidirectional gated recurrent unit (GRU) network, which is able to efficiently process sequence data and reduce computational complexity while capturing temporal dependencies in the sequences. By training the classifier on the basis of the extracted features, we are able to achieve an accurate distinction between normal and abnormal traffic. To validate the effectiveness of our approach, we conducted experiments on several public datasets, and the experimental results show that the BERT-based network IDS significantly outperforms traditional machine learning (ML) and deep learning (DL) methods in terms of detection performance. The BERT model enhances detection accuracy and effectively captures complex attack patterns, compared to traditional methods. Our research not only demonstrates the great potential of NLP technology in the field of network security but also provides a new research direction for cross-domain applications. By treating network traffic as a “language,” our approach resolves issues that traditional methods struggle with while being scalable and adaptable to various network environments and attack patterns.

Machine Reading Comprehension (MRC) is a challenging task in Natural Language Processing (NLP) field. It is a fast evolving field in the recent years because of the development of large-scale datasets and the deep learning technology. Even though, numerous MRC models have been developed for Question Answer (QA) system, the existing MRC models are not on par with reading comprehension. In order to assess and improve the performance of the projected datasets, an advanced level of evaluation metrics have been introduced. Therefore, this work has conducted a survey over the existing MRC models and evaluation metrics utilized in them to address the lack of MRC with human reading.

Reading comprehension is one of the most critical abilities in human intelligence, and machine reading comprehension is one of the most popular research directions in the field of natural language processing. Machine reading is one of the key aspects to achieve artificial intelligence. This paper first analyzes the current state of development of machine reading comprehension, and based on the current problems in machine reading comprehension, this paper applies various attention mechanisms in the reading comprehension model, so that the information of articles and questions can be fused with each other to extract the key information and predict the answers more accurately. This paper also designs experiments and verifies that this method has a higher accuracy rate compared with other methods.

POS tagging is a basic Natural Language Processing (NLP) task that tags the words in an input text according to its grammatical values. Although POS Tagging is a fundamental application for very resourced languages, such as Limbu, is still unknown due to only few tagged datasets and linguistic resources. This research project uses deep learning techniques, transfer learning, and the BiLSTM-CRF model to develop an accurate POS-tagging system for the Limbu language. Using annotated and unannotated language data, we progress in achieving a small yet informative dataset of Limbu text. Skilled multilingual tutoring was modified to enhance success on low-resource language tests. The model as propose attains 90% accuracy, which is very much better than traditional rule-based and machine learning methods for Limbu POS tagging. The results indicate that deep learning methods can address linguistic issues facing low-resource languages even with limited data. In turn, this study provides a cornerstone for follow up NLP-based applications of Limbu and similar low-resource languages, demonstrating how deep learning can fill the gap where data is scarce.

Investigating the learning approaches of students in nursing education

In recent years, domain generation algorithms (DGAs) are the foundational mechanisms for many malware families. Mainly, due to the fact that DGA can generate immense number of pseudo random domain names to associate to a command and control (C2) infrastructures. This paper focuses on to detect and classify the pseudo random domain names without relying on the feature engineering or any other linguistic, contextual or semantics and statistical information by adopting deep learning approaches. A deep learning approach is a complex model of traditional machine learning mechanism that has received renewed interest by solving the long-standing tasks in artificial intelligence (AI) related to the field of natural language processing, image recognition, speech processing and many others. They have immense capability to extract optimal feature representations by taking input as in the form of raw input texts. To leverage this and to transfer the performance enhancement in aforementioned areas towards characterize, detect and classify the DGA generated domain names to a specific malware family, this paper adopts deep learning mechanisms with a known one million benign domain names from Alexa, OpenDNS and a corpus of malicious domain names generated from 17 DGA malware families in real time for training in character and bigram level and a trained model has been evaluated on the OSNIT data set in real-time. Specifically, to understand the effectiveness of various deep learning mechanisms, we used recurrent neural network (RNN), identity-recurrent neural network (I-RNN), long short-term memory (LSTM), convolution neural network (CNN), and convolutional neural network-long short-term memory (CNN-LSTM) architectures. Additionally, to find out an optimal architecture, experiments are done with various configurations of network parameters and network structures. All experiments run up to 1000 epochs with a learning rate set in the range [0.01-0.5]. Overall, deep learning approaches, particularly family of recurrent neural network and a hybrid network (where the first layer is CNN and a subsequent layer is LSTM) have showed significant performance with a highest detection rate 0.9945 and 0.9879 respectively. The main reason is deep learning approaches have inherent mechanisms to capture hierarchical feature extraction and long range-dependencies in sequence inputs.

• Novel FlexiCombFE framework for brain tumor detection using MRI images. • Combines flexible patch-based feature extraction with multiple selectors. • Achieves 99.35% accuracy, outperforming several deep learning approaches. • Modular design allows detailed analysis of classification effects. • Provides interpretable feature engineering alternative to deep learning. Deep learning models are prevalently employed for image classification, but significant opportunities remain within the domain of feature engineering. This study introduces FlexiCombFE, a novel, flexible, combination-based feature engineering framework for brain tumor detection using various fixed-size patch divisions. This framework employs three distinct feature extractors (Local Phase Quantization, Local Binary Pattern, and Pyramidal Histogram of Oriented Gradients) to generate a total of seven primary feature vectors. By using four types of fixed-size patch divisions, 28 feature vectors are generated. Then, three feature selectors (Chi-squared, Neighborhood Component Analysis, and ReliefF) create 84 selected feature vectors. In the classification phase K-nearest neighbors and support vector machine classifiers yield 168 classifier-specific outcomes. An information fusion generates 166-voted outcomes, with the most accurate classification outcome selected as the final output. This self-organizing feature engineering model achieved a classification accuracy of 99.35% on a brain tumor image dataset, outperforming several deep learning approaches. The modular design of the proposed framework allows for detailed analysis of the classification effects of individual methods, providing optimal feature engineering strategies for medical image classification tasks.

Machine reading comprehension (MRC) is a challenging natural language processing (NLP) task. It has a wide application potential in the fields of question answering robots, human‐computer interactions in mobile virtual reality systems, etc. Recently, the emergence of pretrained models (PTMs) has brought this research field into a new era, in which the training objective plays a key role. The masked language model (MLM) is a self‐supervised training objective widely used in various PTMs. With the development of training objectives, many variants of MLM have been proposed, such as whole word masking, entity masking, phrase masking, and span masking. In different MLMs, the length of the masked tokens is different. Similarly, in different machine reading comprehension tasks, the length of the answer is also different, and the answer is often a word, phrase, or sentence. Thus, in MRC tasks with different answer lengths, whether the length of MLM is related to performance is a question worth studying. If this hypothesis is true, it can guide us on how to pretrain the MLM with a relatively suitable mask length distribution for MRC tasks. In this paper, we try to uncover how much of MLM’s success in the machine reading comprehension tasks comes from the correlation between masking length distribution and answer length in the MRC dataset. In order to address this issue, herein, (1) we propose four MRC tasks with different answer length distributions, namely, the short span extraction task, long span extraction task, short multiple‐choice cloze task, and long multiple‐choice cloze task; (2) four Chinese MRC datasets are created for these tasks; (3) we also have pretrained four masked language models according to the answer length distributions of these datasets; and (4) ablation experiments are conducted on the datasets to verify our hypothesis. The experimental results demonstrate that our hypothesis is true. On four different machine reading comprehension datasets, the performance of the model with correlation length distribution surpasses the model without correlation.

Purpose:There has been burgeoning interest in applying machine learning methods for predicting radiotherapy outcomes. However, the imbalanced ratio of a large number of variables to a limited sample size in radiation oncology constitutes a major challenge. Therefore, dimensionality reduction methods can be a key to success. The study investigates and contrasts the application of traditional machine learning methods and deep learning approaches for outcome modeling in radiotherapy. In particular, new joint architectures based on variational autoencoder (VAE) for dimensionality reduction are presented and their application is demonstrated for the prediction of lung radiation pneumonitis (RP) from a large-scale heterogeneous dataset.Methods:A large-scale heterogeneous dataset containing a pool of 230 variables including clinical factors (e.g., dose, KPS, stage) and biomarkers (e.g., single nucleotide polymorphisms (SNPs), cytokines and micro-RNAs) in a population of 106 non-small cell lung cancer (NSCLC) patients who received radiotherapy was used for modeling RP. 22 patients had grade 2 or higher RP. Four methods were investigated, including feature selection (case A) and feature extraction (case B) with traditional machine learning methods, a VAE-MLP joint architecture (case C) with deep learning and lastly, the combination of feature selection and joint-architecture (case D). For feature selection, Random forest (RF), Support Vector Machine (SVM) and multi-layer perceptron (MLP) were implemented to select relevant features. Specifically, each method was run for multiple times to rank features within several cross-validated (CV) resampled sets. A collection of ranking lists were then aggregated by top 5% and Kemeny graph methods to identify the final ranking for prediction. A synthetic minority oversampling technique (SMOTE) was applied to correct for class imbalance during this process. For deep learning, a VAE-MLP joint architecture where a VAE aimed for dimensionality reduction and an MLP aimed for classification was developed. In this architecture, reconstruction loss and prediction loss were combined into a single loss function to realize simultaneous training and weights were assigned to different classes to mitigate class imbalance. To evaluate the prediction performance and conduct comparisons, the area under receiver operating characteristic (ROC) curves (AUCs) were performed for nested CVs for both handcrafted feature selections and the deep learning approach. The significance of differences in AUCs was assessed using the DeLong test of U-statistics.Results:An MLP-based method using weight pruning (WP) feature selection yielded the best performance among the different hand-crafted feature selection methods (case A), reaching an AUC of 0.804 (95% CI: 0.761–0.823) with 29 top features. A VAE-MLP joint architecture (case C) achieved a comparable but slightly lower AUC of 0.781 (95% CI: 0.737–0.808) with the size of latent dimension being 2. The combination of handcrafted features (case A) and latent representation (case D) achieved a significant AUC improvement of 0.831 (95% CI: 0.805–0.863) with 22 features (p-value = 0.000642 compared with handcrafted features only (Case A) and p-value=0.000453 compared to VAE alone (Case C)) with an MLP classifier.Conclusion:The potential for combination of traditional machine learning methods and deep learning VAE techniques has been demonstrated for dealing with limited datasets in modeling radiotherapy toxicities. Specifically, latent variables from a VAE-MLP joint architecture are able to complement handcrafted features for the prediction of RP and improve prediction over either method alone.