Sliding-Bert: Striding Towards Conversational Machine Comprehension in Long Contex

The domain of artificial intelligence revolutionizes the way in which humans interact with machines. Machine comprehension is one of the latest fields under natural language processing that holds the capability for huge improvement in artificial intelligence. Machine comprehension technique gives systems the ability to understand a passage given by user and answer questions asked from it, which is an evolved version of traditional question answering technique. Machine comprehension is a main technique that falls under the category of natural language understanding, which exposes the amount of understanding required for a model to find the area of interest from a passage. The scope for the implementation of this technique is very high in India due to the availability of different regional languages. This work focused on the incorporation of machine comprehension technique in code-mixed Hindi language. A detailed comparison study on the performance of dataset in several deep learning approaches including End to End Memory Network, Dynamic Memory Network, Recurrent Neural Network, Long Short-Term Memory Network and Gated Recurrent Unit are evaluated. The best suited model for the dataset used is identified from the comparison study. A new architecture is proposed in this work by combining two of the best performing networks. To improve the model with respect to various ways of answering questions from a passage the natural language processing technique of distributed word representation was performed on the best model identified. The model was improved by applying pre-trained fastText embeddings for word representations. This is the first implementation of machine comprehension models in code-mixed Hindi language using deep neural networks. The work analyses the performance of all five models implemented, which will be helpful for future researches on Machine Comprehension technique in code-mixed Indian languages.

NLP, short for Natural Language Processing, refers to a domain of research that centers on the interaction between computers and human language. Its objective is to devise algorithms and models that empower machines to comprehend, interpret, and produce human language in a coherent and meaningful manner. NLP is of great importance due to its ability to enable effective communication, facilitate information retrieval and extraction, analyse sentiment and opinions, break down language barriers, summarize text, recognize named entities, extract structured information, generate human-like text, process voice commands, and support cognitive computing. As NLP continues to advance, it holds immense potential for transforming various industries and enhancing human-machine interactions across numerous domains. This study emphasizes the significance of NLP in narrowing the divide between human language and machine comprehension, resulting in a broad spectrum of applications and advantages.

Deep neural networks reach state-of-the-art performance for wide range of natural language processing, computer vision and speech applications. Yet, one of the biggest challenges is running these complex networks on devices such as mobile phones or smart watches with tiny memory footprint and low computational capacity. We propose on-device Self-Governing Neural Networks (SGNNs), which learn compact projection vectors with local sensitive hashing. The key advantage of SGNNs over existing work is that they surmount the need for pre-trained word embeddings and complex networks with huge parameters. We conduct extensive evaluation on dialog act classification and show significant improvement over state-of-the-art results. Our findings show that SGNNs are effective at capturing low-dimensional semantic text representations, while maintaining high accuracy.

Pretrained language models have achieved great success in a wide range of natural language processing (NLP) problems, because they learn language representations from large-scale text corpora and can adapt to downstream tasks by finetuning them on annotated task data. However, such success relies on both large-scale text and annotated data, so the lack of training data is a major practical problem for many languages, especially low-resource languages. In this paper, we explore whether a pretrained English language model can benefit non-English NLP systems in low-resource scenarios, i.e., with limited text corpora or annotated data. To achieve this, we first propose cross-lingual knowledge transfer methods and then validate our methods in low-resource scenarios. Specifically, our cross-lingual knowledge transfer methods are applied in the training stages of language model pretraining or downstream finetuning. At the two stages, the methods are designed for the transfer of upstream general knowledge or downstream task-specific knowledge, respectively. In the experiments, we perform pretraining and finetuning with limited non-English data to simulate the low-resource scenarios. We evaluate our methods on ten downstream tasks over a wide range of languages, and present systematic comparisons among various knowledge transfer methods. Experimental results show that our methods successfully leverage a pretrained English language model to improve task performance in other languages. Besides, we demonstrate the multilinguality of the English language model in various application scenarios. Our findings imply the possibility to improve low-resource-language NLP systems with large-scale English language models.

Chapter XIV - QUESTION ANSWERING

Machine Comprehension or Question Answering (QA) is one of the most challenging natural language processing tasks due to the language’s dynamic nature and understanding the context of the question. In this paper, we propose a similarity attention layer with an aim to reduce human labor by automating tedious QA tasks using the attention mechanism in deep learning model; it uses attention scores and obtains good results even without pre-training. The QA using attention has immense scope in search engine optimization, page ranking, and chatbots. The traditional rule-based models and statistical methods underperform due to variations in the language. This dynamic nature of the language is well captured by the nonlinear learning of the neural networks. The conventional encoder-decoder architecture of neural networks for QA works well in the case of short sentences. However, the performance comes down for paragraphs and very long sentences as it is difficult for the network to memorize the super-long sentences. In contrast, the attention model helps the network focus on smaller attention areas in the complex input paragraph, part by part, until the entire text is processed. The results are very promising; our (single) model outperforms the existing ensemble method too.

Word embedding is the state-of-the-art representation to capture semantic information of terms. It benefits a wide range of natural language processing and related applications, not only in general fields of artificial intelligence but also in bioinformatics. Although recent efforts of using word embedding to represent medical concepts have provided remarkable analyses, many essential problems remain unsolved. Examples include representation of complex concepts (i.e., formed by multiple tokens), leveraging of a large corpus to maximize the trainable concepts, and downstream analyses on a biomedical-related dataset. Our study focused on training effective representations for biomedical concepts including complex ones. We used an efficient technique to index all possible concepts of UMLS thesaurus (Unified Medical Language System) in a huge corpus of 15,4 billion tokens. By this way, we can obtain the vector representations for more than 650,000 concepts, the largest ever reported resource to date. Furthermore, evaluations of trained vectors on retrieval task show superior performance compared to recent studies.

A user can scan through a text easily, but it is not the case for spoken content, because they cannot be directly displayed on-screen. As a result, accessing large collections of spoken content is much more difficult and time-consuming than doing so for the text content. It would therefore be helpful to develop machines that understand spoken content. In this paper, we propose two new tasks for machine comprehension of spoken content. The first is a listening comprehension test for TOEFL, a challenging academic English examination for English learners who are not the native English speakers. We show that the proposed model outperforms the naive approaches and other neural network based models by exploiting the hierarchical structures of natural languages and the selective power of attention mechanism. For the second listening comprehension task - spoken SQuAD - we find that speech recognition errors severely impair machine comprehension; we propose the use of subword units to mitigate the impact of these errors.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

On a wide range of natural language processing and information retrieval tasks, transformer-based models, particularly pre-trained language models like BERT, have demonstrated tremendous effectiveness. Due to the quadratic complexity of the self-attention mechanism, however, such models have difficulties processing long documents. Recent works dealing with this issue include truncating long documents, in which case one loses potential relevant information, segmenting them into several passages, which may lead to miss some information and high computational complexity when the number of passages is large, or modifying the self-attention mechanism to make it sparser as in sparse-attention models, at the risk again of missing some information. We follow here a slightly different approach in which one first selects key blocks of a long document by local query-block pre-ranking, and then few blocks are aggregated to form a short document that can be processed by a model such as BERT. Experiments conducted on standard Information Retrieval datasets demonstrate the effectiveness of the proposed approach.

Semantic textual similarity(STS) and natural language inference(NLI) are important tasks in natural language processing(NLP) such as information retrieval, text classification, subject extraction, text summarization, machine translation and plagiarism detection. Lack of appropriate datasets in the Persian language is a major obstacle to progress in this area. Therefore, in this paper, we present FarSick, a new dataset for STS and NLI tasks in the Persian language. FarSick is the first relatively large-scale STS dataset for the Persian language. It includes 9804 pairs of Persian sentences with labels for similarity and inference for each pair of sentences. This dataset is collected by translating and editing the sentences of SICK dataset. We also measured the performance of traditional, statistical and deep learning models on it, e.g. transformers, Convolution Neural Networks, Bidirectional LSTMs, weighted average of word vectors, etc. We used different pre-trained embeddings, word2vec, glove, fastText and Bert sentence transformer. We used accuracy metric to test NLI tasks and Pearson metric to test STS tasks. The dataset is available at https://github.com/ZahraGhasemi-AI/FarSick.

BackgroundClinical trial protocols are the foundation for advancing medical sciences, however, the extraction of accurate and meaningful information from the original clinical trials is very challenging due to the complex and unstructured texts of such documents. Named entity recognition (NER) is a fundamental and necessary step to process and standardize the unstructured text in clinical trials using Natural Language Processing (NLP) techniques.MethodsIn this study we fine-tuned pre-trained language models to support the NER task on clinical trial eligibility criteria. We systematically investigated four pre-trained contextual embedding models for the biomedical domain (i.e., BioBERT, BlueBERT, PubMedBERT, and SciBERT) and two models for the open domains (BERT and SpanBERT), for NER tasks using three existing clinical trial eligibility criteria corpora. In addition, we also investigated the feasibility of data augmentation approaches and evaluated their performance.ResultsOur evaluation results using tenfold cross-validation show that domain-specific transformer models achieved better performance than the general transformer models, with the best performance obtained by the PubMedBERT model (F1-scores of 0.715, 0.836, and 0.622 for the three corpora respectively). The data augmentation results show that it is feasible to leverage additional corpora to improve NER performance.ConclusionsFindings from this study not only demonstrate the importance of contextual embeddings trained from domain-specific corpora, but also shed lights on the benefits of leveraging multiple data sources for the challenging NER task in clinical trial eligibility criteria text.

With growing applications of Machine Learning in daily lives Natural Language Processing (NLP) has emerged as a heavily researched area. Finding its applications in tasks ranging from simple Q/A chatbots to Fully fledged conversational AI, NLP models are vital. Word and Sentence embedding are one of the most common starting points of any NLP task. A word embedding represents a given word in a predefined vector-space while maintaining vector relations with similar or dis-similar entities. As such different pretrained embedding such as Word2Vec, GloVe, fasttext have been developed. These embedding generated on millions of words are however very large in terms of size. Having embedding with floating point precision also makes the downstream evaluation slow. In this paper we present a novel method to convert continuous embedding to its binary representation, thus reducing the overall size of the embedding while keeping the semantic and relational knowledge intact. This will facilitate an option of porting such big embedding onto devices where space is limited. We also present different approaches suitable for different downstream tasks based on the requirement of contextual and semantic information. Experiments have shown comparable result in downstream tasks with 7 to 15 times reduction in file size and about 5 % change in evaluation parameters.

Sentiment analysis, a crucial aspect of Natural Language Processing (NLP), plays a pivotal role in understanding public opinion, customer feedback, and user sentiments in various domains. In this study, we present a comprehensive approach to sentiment analysis that incorporates both textual and emoji data, leveraging diverse datasets from sources such as social media, customer reviews, and surveys. Our methodology consists of several key steps, including data collection, pre-processing, feature extraction, feature fusion, and feature selection. For data pre-processing, we apply techniques such as tokenization, lowercasing, stop word removal, and stemming to ensure uniform and meaningful text representation. We also extract emojis from text using regular expressions and convert them into textual representations, facilitating unified processing. Text data is transformed into feature vectors using Term frequency_ Inverse Term frequency (TF-IDF) weighting-based Bag of Words (TF-IDF_BoW) with word frequencies, pre-trained Word2Vec word embeddings, and n-grams to capture local word patterns. Emoji data, on the other hand, is processed using pre-trained emoji embeddings (Emoji2Vec) and emoji frequency counting. The Feature Fusion stage involves the combination of text and emoji features into a single feature vector, with a weighted concatenation approach. In the Feature Selection phase, introduced a Self-Improved Siberian Tiger Optimization (SI-STO), a novel feature selection technique, to identify the most relevant features for sentiment prediction. The sentiment classification model SentEmoNet combines LSTM with an Attention Mechanism and a CNN for capturing text patterns and extracting local features from text and emoji embeddings. We use standard sentiment analysis metrics to assess performance in the model evaluation phase. The proposed model has achieved an accuracy of 97.87%, which is better than the existing ones.

Introduction: Tumor single-cell data provide unique insights into the cellular dynamics of cancer progression and therapeutic outcomes. Inspired by advancements in natural language processing (NLP), foundation models are increasingly used in single-cell analyses to uncover biological insights and identify clinical biomarkers. However, the performance of these models on cancer-specific tasks remains underexplored. We present an extensive evaluation of foundation models trained on millions of single-cell profiles for cancer-focused investigations. Methods: We evaluated nine predictive models-three traditional methods and six foundation models (four Geneformer and two scGPT models), across various downstream tasks. Traditional models used embeddings from high variable genes (HVG), principal components (PCA), and generative model embedding (scVI), each paired with a random forest classifier for downstream classification. The evaluated foundation models varied in architecture sizes, pretraining dataset sizes (30M to 95M cell profiles), and training datasets, which were either generic or cancer-specific. We assessed the quality of representations by measuring their predictive performance on lung cancer treatment status (TKI treated vs. treatment-naive patients) and breast cancer subtypes (ER+ vs. TNBC). We examined how model characteristics influenced their performance in these domains for all tasks. Results: Zero-shot evaluations of the foundation models showed superior single-cell data clustering, measured by average silhouette width (ASW), compared to traditional methods (HVG, PCA, scVI), suggesting that pre-trained embeddings from millions of single-cell profiles may offer enhanced representations. We thus evaluated the learned representation on downstream predictive tasks, showing that foundation models with a larger pretraining dataset (95M cells), cancer-specific training, larger input size (4096 genes), and deeper model (12 layers) achieved the best performance among foundation models (F1=0.97) for predicting treatment status in lung cancer. In contrast, smaller and generic foundation models had significantly lower performance, however, additional self-supervised training on task-specific data rescued their performance to match larger foundation models. Similar trends were observed in predicting breast cancer subtypes. Notably, the best foundation model’s performance was not significantly different from the best traditional method (PCA) in predicting breast cancer subtypes. Conclusions: These results highlight key factors driving foundation model performance on cancer-specific tasks. While current foundation models show inconsistent contributions to cancer-specific questions, observed performance trends highlight their potential to enhance single-cell data representation and drive significant advancements in cancer research and precision medicine. Citation Format: Haitham Elmarakeby, Ahmed Roman, Shreya Johri, Eliezer M. Van Allen. Evaluation of single-cell foundation models for cancer outcome predictions [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 5084.