MRAT-SQL+GAP: A Portuguese Text-to-SQL Transformer

Large pretrained masked language models have become state-of-the-art solutions for many NLP problems. While studies have shown that monolingual models produce better results than multilingual models, the training datasets must be sufficiently large. We trained a trilingual LitLat BERT-like model for Lithuanian, Latvian, and English, and a monolingual Est-RoBERTa model for Estonian. We evaluate their performance on four downstream tasks: named entity recognition, dependency parsing, part-of-speech tagging, and word analogy. To analyze the importance of focusing on a single language and the importance of a large training set, we compare created models with existing monolingual and multilingual BERT models for Estonian, Latvian, and Lithuanian. The results show that the newly created LitLat BERT and Est-RoBERTa models improve the results of existing models on all tested tasks in most situations.KeywordsNatural language processingBERTTransformersEstonianLatvianLithuanian

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

Multilingual word vector technology has experienced an evolution from word alignment to unsupervised alignment to multilingual aligned word vectors; and language modelling has evolved from monolingual models to multilingual models based on the Transformer architecture. Both of them are synergistically applied to improve cross-language generalization capability and low-resource language modelling, showing significant advantages. The synergistic evolution of multilingual word vectors and language models promotes the development of cross-language intelligent systems, which will further enhance the machine's ability to understand and generate multilingual scenarios in the future.

Terminology in Speech-Language and Hearing Science. To propose a specific thesaurus about the Speech-Language and Hearing Science, for the English, Portuguese and Spanish languages, based on the existing keywords available on the Health Sciences Descriptors (DeCS). Methodology was based on the pilot study developed by Campanatti-Ostiz and Andrade; that had as a purpose to verify the methodological viability for the creation of a Speech-Language and Hearing Science category in the DeCS. The scientific journals selected for analyses of the titles, abstracts and keywords of all scientific articles were those in the field of the Speech-Language and Hearing Science, indexed on the SciELO. 1. Recovery of the Descriptors in the English language (Medical Subject Headings--MeSH); 2. Recovery and hierarchic organization of the descriptors in the Portuguese language was done (DeCS). The obtained data was analyzed as follows: descriptive analyses and relative relevance analyses of the DeCS areas. Based on the first analyses, we decided to select all 761 descriptors, with all the hierarchic numbers, independently of their occurrence (occurrence number--ON), and based on the second analyses, we decided to propose to exclude the less relevant areas and the exclusive DeCS areas. The proposal was finished with a total of 1676 occurrences of DeCS descriptors, distributed in the following areas: Anatomy; Diseases; Analytical, Diagnostic and Therapeutic Techniques and Equipments; Psychiatry and Psychology; Phenomena and Processes; Health Care. The presented proposal of a thesaurus contains the specific terminology of the Brazilian Speech-Language and Hearing Sciences and reflects the descriptors of the published scientific production. Being the DeCS a trilingual vocabulary (Portuguese, English and Spanish), the present descriptors organization proposition can be used in these three languages, allowing greater cultural interchange between different nations.

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.

This paper presents UniBERT, a compact multilingual language model that uses an innovative training framework that integrates three components: masked language modeling, adversarial training, and knowledge distillation. Pre-trained on a meticulously curated Wikipedia corpus spanning 107 languages, UniBERT is designed to reduce the computational demands of large-scale models while maintaining competitive performance across various natural language processing tasks. Comprehensive evaluations on four tasks, named entity recognition, natural language inference, question answering, and semantic textual similarity, demonstrate that our multilingual training strategy, enhanced by an adversarial objective, significantly improves cross-lingual generalization. Specifically, UniBERT models show an average relative improvement of 7.72% over traditional baselines, which achieved an average relative improvement of only 1.12%, and statistical analysis confirms the significance of these gains (p value = 0.0184). This work highlights the benefits of combining adversarial training and knowledge distillation to build robust and scalable language models, thus advancing the field of multilingual and cross-lingual natural language processing.

Topic identification techniques applied to dynamic language model adaptation for automatic speech recognition

To equip students for the advanced digital reality that awaits them, infusing computational thinking into their learning habits is vital. This is done through ensuring that technologies are integrated within the learners’ regular practices. This problem begs the question of how to best prepare teachers to achieve optimal or effective teaching performance. This study tries to answer this question by demonstrating how teacher training can be improved and made more effective through the application of distinct learning styles and Myers–Briggs Type Indicator (MBTI) personality types. In this manner, every teacher can receive proper training that addresses their distinct needs and classroom challenges utilizing their preferred personal learning elements. Understanding and accommodating teachers’ unique learning styles (e.g., visual, auditory, kinesthetics, or reading/writing) and MBTI personality types (e.g., ENFP, ISTJ, etc.) can enable teacher preparation programs to tailor their instructional methods for unplugged activities. Using this technique, students will be more interested and motivated in learning, thus they will better understand and retain computational thinking concepts. Teachers’ learning styles are then classified using an LSTM model while teachers’ MBTI personality types are classified based on their input questions using LinearSVC. This model is developed using Python and the Keras library; after training it on a huge dataset of labeled text data that has been pre-processed by removing punctuation marks and converting text to lowercase. Separate pickle files were saved for the tokenizer and label encoder in order to facilitate easy loading. The performance of these classification models is evaluated using standard metrics such as accuracy, precision, recall, and F1-score to ensure robust and effective categorization. The GPT-3.5 language model will be attached to the present model with a view to offering teachers who are under training explanations that best suit their identified learning styles as well as MBTI personality type. For this, GPT-3.5 architecture will be fine-tuned to generate responses that conform exactly to the teacher’s preferences as it gives them profound and reasoned explanations, meeting each teacher’s specific needs regarding individualized instruction and guidance. The GPT-3.5 implementation plan will illustrate how personalized teacher advice can be generated by utilizing such a model. In addition, given a trained machine learning model, it takes the input from a teacher and classifies his/her MBTI personality type along with their learning style at once. Then the GPT-3.5 model generates an in-depth explanation very specific to the teacher’s preferences with its prompt. In conclusion, teacher preparation in regard to embedding computational thinking in lessons with the use of unplugged activities should address the learning styles and MBTI personality types. This will provide a great chance to improve the learning experience of these students. Machine learning and natural language processing technologies embedded into teacher preparation programs promote the personalization of support aimed at engaging individual learners and aspiring educators in entrepreneurship. In this chapter, we illustrate how these thoughts can be implemented and, in particular, describe effective ways of improving the quality of teacher training through the use of computational thinking and unplugged activities.

While large language models (LLM) have made impressive progress in natural language processing, it remains unclear how to utilize them in improving automatic speech recognition (ASR). In this work, we propose to train a single multilingual language model (LM) for shallow fusion in multiple languages. We push the limits of the multilingual LM to cover up to 84 languages by scaling up using a mixture-of-experts LLM, i.e., generalist language model (GLaM). When the number of experts increases, GLaM dynamically selects only two at each decoding step to keep the inference computation roughly constant. We then apply GLaM to a multilingual shallow fusion task based on a state-of-the-art end-to-end model. Compared to a dense LM of similar computation during inference, GLaM reduces the WER of an English long-tail test set by 4.4% relative. In a multilingual shallow fusion task, GLaM improves 41 out of 50 languages with an average relative WER reduction of 3.85%, and a maximum reduction of 10%. Compared to the baseline model, GLaM achieves an average WER reduction of 5.53% over 43 languages.

The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT^a, a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%.

The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations that transfer across tasks and scale, increasing the impact of modeling research. However, with the emergence of state-of-the-art 100B+ parameters models, large language models are increasingly expensive to accurately design and train. Notably, it can be difficult to evaluate how modeling decisions may impact emergent capabilities, given that these capabilities arise mainly from sheer scale.Targeting a multilingual language model in the 100B+ parameters scale, our goal is to identify an architecture and training setup that makes the best use of our 1,000,000 A100-GPU-hours budget. Specifically, we perform an ablation study comparing different modeling practices and their impact on zero-shot generalization. We perform all our experiments on 1.3B models, providing a compromise between compute costs and the likelihood that our conclusions will hold for the target 100B+ model. In addition, we study the impact of various popular pretraining corpora on zero-shot generalization. We also study the performance of a multilingual model and how it compares to the English-only one. Finally, we consider the scaling behaviour of Transformers to chose the target model size, shape, and training setup.

Objective: To compare the performance of multiple machine learning algorithms in predicting recurrence after resection of early-stage hepatocellular carcinoma(HCC). Methods: Clinical data of 882 early-stage HCC patients who were admitted to the First Affiliated Hospital of Nanjing Medical University from May 2009 to December 2019 and treated with curative surgical resection were retrospectively collected. There were 701 males and 181 females,with an age of (57.3±10.5)years(range:21 to 86 years). All patients were randomly assigned in a 2∶1 ratio, the training dataset consisted of 588 patients and the test dataset consisted of 294 patients. The construction of machine learning-based prediction models included random survival forest(RSF),gradient boosting machine,elastic net regression and Cox regression model. The prediction accuracy of the model was measured by the concordance index(C-index). The prediction error of the model was measured by the integrated Brier score. Model fit was assessed by the calibration plot. The performance of machine learning models with that of rival model and HCC staging systems was compared. All models were validated in the independent test dataset. Results: Median recurrence-free survival was 61.7 months in the training dataset while median recurrence-free survival was 61.9 months in the validation dataset, there was no significant difference between two datasets in terms of recurrence-free survival(χ²=0.029,P=0.865). The RSF model consisted of 5 commonly used clinicopathological characteristics, including albumin-bilirubin grade,serum alpha fetoprotein,tumor number,type of hepatectomy and microvascular invasion. In both training and test datasets,the RSF model provided the best prediction accuracy,with respective C-index of 0.758(95%CI:0.725 to 0.791) and 0.749(95%CI:0.700 to 0.797),and the lowest prediction error,with respective integrated Brier score of 0.171 and 0.151. The prediction accuracy of RSF model for recurrence after resection of early-stage HCC was superior to that of other machine learning models,rival model(ERASL model) as well as HCC staging systems(BCLC,CNLC and TNM staging),with statistically significant difference(P<0.01). Calibration curves demonstrated good agreement between RSF model-predicted probabilities and observed outcomes.All patients could be stratified into low-risk,intermediate-risk or high-risk group based on RSF model;statistically significant differences among three risk groups were observed in both training and test datasets(all P<0.01). The risk stratification of RSF model was superior to that of TNM staging. Conclusion: The proposed RSF model assembled with 5 commonly used clinicopathological characteristics in this study can predict the recurrence risk with favorable accuracy that may facilitate clinical decision-support for patients with early-stage HCC.

With the recent explosion in popularity of voice assistant devices, there is a growing interest in making them available to user populations in additional countries and languages. However, to provide the highest accuracy and best performance for specific user populations, most existing voice assistant models are developed individually for each region or language, which requires linear investment of effort. In this paper, we propose a general multilingual model framework for Natural Language Understanding (NLU) models, which can help bootstrap new language models faster and reduce the amount of effort required to develop each language separately. We explore how different deep learning architectures affect multilingual NLU model performance. Our experimental results show that these multilingual models can reach same or better performance compared to monolingual models across language-specific test data while require less effort in creating features and model maintenance.

This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-ofthe-art performance on SentEval (Conneau and Kiela, 2018), even outperforming models learned using supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin, e.g. 10% improvement upon baseline models on cross-lingual semantic search. We explore the same language bias of the learned representations, and propose a simple, post-training and model agnostic approach to remove the language identifying information from the representation while still retaining sentence semantics.

In this paper, we present our approach for sentiment classification on Spanish-English code-mixed social media data in the SemEval-2020 Task 9. We investigate performance of various pre-trained Transformer models by using different fine-tuning strategies. We explore both monolingual and multilingual models with the standard fine-tuning method. Additionally, we propose a custom model that we fine-tune in two steps: once with a language modeling objective, and once with a task-specific objective. Although two-step fine-tuning improves sentiment classification performance over the base model, the large multilingual XLM-RoBERTa model achieves best weighted F1-score with 0.537 on development data and 0.739 on test data. With this score, our team jupitter placed tenth overall in the competition.