Thai Sentence Completeness Classification using Fine-Tuned WangchanBERTa

The increasing demand for cross-lingual communication in a globally interconnected world has spurred significant advancements in Natural Language Processing (NLP), a branch of artificial intelligence aimed at enabling computers to comprehend, generate, and process human language. Over the past few years, NLP has transitioned from traditional rule-based and statistical approaches to deep learning-based methods, with pre-trained models like BERT and GPT demonstrating remarkable performance across a variety of tasks, including text classification and machine translation. However, the linguistic diversity of the world's languages presents ongoing challenges, as many NLP models are primarily developed for high-resource languages like English, leaving other languages underserved. This paper examines the linguistic features of three representative languagesChinese, Japanese, and Englishand explores NLP methodologies tailored to their distinct grammatical, lexical, and structural properties. Through comparative linguistic analysis, this study discusses the challenges of multilingual NLP and the potential for cross-lingual applications. The findings underscore the importance of adapting NLP models to different language characteristics to enhance the effectiveness and inclusivity of cross-lingual processing.

Transformers have made a significant breakthrough in natural language processing. These models are trained on large datasets and can handle multiple tasks. We compare monolingual and multilingual transformer models for semantic relatedness and verse retrieval. We leveraged data from the original QurSim dataset (Arabic) and used authentic multi-author translations in 22 languages to create a multilingual QurSim dataset, which we released for the research community. We evaluated the performance of monolingual and multilingual LLMs for Arabic and our results show that monolingual LLMs give better results for verse classification and matching verse retrieval. We incrementally built monolingual models with Arabic, English, and Urdu and multilingual models with all 22 languages supported by the multilingual paraphrase-MiniLM-L12-v2 model. Our results show improvement in classification accuracy with the incorporation of multilingual QurSim.

The rapid change and use of computer technology has had a profound influence in the field of language teaching and assessment, and the changes that computer and technology has brought to language assessment in particular are not only in the delivery method, but also in the scoring systems (Chapelle & Douglas, 2006). For instance, a wide variety of tasks be given as a test online worldwide, and scores can be available immediately to the test takers. However, along with such advances come potential problems and concerns.

The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT^a, a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%.

Advances in natural language processing (NLP) and automatic speech recognition and processing technologies offer new opportunities for language testing. Despite their potential uses on a range of language test item types, relatively little work has been done in this area, and it is therefore not well understood by test developers, researchers or users in language assessment. This paper introduces NLP for language assessment as an area of inquiry and practice by describing the historical roots coming from computational linguistics, statistical NLP, speech recognition and processing technologies, language assessment, and computer-assisted language learning. It outlines uses of NLP and speech recognition and processing technologies in language assessment through illustrations of current testing projects, and identifies areas in need of further development.

Pulmonary nodules and nodule characteristics are important indicators of lung nodule malignancy. However, nodule information is often documented as free text in clinical narratives such as radiology reports in electronic health record systems. Natural language processing (NLP) is the key technology to extract and standardize patient information from radiology reports into structured data elements. This study aimed to develop an NLP system using state-of-the-art transformer models to extract pulmonary nodules and associated nodule characteristics from radiology reports. We identified a cohort of 3080 patients who underwent LDCT at the University of Florida health system and collected their radiology reports. We manually annotated 394 reports as the gold standard. We explored eight pretrained transformer models from three transformer architectures including bidirectional encoder representations from transformers (BERT), robustly optimized BERT approach (RoBERTa), and A Lite BERT (ALBERT), for clinical concept extraction, relation identification, and negation detection. We examined general transformer models pretrained using general English corpora, transformer models fine-tuned using a clinical corpus, and a large clinical transformer model, GatorTron, which was trained from scratch using 90 billion words of clinical text. We compared transformer models with two baseline models including a recurrent neural network implemented using bidirectional long short-term memory with a conditional random fields layer and support vector machines. RoBERTa-mimic achieved the best F1-score of 0.9279 for nodule concept and nodule characteristics extraction. ALBERT-base and GatorTron achieved the best F1-score of 0.9737 in linking nodule characteristics to pulmonary nodules. Seven out of eight transformers achieved the best F1-score of 1.0000 for negation detection. Our end-to-end system achieved an overall F1-score of 0.8869. This study demonstrated the advantage of state-of-the-art transformer models for pulmonary nodule information extraction from radiology reports.

Entity Linking (EL) is a technology that links mentions (context-dependent token sequences that may refer to specific entities) in a text to corresponding entities in a knowledge base. It serves as a foundational technology in knowledge processing and natural language processing. Most research on EL focuses on English, and there is little research that targets Japanese. While research on multilingual EL models, which can be used for languages other than English, including Japanese, is advancing, EL that considers the characteristics of individual languages has not been fully achieved. Additionally, recent EL research utilizes word embeddings and knowledge graph embeddings in the construction of EL models, and building an EL model for a specific language requires language-specific adaptations. Pointer Network has the feature of selecting elements of the output sequence from the input sequence. By using Pointer Network, it may be possible to perform EL that is not included in the training data. In this study, we proposed a Japanese EL model based on the Pointer Network (Japanese PNEL model) forWikidata. We evaluated the accuracy of the Japanese PNEL model and analyzed effective features in Japanese EL and English EL.In addition, we have constructed a Japanese EL evaluation dataset by machine translating the English EL evaluation datasets WebQSP, SimpleQuestions, and LC-QuAD2 into Japanese.From the results of comparison experiments with existing multilingual models, we confirmed that the Japanese PNEL model outperformed existing multilingual models in terms of F1 score in evaluation experiments targeting WebQSPs that were machine-translated into Japanese.We analyzed more effective features through experiments on the impact of the difference in the number of dimensions and models for word embedding and knowledge graph embedding on EL accuracy. The results of the ablation study showed that knowledge graph embedding is also as effective in Japanese as in English.

The potential of automated question generation is role play in the multi-fields and multi-applications such as question and answering systems, examination systems, and information retrieval. Before learning the question generated, one should understand how to classify questions. This research aims to generate possible questions considering the possible question categories from question classification based on Natural Language Processing. In this research, we compared the results on Logistic Regression, Support Vector Machine, and Multinomial Naï ve Bayes, which were traditional classification models. The deep learning techniques were Convolutional Neural Networks, Bidirectional Long Short-Term Memory, combined CNN and BiLSTM models, and BERT models. The experimental results show that the preprocessing phase using Natural Language Processing could enhance question classification. The classification of the sentence to question classification attained an average micro $F_{1} -$ score of 91.40% when applied BERT model by pre-trained WangchanBERTa on simple sentences. In contrast, the satisfying score with an average micro $F_{1} -$ score of 82.07% (from 80.37% on original input) when applied to add all POS tags unigram + bigram TF-IDF by using the SVM model. The experimental results when the CNN model with GloVe on adding focusing POS tags is a satisfactory score with an average micro $F_{1} -$ score of 79.79%.

Multilingual pre-trained models make it possible to develop natural language processing (NLP) applications for low-resource languages (LRLs) using the model of resource-rich languages (RRLs). However, the structural characteristics of the target languages can impact task-specific learning. In this paper, we investigate the influence of structural diversities of languages on the system’s overall performance. Specifically, we propose a customized approach to leverage task-specific data of low-resource language families via transfer learning from RRL. Our findings are based on question-answering tasks using the XLM-R, mBERT, and IndicBERT transformer models and Indic languages (Hindi, Bengali, and Telugu). On the XQuAD-Hindi dataset, the few-shot learning using Bengali improves the benchmark mBERT (F1/EM) score by +(10.86/7.87) and XLM-R score by +(3.84/4.42). Few-shot learning using Telugu has also improved the mBERT score by +(10.42/7.36) and +(3.04/2.72) for XLM-R. In addition, our model has demonstrated benchmark-compatible performance in a zero-shot setup with single-epoch task learning. This approach can be adapted for other NLP tasks for LRLs.

Natural Language Processing (NLP) is an essential element of computational linguistics and artificial ‎intelligence, enabling fluid interactions between humans and machines. Social networking networks ‎produce substantial volumes of user-generated text daily, offering both opportunities and challenges for ‎NLP researchers. Social media discourse's informal, dynamic, and context-dependent characteristics ‎necessitate specific NLP techniques for precise processing and analysis. This study thoroughly examines NLP applications in social media, including essential tasks such as sentiment analysis, topic ‎modeling, misinformation detection, and hate speech identification. It examines the influence of machine ‎learning and deep learning methodologies, particularly transformer models, on the advancement of NLP ‎capabilities. This study also emphasizes the ethical issues related to NLP-driven social media apps, ‎including data privacy, algorithmic bias, and the regulation of misinformation. The paper continues by ‎discussing emerging research paths, highlighting the necessity for adaptable and ethical NLP solutions in ‎the changing social media environment‎.

This comprehensive review explores the utilization of Artificial Intelligence (AI) in language assessment and its transformative potential. Language assessment plays a vital role in education, employment, and societal integration. Language assessment has traditionally relied on human evaluators, who assess and score language performance based on standardized criteria. However, this manual assessment approach has limitations, including subjectivity, inter-rater variability, and scalability issues. With the rapid advancements in AI technologies, language assessment has experienced significant changes, offering more accurate, efficient, and innovative evaluation methods. The method used in this study is systematic review which critically examines the application of AI in language assessment, highlighting its benefits, challenges, and future prospects. The review covers various aspects, including automated scoring and evaluation, intelligent tutoring systems, natural language processing, advantages and benefits, challenges and considerations, and future directions. By harnessing the use of AI, language assessment can achieve unprecedented levels of objectivity, scalability, and personalization, while addressing ethical considerations. This review contributes to the understanding of AI's impact on language

Geriatric syndromes (GS) are complex conditions that affect older adults and often require multidisciplinary assessment. Natural language processing (NLP) has emerged as a promising tool for extracting relevant clinical information from unstructured text in electronic health records (EHRs). However, the application of NLP in detecting and monitoring GS remains an evolving area of research. This systematic review explores the role of NLP in the identification and analysis of GS, examining its applications, methodologies, and effectiveness. Furthermore, this review discusses the existing challenges, limitations, and future directions to advance NLP applications in the GS research. We conducted a systematic literature search across ten databases to identify studies that applied NLP to GS detection. Articles were screened using predefined inclusion and exclusion criteria, and relevant studies were evaluated for quality using PROBAST. Data were extracted on study characteristics, datasets, annotation processes, NLP approaches, performance metrics, population demographics, and clinical applications. A PRISMA flow diagram was used to illustrate the study selection process. A total of 65 studies were included, where the majority of the studies used traditional rule-based and machine learning approaches. Publicly available datasets were scarce, and most studies used their private dataset, leading to significant variability in data sources and formats. Annotation methodologies differed across studies, with minimal shared guidelines or standards, making direct comparisons challenging. Performance metrics varied across syndromes, with F1-score, precision, and recall as the most commonly reported. Key challenges included the lack of dataset uniformity, differences in annotation practices, and the absence of external validation. NLP has shown potential in GS analysis, particularly for the detection of syndromes and epidemiological research. However, the majority of studies only focused on one syndrome, and variability in dataset availability, annotation processes, and model performance present challenges to broader implementation. Future research should focus on improving the comprehensiveness of GS identification, dataset standardisation, enhancing model generalisability, and integrating NLP approaches into clinical workflows.

The recent advances of Natural Language Processing (NLP) have significantly benefited people across the globe who speak and write some specific languages, commonly termed high-resource languages (HRLs). These are languages with abundant digital resources. One of the most significant areas where NLP has recently made its mark is environment preservation. However, a significant digital divide is observed for low-resource languages (LRLs), which are not rich in terms of digital resources. Even though many LRLs, like Assamese, are very important for the cultural and linguistic identity of many indigenous communities, they remain digitally underrepresented due to a lack of annotated corpora and computational tools. This gap is even more crucial in the environmental domain, where knowledge is very limited even for high-resource languages. Multilingual information is very important for NLP applications related to climate change, disaster management and other issues related to environmental science. This paper reports a study of the challenges and strategies to address the gap in NLP, both in the context of the global and Indian landscape. The paper also highlights the key problems while addressing issues related to environmental science due to the lack of digital resources. Some of the significant factors include the absence of large corpora, labelled datasets and socio-economic factors. The paper proposes to emphasize data collection and digitization, among many other measures, to address this gap. The integration of multilingual pretrained models and transfer learning approaches also provides a pathway for enhancing performance with limited resources. The paper also presents a detailed analysis of resources available for global and Indian languages in addition to proposing a set of strategic actions, including government policies, and others, for inclusive development in NLP and sustainable environment. Bridging the divide and enabling linguistic equity will ensure participation of all sections of the society in the advancement of inclusive technological growth and solutions to the emerging problems.

Part-of-speech (POS) tagging is a critical task in natural language processing (NLP), influencing the performance of downstream applications such as semantic parsing and machine translation. This study presents a novel approach to POS tagging by incorporating relative positional encoding within the transformer model. Unlike traditional absolute positional encoding, the proposed method leverages token dependencies more effectively, enhancing the transformer’s self-attention mechanism. The architecture integrates a rule-based module to correct misclassifications, further refining the results. Experiments on the Groningen Meaning Bank (GMB) dataset demonstrate the model's superiority, achieving an accuracy of 99.68%, a significant improvement over the 98.60% accuracy of models with absolute positional encoding. Additional metrics, including precision (0.92), recall (0.89), and F1-score (0.90), further confirm the model's effectiveness. The findings highlight the potential of relative positional encoding in improving contextual understanding and model performance, providing a robust solution for POS tagging tasks in NLP.

Automatic questions and answers generated are challenging tasks that attract broad engagement. This paper presents an automatic question and answer generation from Thai sentences. We adopted question classification to predict the expected wh-question type before generating question and answer. This experimental study tested the question and answer generation on the sampling set of 300 Thai sentences based on the wh-question type that trained classify category from 15,000 samples. Automatic evaluation is performed on a reference set using the BLEU metrics and human evaluation. The experiment results show that the proposed method can generate questions and answers from Thai sentences. However, the performance based on the efficiency of Natural Language Processing includes tokenization, Named Entities Recognition, and Part of Speech tagging task because it could effectively extract focusing words correctly and suitable to answer the question that enhances the trustworthiness of future work in question and answer generation from Thai texts.KeywordsQuestion and answer generationAutomatic question generationNatural language processingThai sentences analysis