A Comprehensive Methodology for Extracting Signal from Social Media Text Using Natural Language Processing and Machine Learning

The Impact of Arabic Part of Speech Tagging on Sentiment Analysis: A New Corpus and Deep Learning Approach

Objective Social media is becoming increasingly popular as a platform for sharing personal health-related information. This information can be utilized for public health monitoring tasks, particularly for pharmacovigilance, via the use of natural language processing (NLP) techniques. However, the language in social media is highly informal, and user-expressed medical concepts are often nontechnical, descriptive, and challenging to extract. There has been limited progress in addressing these challenges, and thus far, advanced machine learning-based NLP techniques have been underutilized. Our objective is to design a machine learning-based approach to extract mentions of adverse drug reactions (ADRs) from highly informal text in social media. Methods We introduce ADRMine, a machine learning-based concept extraction system that uses conditional random fields (CRFs). ADRMine utilizes a variety of features, including a novel feature for modeling words’ semantic similarities. The similarities are modeled by clustering words based on unsupervised, pretrained word representation vectors (embeddings) generated from unlabeled user posts in social media using a deep learning technique. Results ADRMine outperforms several strong baseline systems in the ADR extraction task by achieving an F-measure of 0.82. Feature analysis demonstrates that the proposed word cluster features significantly improve extraction performance. Conclusion It is possible to extract complex medical concepts, with relatively high performance, from informal, user-generated content. Our approach is particularly scalable, suitable for social media mining, as it relies on large volumes of unlabeled data, thus diminishing the need for large, annotated training data sets.

Sentiment analysis is the process of computationally recognizing and classifying the attitudes conveyed in each text towards a particular topic, product, etc. which is either positive or negative. Sentiment analysis is one of the interesting applications of natural language processing (NLP) and which is used to analyze the social media. Text in social media is casual and it can be written either in code-switch or monolingual text. Several researchers have implemented sentiment analysis on monolingual text, though sentiments can be expressed in code-switch text. Sentiment analysis can be applied through deep learning (DL), machine learning (ML), or a Lexicon-based approach. Machine learning (ML) and deep learning (DL) methods are time-consuming, computationally expensive, and need training data for analysis. Lexicon-based method does not require training data and requires less time to find the sentiments in comparison with ML and DL. In this paper, we propose the Lexicon-based approach (NBLex) to analyze the sentiments expressed in Kannada-English code-switch text. This is the first effort that targets to perform sentiment analysis in Kannada-English code-switch text using the Lexicon-based approach. The proposed approach performed with better accuracy of 83.2% and 83% of F1-score.

Aspirin-exacerbated respiratory disease (AERD) is an acquired inflammatory condition characterized by the presence of asthma, chronic rhinosinusitis with nasal polyposis, and respiratory hypersensitivity reactions on ingestion of aspirin or other nonsteroidal anti-inflammatory drugs (NSAIDs). Despite AERD having a classic constellation of symptoms, the diagnosis is often overlooked, with an average of greater than 10 years between the onset of symptoms and diagnosis of AERD. Without a diagnosis, individuals will lack opportunities to receive effective treatments, such as aspirin desensitization or biologic medications. Our aim was to develop a combined algorithm that integrates both natural language processing (NLP) and machine learning (ML) techniques to identify patients with AERD from an electronic health record (EHR). A rule-based decision tree algorithm incorporating NLP-based features was developed using clinical documents from the EHR at Mayo Clinic. From clinical notes, using NLP techniques, 7 features were extracted that included the following: AERD, asthma, NSAID allergy, nasal polyps, chronic sinusitis, elevated urine leukotriene E4 level, and documented no-NSAID allergy. MedTagger was used to extract these 7 features from the unstructured clinical text given a set of keywords and patterns based on the chart review of 2 allergy and immunology experts for AERD. The status of each extracted feature was quantified by assigning the frequency of its occurrence in clinical documents per subject. We optimized the decision tree classifier's hyperparameters cutoff threshold on the training set to determine the representative feature combination to discriminate AERD. We then evaluated the resulting model on the test set. The AERD algorithm, which combines NLP and ML techniques, achieved an area under the receiver operating characteristic curve score, sensitivity, and specificity of 0.86 (95% CI 0.78-0.94), 80.00 (95% CI 70.82-87.33), and 88.00 (95% CI 79.98-93.64) for the test set, respectively. We developed a promising AERD algorithm that needs further refinement to improve AERD diagnosis. Continued development of NLP and ML technologies has the potential to reduce diagnostic delays for AERD and improve the health of our patients.

Stress is an almost basic human instinct, especially in the internet age. People’s health is being endangered by psychological stress. It is critical to diagnose stress early on to provide proactive therapy. In the age of the Internet, social media has an enormous impact on human thinking. It can cause mental health issues such as stress but can also play an important part in detecting it. Advancements in machine learning and natural language processing have enabled information extraction from a huge amount of raw textual data. In this study machine learning approach is used to detect stress in a social media text. Data used in this study is from Reddit. Classical and Ensemble machine learning approaches were used for stress detection. Classical Techniques include Decision Tree, Logistic Regression, support vector machine, Random Forest, and Na ̈ıve Bayes. Ensemble approaches used are boosting, bagging, and voting. The ensemble approach was able to provide better results than all machine learning baselines on the dataset. Also, it was able to outperform all non-transformed-based neural network architectures discussed in the baseline. The best-performing model in this study is Logistic regression with a 76.6% F1 Score.

In this manuscript, the rise of misinformation and harmful behavior on social media platforms has made it crucial to develop systems that ensure the authenticity of shared content and user accounts is presented. Parsing of social media feeds for forensic investigation aims to identify fake news and detect fake social media accounts (focusing on platforms like Instagram). The main goal of parsing of social media feeds for forensic investigation is to protect users from misleading information spread by fake accounts. To achieve this, various modules are integrated into the system, including Data Collection and Pre-Processing, Fake News Detection and Account Authenticity Verification. Web scraping tools such as BeautifulSoup are used to gather real-time data, while machine learning algorithms, including Random Forest and Large Language Model (LLM) are employed to classify fake news and detect suspicious account behaviors. The system also uses NLP models to identify fake news patterns and generate comprehensive reports summarizing the findings of each module. By combining Natural Language Processing (NLP) and machine learning techniques, parsing of social media feeds for forensic investigation provides an essential platform for users to protect themselves from misleading information on social networks.

Malay social media text is a text written on social media networks like Twitter. Commonly, this text comprises non-standard words, filled with dialects, foreign languages, word abbreviations, grammatical neglect, spelling errors, and many more. It is well known that this type of text is difficult to process due to its high noise and distinct text structure. Such problems can be resolved using rigorous text normalization, which is critical before any technique can be implemented and evaluated on social media text. In this paper, an improved normalization method towards Malay social media text was proposed by converting non-standard Malay words using a rule-based model. The method normalizes common language words often used by Malaysian users, such as non-standard Malay (like dialect and slangs), Romanized Arabic, and English words. Thus, a Malay text normalizer was proposed using a set of rules that extend across different domains of natural language processing (NLP) and is expected to address the challenges of processing Malay social media text. This study implements the proposed Malay text normalizer in a Part-of-Speech (POS) tagging application to evaluate the normalizer’s performance. The implementation demonstrates a substantial improvement in the POS tagging efficiency over several pre-processing stages, with an improvement of accuracy up to 31.8%. The increase of accuracy in the POS tagging indicates two main points. First, the Malay text normalizer’s rules improve the performance of a Malay text normalizer on social media text. Second, our proposed Malay text normalizer has successfully improved the POS tagging percentage and demonstrates the importance of normalized pre-processing in any NLP application.

Adverse drug events (ADEs) remain a significant burden to public health and a persistent challenge for pharmacovigilance. The proliferation of patient-generated discourse on social media offers a complementary, real-time signal for ADE surveillance. This article provides a concise yet comprehensive review of recent natural language processing (NLP) research on identifying ADEs in social media text. We systematically reviewed 100 peer-reviewed studies (2017-2025) on NLP/AI for detecting or analysing ADEs in social media. Searches in Google Scholar targeted English-language journal and conference papers; patents and protocols were excluded. Of 130 records screened, 6 were protocols and 24 were excluded because the full text could not be located or the item was a conference abstract lacking methodological detail (i.e., no description of approaches or experiments), yielding a final sample of 100 studies. One reviewer performed screening, with full-text eligibility verified by a second. We extracted objectives, data sources/languages, preprocessing and annotation practices, datasets, model families, evaluation metrics, and stated limitations. Studies were grouped into five task categories-classification, extraction, normalization, corpus creation, and broader analytical work-with evidence tables summarizing contributions, toolchains, datasets, and performance. Recurrent challenges include noisy/imbalanced data, multilingual and code-mixed content, and variability in annotation standards. Twitter remains the primary data source: 60% of studies analyse Twitter alone and a further 18% combine Twitter with other platforms (78% in total). English overwhelmingly dominates; only about 5% of studies draw on non-English sources (e.g., French, Chinese, Arabic). Standard pre-processing-URL removal, tokenisation, and lowercasing-is near-universal. Transformer-based models predominate, with BERT and its biomedical or "tweet" variants (e.g., RoBERTa, BioBERT, BERTweet) used in more than 60% of approaches. Persistent obstacles include severe class imbalance and ambiguous or implicit drug-event expressions. Although shared tasks such as SMM4H provide widely used benchmarks, comprehensive annotation guidelines remain uncommon (12% of papers). Recent work increasingly incorporates multimodal inputs and integrates structured biomedical knowledge, yet gaps persist in multilingual coverage, temporal/longitudinal modelling, and real-world deployment. To our knowledge, this is the first review to synthesise findings from a corpus of 100 peer-reviewed studies on ADE detection in social media using NLP. By organising the literature by task type and tracing methodological trends and limitations, it provides practical guidance for researchers and practitioners. The review also outlines actionable directions for future work, including model explainability, support for low-resource languages, and closer collaboration with regulatory authorities to enable real-world deployment.

BackgroundMachine learning systems are part of the field of artificial intelligence that automatically learn models from data to make better decisions. Natural language processing (NLP), by using corpora and learning approaches, provides good performance in statistical tasks, such as text classification or sentiment mining.ObjectiveThe primary aim of this systematic review was to summarize and characterize, in methodological and technical terms, studies that used machine learning and NLP techniques for mental health. The secondary aim was to consider the potential use of these methods in mental health clinical practiceMethodsThis systematic review follows the PRISMA (Preferred Reporting Items for Systematic Review and Meta-analysis) guidelines and is registered with PROSPERO (Prospective Register of Systematic Reviews; number CRD42019107376). The search was conducted using 4 medical databases (PubMed, Scopus, ScienceDirect, and PsycINFO) with the following keywords: machine learning, data mining, psychiatry, mental health, and mental disorder. The exclusion criteria were as follows: languages other than English, anonymization process, case studies, conference papers, and reviews. No limitations on publication dates were imposed.ResultsA total of 327 articles were identified, of which 269 (82.3%) were excluded and 58 (17.7%) were included in the review. The results were organized through a qualitative perspective. Although studies had heterogeneous topics and methods, some themes emerged. Population studies could be grouped into 3 categories: patients included in medical databases, patients who came to the emergency room, and social media users. The main objectives were to extract symptoms, classify severity of illness, compare therapy effectiveness, provide psychopathological clues, and challenge the current nosography. Medical records and social media were the 2 major data sources. With regard to the methods used, preprocessing used the standard methods of NLP and unique identifier extraction dedicated to medical texts. Efficient classifiers were preferred rather than transparent functioning classifiers. Python was the most frequently used platform.ConclusionsMachine learning and NLP models have been highly topical issues in medicine in recent years and may be considered a new paradigm in medical research. However, these processes tend to confirm clinical hypotheses rather than developing entirely new information, and only one major category of the population (ie, social media users) is an imprecise cohort. Moreover, some language-specific features can improve the performance of NLP methods, and their extension to other languages should be more closely investigated. However, machine learning and NLP techniques provide useful information from unexplored data (ie, patients’ daily habits that are usually inaccessible to care providers). Before considering It as an additional tool of mental health care, ethical issues remain and should be discussed in a timely manner. Machine learning and NLP methods may offer multiple perspectives in mental health research but should also be considered as tools to support clinical practice.

Language steganography in social networks primarily focuses on embedding secret information into social media text efficiently to achieve covert communication. The misuse of such techniques could pose significant potential threats to public cyberspace, such as the spread of malicious code, commands, or viruses. Existing social text steganalysis techniques mainly focus on the analysis of individual social media texts. However, the information content in a single text is very limited, leading to poor detection performance in practical applications. To address this challenge, this paper proposes a social text steganalysis method that combines large-scale language models with common-sense knowledge graphs (STLC-KG). This method first uses knowledge graphs to expand the knowledge contained in the text under investigation, enriching its linguistic expression, and then utilizes large-scale language models to extract the linguistic features of the social text. The results of tests conducted on three mainstream social media platforms demonstrate that the proposed method significantly improves the performance of social text steganalysis.

Social media, a buzz term in the modern world, refers to various online platforms like social networks, forums, blogs and blog comments, microblogs, wikis, media sharing platforms, social bookmarks through which communication between individuals, communities, or groups takes place. People over social media do not only share their ideas and opinions, but it has become an important source through which businesses promote their products. Analyzing huge data generated over social media is useful in various tasks like analyzing customer trends, forecast sales, understanding opinions of people on different hot topics, views of customers about services/products, and many more. Different natural language processing (NLP) techniques are used for crawling and processing social media data to get useful insights out of this. In this chapter, the focus is on various NLP techniques used to process the social media data. Challenges faced by NLP techniques to process social media data are also put forward in this chapter.

Objectives: To provide an organised literature on the detection of Abusive language on Twitter using natural language processing (NLP). Methods: In this study, the survey has been conducted on different methods and research conducted on the types of Abusive language used in social media, why it is important? How it has been detected in real time social media platforms and the performance metrics that are used by researchers in evaluating the performance of the detection of abusive language on Twitter by the users. Results: Giving an organised review of past methodologies, including methods, important features and core algorithms, this study arranges and depicts the present condition about this area. The study also talks about the intricacy of hate speech idea which is characterised in numerous stages ad settings. This area of study has an obvious potential for societal effect, especially in digital media and online networks. A crucial step in propelling automatic hate speech detection is the advancement and systemisation of common assets, for example, clarified data sets in numerous dialects, rules, and calculations. Conclusion: This survey study contains all the relevant references related to detection of abusive language on social media using NLP and machine learning methods. Ultimately, it can be as source of references to the other researchers in finding the literatures that are relevant to their research area in the detection of Abusive language on Twitter. Keywords: Abusive Language, Natural Language Processing, Social media analysis, Text Classification and Analysis

Social networking sites have become an integral part of everyday life, where people interact, cooperate and quarrel with each other. Social media also encourages them to express their opinions and share their comments about their lives’ events or about the product they use. Opinions can be direct without any comparison (I like ABC phone) or they can be comparative (X-phone’s camera is better than Y-phone). Comparative opinions are useful in many applications, e.g. marketing intelligence, product benchmarking, and e-commerce. The automatic mining of comparative opinions is an important text mining problem and an area of increasing interest for different languages. This paper focuses on identification of comparative sentence from non-comparative ones in Arabic texts. A corpus was developed consisting of YouTube comments. This paper describes research experiments that aimed to apply data/text mining algorithms, natural language processing and linguistic classification for Arabic comparative text discovery. The results of these experiments along with the analysis are also presented. The results were promising reaching to 91% accuracy for the detection of comparative opinions.

While radiologists regularly issue follow-up recommendations, our preliminary research has shown that anywhere from 35 to 50% of patients who receive follow-up recommendations for findings of possible cancer on abdominopelvic imaging do not return for follow-up. As such, they remain at risk for adverse outcomes related to missed or delayed cancer diagnosis. In this study, we develop an algorithm to automatically detect free text radiology reports that have a follow-up recommendation using natural language processing (NLP) techniques and machine learning models. The data set used in this study consists of 6000 free text reports from the author's institution. NLP techniques are used to engineer 1500 features, which include the most informative unigrams, bigrams, and trigrams in the training corpus after performing tokenization and Porter stemming. On this data set, we train naive Bayes, decision tree, and maximum entropy models. The decision tree model, with an F1 score of 0.458 and accuracy of 0.862, outperforms both the naive Bayes (F1 score of 0.381) and maximum entropy (F1 score of 0.387) models. The models were analyzed to determine predictive features, with term frequency of n-grams such as "renal neoplasm" and "evalu with enhanc" being most predictive of a follow-up recommendation. Key to maximizing performance was feature engineering that extracts predictive information and appropriate selection of machine learning algorithms based on the feature set.

Understanding emotions in social media text is crucial for mental health monitoring, sentiment analysis, and misinformation detection applications. This study presents an Improved Graph Convolutional Network (IGCN) that leverages the network structure of social media text to enhance emotion classification. Unlike conventional GCN models, our approach integrates a Pointwise Mutual Information (PMI) based graph construction method, which improves the representation of semantic relationships between words. Additionally, an attention mechanism selectively emphasizes contextually significant words, enhancing interpretability and classification accuracy.Key contributions:•Graph-based sentiment modelling to capture deep semantic relationships in text.•Improved interpretability through attention-weighted word importance.•Scalability for large social media datasets, ensuring efficient processing.Through extensive comparative experiments, our model achieves 78.64% and 92.38% accuracy, demonstrating the effectiveness of GNNs in large-scale emotion classification. This research underscores the transformative potential of graph-based NLP models for decoding emotional tones in social media, paving the way for more accurate and insightful sentiment analysis. The research is conducted on two large-scale datasets: Twitter_EA: Categorizing tweets into six emotions—Sadness, Joy, Love, Anger, Fear, and Surprise. Emotion Recognition Dataset: Labelling emotions as Anxiety, Depression, Happiness, and Stress.