Basic Text Analysis, Preprocessing, Bag-of-Words, TF-IDF, and Exploratory Statistics

Numerous e-news channels publish the daily happenings in the world from different sources. These huge amounts of news articles have lamentably conceived the information overload issue among the users. Hence text mining, which aims in extracting previously unknown information from unstructured text, has been widely used by several researchers to segregate full news articles however, the news headlines categorization is still specifically limited. Therefore, considering this limitation, the current research aims to propose a framework that will self-learn and automatically classify any given news headline into its corresponding news category using artificial intelligence methods i.e. text mining and machine learning algorithms. The proposed framework consists of three stages: Exploratory Data Analysis, Text Pre-processing, and Text Classification. For exploratory data analysis, the top 10 most frequent balanced news categories are chosen so that further processing of data can be done on a more balanced version of the dataset. After exploring the data, text pre-processing techniques are applied to make the data transformed, normalized, and structured. Finally, text classification is carried out with two approaches: unsupervised classification using Mean Shift and K-means algorithms and supervised classification using Logistic Regression with Bag of Words and TF-IDF algorithm. To depict the working of the proposed framework, a case study is presented on a news headlines dataset which accurately performed news headlines classification.

Today, data analysis plays a key role in studying and using large amounts of information to make informed decisions. Thus data is one of the most valuable resources and analytical tools are becoming increasingly important. Data analysis allows us to identify patterns and relationships between different factors. As for the methods of analysis, they are very diverse and depend on the specific task and the nature of the data, each method has its advantages and limitations, and the choice of a particular one depends on the context. Recent investigations have proved that LLM showed good results in textual information analysis and might be one of the textual analytical tools. Therefore, the main focus of our research is centred around the ability of LLM to perform data interpretation operations, arithmetic and statistical operations in the meaning space. In the evaluation of the proposed concepts, simple cases were considered. It was enough to better understand the effectiveness of LLM as one of the tools for exploratory data analysis. The practical results of the research indicate that the concept has some advantages over the closest analogues, as well as identify several scientific problems that can be solved in subsequent studies. Additionally, research tools were developed as a chatbot system in the Telegram environment. Keywords: exploratory data analysis, prompt engineering, large language models, chatbot systems, “meaning” space.

Information extraction, retrieval, and text categorization are only a few of the significant research fields covered by "bio medical text classification." This study examines many text categorization techniques utilised in practise, as well as their strengths and weaknesses, in order to improve knowledge of various information extraction opportunities in the field of data mining. We compiled a dataset with a focus on three categories: "Thyroid Cancer," "Lung Cancer," and "Colon Cancer." This paper presents an empirical study of a classifier. The investigation was carried out using biomedical literature benchmarks. Many metaheuristic algorithms are investigated, including genetic algorithms, particle swarm optimisation, firefly, cuckoo, and bat algorithms. In addition, the proposed multiple classifier system outperforms ensemble learning, ensemble pruning, and traditional classification methods. Based on the data, we forecast if it is Thyroid Cancer, Lung Cancer, or Colon Cancer using basic EDA, text preprocessing, and several models such as Logistic Regression, Decision Tree Classification, and Random Forest Classification.

The abundance of unstructured data, primarily consisting of text, coupled with the development of modern techniques to analyze them, offer new opportunities for theory generation, theory testing, and contextualization within the field of career studies. This chapter explains the process of text mining as it may be applied to the study of careers, starting with text preprocessing and ending with the application of analytical techniques. Both conventional techniques such as latent semantic and principal components analysis, and more novel text analytical techniques such as word embeddings are discussed with the hope of inspiring careers researchers to further embrace text analysis as a means to answer novel research questions. Furthermore, this chapter elaborates on how to evaluate the validity of information extracted from text. In order to illustrate the text analytical process, we provide an example that leverages an abundant and growing source of data that may be of interest to career researchers, namely, vacancy data. In the example, vacancy data are used to identify those knowledge, skills and abilities that (co)determine the salary associated with particular occupations. We end the chapter by providing a guide on how text mining can be applied to study existing and new career concepts.

Data Science and Computational Linguistics on a Collection of Interviews with Family Caregivers in Heart Failure

Topic Modelling has established itself as one of the major text-as-data methodologies within the social sciences in general, and in communications science, in particular. The core strength of TM approaches is the fact that it is essentially a 2-in-1 method, as it both generates clusters into which the texts may fall as well as classifies the texts according to the clusters. Previous research has pointed out that pre-processing text corpora is as much a part of text analysis as the latter stages. Named Entity Recognition, however, is not often thought of when pre-processing texts for analysis and has thus far not received much attention in relation to the TM pipeline. If simply retaining or removing stop words can produce different interpretations of the outcomes of TM, retaining or removing NEs also has consequences on outcomes and interpretations. The current paper analyses the effects that removing/retaining NEs has on the interpretability of topic models. Both model statistics and human validation are used to address this issue. The results show differences in topics models trained on corpora with and without NEs. TMs trained on corpora where NEs are removed exhibit different structural characteristics, and, more importantly, are perceived differently by human coders. We attempt to formulate recommendations regarding the pre-processing of NEs in TM applications.

Exploratory analysis of unstructured text is a difficult task, particularly when defining and extracting domain-specific concepts. We present iSeqL, an interactive tool for the rapid construction of customized text mining models through sequence labeling. With iSeqL, analysts engage in an active learning loop, labeling text instances and iteratively assessing trained models by viewing model predictions in the context of both individual text instances and task-specific visualizations of the full dataset. To build suitable models with limited training data, iSeqL leverages transfer learning and pre-trained contextual word embeddings within a recurrent neural architecture. Through case studies and an online experiment, we demonstrate the use of iSeqL to quickly bootstrap models sufficiently accurate to perform in-depth exploratory analysis. With less than an hour of annotation effort, iSeqL users are able to generate stable outputs over custom extracted entities, including context-sensitive discovery of phrases that were never manually labeled.

Purpose This study aims to explore the application of the Elasticsearch algorithm to assess the degree of implementation of Operational Excellence 5.0 (OE 5.0) among Spanish insurance companies. It examines how corporate sustainability reporting, aligned with the EU Corporate Sustainability Reporting Directive (CSRD), reflects strategic emphasis on OE 5.0 and its contribution to Sustainable Development Goals (SDGs). Design/methodology/approach Annual reports published by six insurance companies operating in Spain were analysed using Elasticsearch, which applies inverted indexing, BM25 scoring, and vector search techniques to identify textual emphasis on SDG 8 (Decent work and economic growth), SDG 9 (Industry, innovation and infrastructure) and SDG 12 (Responsible consumption and production). A dictionary-based approach was used to ensure transparency and replicability. Findings The results reveal significant heterogeneity in the emphasis on OE 5.0 principles. While Allianz Partners and MAPFRE demonstrate strong alignment with multiple SDGs, others, such as AXA Insurance, show a more selective focus, particularly regarding responsible consumption. These findings suggest uneven diffusion of OE 5.0, influenced by organisational priorities and resources. Research limitations/implications The exploratory analysis is limited to six Spanish insurers and two reporting years (2023–2024). Broader cross-sector and longitudinal studies are needed for validation. Originality/value This study contributes to the emerging literature on Lean Six Sigma and Industry 5.0 by demonstrating how automated text analysis can serve as a diagnostic tool for evaluating OE 5.0 implementation. It bridges information retrieval technologies with continuous improvement methodologies, highlighting a novel path for operational strategy research.

The proliferation of esports in recent years has attracted public and research attention. However, there has been little examination of verbal communication within teams. This study is the first to analyze how esport athletes use verbal communication within teams during a high-stakes competition. This exploratory qualitative analysis focuses on the content of intrateam verbal communication between five teammates competing in Counter Strike: Global Offensive and explores communication’s role in effective performance. Audio logs of a publicly accessible video of a high-stakes esport competition that included full audio of intrateam communication were transcribed verbatim. Subsequently, textual analysis was used to identify emergent themes and patterns in the transcription. Five emergent themes were discovered: emotional feedback, decision-making communication, points of emphasis, moments of uncertainty, and confirmations. The results highlight the importance of esport teams using effective communication, team roles, and leadership. Future research and interventions on effective communication in esport may benefit from testing the emergent themes present in this competition.

A common view in policy and business circles is that certain elements of trade agreements (General Agreement on Tariff and Trade rules, the World Trade Organization Agreement on Agriculture, and the North American Free Trade Agreement) and the Canadian Agreement on Internal Trade significantly limit the policy and program instruments available to support the development of local/sustainable food systems. This exploratory textual analysis of select trade articles, filtered through a local/sustainable lens, suggests that Canadian governments can put in place more substantial policy and program drivers without triggering trade disputes. Of particular note is that local/sustainable foods may not be considered equivalent to imported conventional ones, and therefore many provisions of the trade agreements may not be applicable. Equally important, the rules do permit certain kinds of support, there are numerous exemptions and thresholds for application of measures, and many current actors in local/sustainable implementation may not be subject to the agreements. Based on this textual analysis, pertinent instrument design features are proposed that would allow governments and other parties to support local/sustainable food systems without triggering trade disputes.

This chapter builds on the techniques introduced in the previous Chaps. 12 and 13. Specifically, we will demonstrate how unsupervised pattern recognition approaches can be applied to text data to answer a research question. Because pre-processing and “knowing” your data are especially important when using unsupervised approaches with text, we will introduce additional techniques with an emphasis on exploratory data analysis tools such as token frequency analysis, basic text analytics, and n-gram analysis to explore large text-based datasets before using this information in support of the application of unsupervised natural language processing (NLP) techniques.

“A Home to Live in or Monetize?” An Exploratory Textual Analysis of the Potential Uses of Lisbon and Porto Real Estate During the COVID-19 Pandemic

Sonification is a complementary technique to visualization that uses sound to describe relationships in data. We describe work to aid exploratory textual analysis by sonifying textual variants. The sonification presented focuses on using pitch and tones to help the user listen to differences in the structure between variations of a text or texts encoded in Text Encoding Initiative (TEI) XML. Extracting hyperstructures, we describe our conversion of TEI elements and attributes into sounds for a listener. We discuss our approaches to creating the sounds used to represent the data from the Bodleian Libraries’ First Folio project and early visualizations, and we consider the issues raised by the use of this novel technique. The use of sound provides an exciting alternative way of exploring textual structures to determine differences between them. While the novelty in this area is a major challenge, we suggest that this method can be useful in the exploration of variants between texts marked up with TEI.

Customer reviews on e-commerce platforms have become an essential source of information for understanding user perceptions, satisfaction levels, and product quality. However, most existing studies still focus on sentiment polarity classifying opinions only as positive or negative without examining deeper emotional expressions that may reflect customer experiences more comprehensively. To address this gap, this study applies a Natural Language Processing (NLP) approach using the pre-trained DistilBERT transformer model to detect emotional patterns in women’s fashion product reviews. The dataset, obtained from Kaggle’s Women’s E-Commerce Clothing Reviews, contains approximately 23,000 entries and includes review texts along with demographic and product-related attributes. The research workflow consists of data cleaning, feature engineering, exploratory text analysis, and emotion detection using the DistilBERT-based emotion classifier. All analyses were performed using Python in the Google Colab environment. The results reveal that positive emotions, particularly joy and admiration, dominate customer feedback, indicating strong satisfaction with product fit and quality. Conversely, negative emotions such as anger and sadness appear more frequently in reviews mentioning sizing inconsistencies, fabric issues, or unmet expectations. The combination of sentiment context, emotional tone, and engineered features provides a more nuanced understanding of customer behavior compared to sentiment polarity alone. These findings highlight the potential of emotion-aware analytical approaches to support e-commerce businesses in improving product development, enhancing customer experience, and designing data-driven marketing strategies.

PurposeUsing the affect theory of social exchange, this study investigates the mediating role of students' affective commitment between their personality traits (extraversion and agreeableness) and academic performance.Design/methodology/approachThis research employs mixed-method study, that is exploratory text analysis using 123 responses followed by a survey of 300 responses among the management students to test the proposed model.FindingsResults reveal a direct positive association of extraversion and agreeableness with students' affective commitment towards their academic institution. Additionally, negative indirect effects of affective commitment were found between the two personality traits and academic performance.Originality/valueThe study highlights both positive and negative outcomes of so-called favorable personality types of extraversion and agreeableness, thereby, building a prima facie case for promoting personality diversity in management institutions.