Implementation of Effective Models for Classifying Medical Data Using Text Mining

Self-attention based recurrent convolutional neural network for disease prediction using healthcare data

This study aims to develop high-performing Machine Learning and Deep Learning models in predicting hospital length of stay (LOS) while enhancing interpretability. We compare performance and interpretability of models trained only on structured tabular data with models trained only on unstructured clinical text data, and on mixed data. The structured data was used to train fourteen classical Machine Learning models including advanced ensemble trees, neural networks and k-nearest neighbors. The unstructured data was used to fine-tune a pre-trained Bio Clinical BERT Transformer Deep Learning model. The structured and unstructured data were then merged into a tabular dataset after vectorization of the clinical text and a dimensional reduction through Latent Dirichlet Allocation. The study used the free and publicly available Medical Information Mart for Intensive Care (MIMIC) III database, on the open AutoML Library AutoGluon. Performance is evaluated with respect to two types of random classifiers, used as baselines. The best model from structured data demonstrates high performance (ROC AUC = 0.944, PRC AUC = 0.655) with limited interpretability, where the most important predictors of prolonged LOS are the level of blood urea nitrogen and of platelets. The Transformer model displays a good but lower performance (ROC AUC = 0.842, PRC AUC = 0.375) with a richer array of interpretability by providing more specific in-hospital factors including procedures, conditions, and medical history. The best model trained on mixed data satisfies both a high level of performance (ROC AUC = 0.963, PRC AUC = 0.746) and a much larger scope in interpretability including pathologies of the intestine, the colon, and the blood; infectious diseases, respiratory problems, procedures involving sedation and intubation, and vascular surgery. Our results outperform most of the state-of-the-art models in LOS prediction both in terms of performance and of interpretability. Data fusion between structured and unstructured text data may significantly improve performance and interpretability.

Every day a huge amount of unstructured and semi structured data is used in all the business sectors. Those data are very complicated to store and process for applying into the decision-making system. Especially, the medical data, clinical data and patient history data are to be accessed in a faster way to bring the feasible solution. In view of this, a high speed, reliable and fault tolerant programming framework is needed [1].Apache Pig is a high level and globally accepted programming language to execute the map reduce tasks over the Hadoop cluster while dealing with unstructured data. This language works on Hadoop Distributed File System (HDFS) and this language is written in Java.In this proposed work, the medical history data of patients are considered to be processed. The existing approaches such as oracle SQL queries and mongo DB based results have been producing the very slower time to process these records. But in pig programming language, these gaps are rectified and produced an efficient result.The data dictionary of this implemented dataset is having 7 fields of records to be process in a phased approach. Each and every phase of analysis, the various fields are considered for further processing and interpretation. With the help of relationship operator’s, the relationship of these dataset fields is identified. Followed by this, the functions are applied to produce the faster segregation on the given dataset. Two types of functions are used here such as math function and evaluation function [2].

The significance of big data is increasingly acknowledged across all sectors, including medicine. Moreover, the trend of data trading is on the rise, particularly in exchanging other data for medical data to rejuvenate the medical industry. This study aimed to discern the facilitating factors of healthcare data trade. We assessed five medical data market platforms on October, 2022, based on three criteria: (a) clarity in articulating the data for sale; (b) transparency in specifying the data costs; and (c) explicit indication that payment grants data access. This helped identify the traded medical data types. Additionally, we anonymously surveyed 43 representatives from medical device companies about their demand for medical data trading, achieving a response rate of 66%. Of the medical data traded on these platforms, 93.34% was structured, while 5.66% was unstructured, indicating an imbalance. Although there was a higher demand for structured medical data, there was also interest in purchasing unstructured medical data. Unstructured big data are crucial for medical device development, fueling the demand for trading such data. Many stakeholders view the data market as essential and are willing to procure medical data. Consequently, medical device companies will need methods to acquire unstructured medical data for developing innovative and enhanced medical devices.

Medical data is increasing rapidly. The rapid fusion of medical record text data, medical laboratory data and image data in many regional hospitals have brought light to the screening, diagnosis and treatment of diseases. Medical text data records the patient's detailed condition and treatment process, with abundant information. As a branch of artificial intelligence, deep learning is gradually penetrating the medical field. The application of deep learning in the medical field shows great application prospects. Therefore, mining the knowledge contained in medical texts can provide better medical services for the majority of patients, which has good theoretical and practical significance. In this paper, the content of medical texts was analyzed, and the test questions and standard answers of the USMLE Step 2 Clinical Skills Exam were used as data sets to explore the core content of the medical chief texts. After comparing and analyzing different methods, such as BERT and DeBERTa, an intelligent follow-up system based on DeBERTa was finally proposed, hoping to help clinicians read medical data more effectively and weaken the problem of poor diagnosis results caused by personal academic ability differences.

Healthcare professionals produce abounding textual data in their daily clinical practice. Text mining can yield valuable insights from unstructured data. Extracting insights from multiple information sources is a major challenge in computational medicine. In this study, our objective was to illustrate how combining text mining techniques with statistical methodologies can yield new insights and contribute to the development of neurological and neuromuscular-related health information. We demonstrate how to utilize and derive knowledge from medical text, identify patient groups with similar diagnostic attributes, and examine differences between groups using demographical data and past medical history (PMH). We conducted a retrospective study for all patients who underwent electrodiagnostic (EDX) evaluation in Israel's Sheba Medical Center between May 2016 and February 2022. The data extracted for each patient included demographic data, test results, and unstructured summary reports. We conducted several analyses, including topic modeling that targeted clinical impressions and topic analysis to reveal age- and sex-related differences. The use of suspected clinical condition text enriched the data and generated additional attributes used to find associations between patients' PMH and the emerging diagnosis topics. We identified 6096 abnormal EMG results, of which 58% (n = 3512) were males. Based on the latent Dirichlet allocation algorithm we identified 25 topics that represent different diagnoses. Sex-related differences emerged in 7 topics, 3 male-associated and 4 female-associated. Brachial plexopathy, myasthenia gravis, and NMJ Disorders showed statistically significant age and sex differences. We extracted keywords related to past medical history (n = 37) and tested them for association with the different topics. Several topics revealed a close association with past medical history, for example, length-dependent symmetric axonal polyneuropathy with diabetes mellitus (DM), length-dependent sensory polyneuropathy with chemotherapy treatments and DM, brachial plexopathy with motor vehicle accidents, myasthenia gravis and NMJ disorders with botulin treatments, and amyotrophic lateral sclerosis with swallowing difficulty. Summarizing visualizations were created to easily grasp the results and facilitate focusing on the main insights. In this study, we demonstrate the efficacy of utilizing advanced computational methods in a corpus of textual data to accelerate clinical research. Additionally, using these methods allows for generating clinical insights, which may aid in the development of a decision-making process in real-life clinical practice.

The availability of massive sets of unstructured data has opened up new opportunities for businesses to gain meaningful insights into what is currently happening in their organization and market place. Unfortunately, unstructured data is often challenging to work with, as it requires specialized toolsets, techniques, knowledge and skills to engage with the data. This study used a systematic review (SLR) process to explore the current and conversant state of affairs in the field of text mining (TM) techniques that exist for the processing of natural language in unstructured text datasets across a broad expanse of applications and domains. A comprehensive literature search yielded 1022 eligible articles from five prominent bibliographic databases, narrowed down to 89 articles for review. Information related to TM techniques, the TM process, applications, challenges and recommendations for the improvement of TM results were extracted and synthesized. Eighteen TM techniques were identified and used to complete a variety of tasks such as data pre-processing, information retrieval, information extraction, text classification, clustering, topic modeling, text summarization and sentiment analysis or opinion mining. No single TM technique may be suitable for all text representation and extraction requirements. Therefore, context and applicability are key factors for TM technique selection.KeywordsText miningText mining techniquesUnstructured dataKnowledge discovery

The development of computer technology is becoming more and more mature. The application of artificial intelligence to the medical field has made important contributions to medical diagnosis and auxiliary detection. Using the powerful computing power of computers to replace humans in the automatic diagnosis of complex diseases has been enthusiastic by the majority of scientific researchers. In the current medical imaging diagnosis system, the location of the lesion is mainly found by observing the two‐dimensional medical image sequence, and the signs of biological information cannot be accurately displayed. Through medical image analysis technology, two‐dimensional images will be divided into three‐dimensional models after image segmentation, image recognition, and three‐dimensional imaging. In this way, it seems that the doctor can “hold the image volume data,” which can greatly improve the scientific and accurate diagnosis. Sex—this article applies artificial intelligence to medical imaging, combined with embedded technology, RFID technology and signal processing technology, and applies the new coronavirus pneumonia image to the artificial intelligence environment after processing, assisting doctors in diagnosis of the disease, and providing relevant information about patients record and manage the diagnosis and diagnosis, save and accumulate the experience and knowledge of famous doctors through the expert system, and then perform corresponding operations and analysis. Through the medical image intelligent analysis system, the safety risk of medical imaging artificial intelligence diagnosis is reduced from 81% to 11%, which greatly reduces the hidden safety hazards for doctors and patients, reduces the workload of doctors, and also reduces the cost of medical care by 79% It is reduced to 20%, which reduces the waiting time of patients and achieves the purpose of improving the accuracy of diagnosis.

The prognosis in patients with relapsed Ewing sarcoma is unfavorable. Our investigation identifies factors predicting for the outcome following relapse. We analyzed type of relapse, time to relapse and overall survival after relapse (OSr) in 714 patients with first recurrence. All patients had been treated within the Cooperative Ewing Sarcoma Studies (CESS) 81 or 86, or the European Intergroup CESS (EICESS 92). OSr time was calculated from diagnosis of first relapse to last follow-up or death. Median follow-up time from diagnosis of primary disease was 2.2 years (mean = 4.0; range: 0.2-24.9). Relapse sites were local in 15%, combined local and systemic in 12%, and systemic in 73%. Among patients with a localized primary tumor, 20% relapsed locally, while 12% showed combined and 68% systemic relapse. When the primary disease was disseminated, 82% developed systemic, 13% combined, and 5% local relapse. Five-year OSr was 0.13 (SE = 0.01). Outcome following local relapse, with a 5-year survival rate of 0.24 (P < 0.001), was superior to outcome after systemic or combined recurrence. Five-year OSr was 0.07 (SE = 0.01) in patients who relapsed 0-2 years after the diagnosis of primary disease, as compared to a 5-year OSr of 0.29 (SE = 0.03) when relapse occurred later. 5-year OSr in Ewing sarcoma is poor (<0.2). Prognostically favorable factors are: late onset (>2 years) and strictly localized relapse.

In the digital era, the rapid growth of social media and online platforms has led to an explosion of unstructured textual data that holds significant business value. Traditional marketing strategies, once reliant on structured data such as demographics and purchase history, now benefit from insights derived from text analytics and sentiment analysis. This paper explores the integration of structured and unstructured data to strengthen marketing intelligence and customer segmentation. By utilizing text mining techniques and Natural Language Processing (NLP), unstructured data such as customer reviews and comments can be analyzed to extract sentiments, identify emerging trends, and refine customer relationship strategies. The study proposes an integrated framework that combines data extraction, transformation, and loading (ETL) processes with a data warehouse system for unified analysis. Using clustering algorithms such as K-Means and visualization tools, insights into customer behavior, preferences, and market segmentation are revealed. The paper also discusses the challenges of handling multilingual and context-dependent text, ethical and privacy considerations, and the technical architecture necessary for business intelligence implementation. Findings suggest that effective integration of textual analytics with structured data can lead to more informed decision-making, improved marketing strategies, and stronger customer engagement.

The automatic diagnosis of skin diseases is of great importance, especially in cases requiring early detection, as it accelerates clinical processes and reduces the margin of error. In this study, a classification model based on Convolutional Neural Network (CNN) architectures was developed on a multi-class visual dataset containing three different skin disease categories. To enhance the model’s performance, data augmentation techniques were applied, and the images were resized to 224×224 pixels. Using a transfer learning approach, the model was trained with preprocessing ResNet-18, AlexNet, and DenseNet-201 architectures. The hyperparameters used during the training process were carefully selected, and the model's training and validation accuracies were monitored. According to the results obtained, the ResNet-18 model demonstrated strong performance with an accuracy of 87.19% on the test set. These findings indicate that deep learning-based architectures can be effectively applied in the multi-class diagnosis of skin diseases.

With the progress of scientific level, artificial intelligence has been applied more and more in the medical field. A large number of medical data are mainly characterized by unstructured information. Through the analysis of these information, a large number of medical case data can be generated, thus generating related value-added information. With the rapid development of artificial intelligence represented by deep learning, artificial intelligence has broad application prospects in clinical practice. The integration of artificial intelligence technology and medical field under the environment of big data is an important trend in the development of medical field. Artificial intelligence technology can effectively analyze and process massive medical data, assist doctors to improve the accuracy and efficiency of diagnosis and treatment, and bring more personalized treatment measures to patients. With the development of medical technology, the development of artificial intelligence medical technology provides a new idea for clinical medicine. The application of artificial intelligence technology in the field of medical diagnosis is studied in this paper.

Hair, having distinct stages of growth, is a dynamic component of the integumentary system. Nonetheless, derangement in its structure and growth pattern often provides vital clues for the diagnosis of systemic diseases. Assessment of the hair structure by various microscopy techniques is, hence, a valuable tool for the diagnosis of several systemic and cutaneous disorders. Systemic illnesses like Comel-Netherton syndrome, Griscelli syndrome, Chediak Higashi syndrome, and Menkes disease display pathognomonic findings on hair microscopy which, consequently, provide crucial evidence for disease diagnosis. With minimal training, light microscopy of the hair can easily be performed even by clinicians and other health care providers which can, thus, serve as a useful tool for disease diagnosis at the patient’s bedside. This is especially true for resource-constrained settings where access and availability of advanced investigations (like molecular diagnostics) is a major constraint. Despite its immense clinical utility and non-invasive nature, hair microscopy seems to be an underutilized diagnostic modality. Lack of awareness regarding the important findings on hair microscopy may be one of the crucial reasons for its underutilization. Herein, we, therefore, present a comprehensive overview of the available methods for hair microscopy and the pertinent findings that can be observed in various diseases.

Determination of Imaging Biomarkers to Decipher Disease Trajectories and Differential Diagnosis of Neurodegenerative Diseases (DIsease TreND)

Research in disease diagnosis is a challenging task due to inconsistent, class imbalance, conflicting, and the high dimensionality of medical data sets. The excellent features of each data set play an important role in improving performance of classifiers that may follow either iterative or non-iterative approaches. In the present study, a comparative study is carried out to show the performance of iterative and non-iterative classifiers in combination with genetic algorithm (GA)-based feature selection approach over some widely used medical data sets. The experiment assists to identify the clinical data sets for which feature reduction is necessary for improving performance of classifiers. For iterative approaches, two popular classifiers, namely C4.5 and RIPPER, are chosen, whereas k-NN and naïve Bayes are taken as non-iterative learners. Fourteen real-world medical domain data sets are selected from the University of California, Irvine (UCI Repository) for conducting experiment over the learners.