Functional and embedded dependency inference: a data mining point of view

Medical data mining and knowledge discovery can benefit from the experience and knowledge of clinicians, however, the implementation of this data mining system is challenging. Unlike traditional data mining methods, in this class of applications we process data with some posterior knowledge and the target function is more complex and even may include the opinion of user. Despite the success of the classical reasoning algorithms in many common data mining applications, they failed to address medical record processing where we need to extract information from incomplete, small samples along with an external rulebase to generate ‘meaningful’ interpretation of biological phenomenon. Swarm intelligence is an alternative class of flexible approaches that is promising in data mining. With full control over the rule extraction target function, particle swarm optimization (PSO) is a suitable approach for data mining subject to a rulebase which defines the quality of rules and constancy with previous observations. In this chapter we describe a complex clinical problem that has been addressed using PSO data mining. A large group of temporal lobe epilepsy patients are studied to find the best surgery candidates. Since there are many parameters involved in the decision process, the problem is not tractable from traditional data mining point of view, while the new approach that uses the field knowledge could extract valuable information. The proposed method allows expert to interaction with data mining process by offering manual manipulation of generated rules. The algorithm adjusts the rule set with regard to manipulations. Each rule has a reasoning which is based on the provided rulebase and similar observed cases. Support vector machine (SVM) classifier and swarm data miner are integrated to handle joint processing of raw data and rules. This approach is used to establish the limits of observations and build decision boundaries based on these critical observations.

Big Data Analytics (BDA) is going to be discussed using data mining algorithms in this paper. Develop security in BDA fields for data security to achieve confidential storage. Big Data refers to data sets that are very vast, very complicated, and very dynamic and that have many independent sources. Big data is quickly implemented in all research and engineering fields, including physical, biological, and biomedical science, thanks to rapid developments in networking, data storage, and data gathering capacities. From a data mining point of view, this study gives a theorem characterizing the aspects of the Big Data revolution and suggests a Big Data processing model. This data-driven paradigm considers security and privacy concerns, data mining and analysis, and the modeling of user interests. Here, takes a look at a few of the knottier aspects of big data and the data-driven approach, and calculate the precisions, recalls, and measures, which are 90% of different data mining algorithms.

Sequence analysis is a major area in bioinformatics encompassing the methods and techniques for studying the biological sequences, DNA, RNA, and proteins, on the linear structure level. The focus of this area is generally on the identification of intra– and inter–molecular similarities. Identifying intra–molecular similarities boils down to detecting repeated segments within a given sequence, while identifying inter–molecular similarities amounts to spotting common segments among two or multiple sequences. From a data mining point of view, sequence analysis is nothing but string- or pattern mining specific to biological strings. For a long time, this point of view, however, has not been explicitly embraced neither in the data mining nor in the sequence analysis text books, which may be attributed to the co-evolution of the two apparently independent fields. In other words, although the word “data-mining” is almost missing in the sequence analysis literature, its basic concepts have been implicitly applied. Interestingly, recent research in biological sequence analysis introduced efficient solutions to many problems in data mining, such as querying and analyzing time series [49,53], extracting information from web pages [20], fighting spam mails [50], detecting plagiarism [22], and spotting duplications in software systems [14].

During recent years, the empirical research on corruption has grown considerably. Possible links between government corruption and terrorism have attracted an increasing interest in this research field. Most of the existing literature discusses the topic from a socio-economical perspective and only few studies tackle this research field from a data mining point of view. In this paper, we apply data mining techniques onto a cross-country database linking macro-economical variables to perceived levels of corruption. In the first part, self organizing maps are applied to study the interconnections between these variables. Afterwards, support vector machines are trained on part of the data and used to forecast corruption for other countries. Large deviations for specific countries between these models’ predictions and the actual values can prove useful for further research. Finally, projection of the forecasts onto a self organizing map allows a detailed comparison between the different models’ behavior.

Students starting university have different characteristics, which can impact their performance in the classroom. In this study, 743 freshmen were surveyed. The collected variables are grouped into five categories: demographic data, learning approach, personality, emotional intelligence, and perceived social support. These characteristics provide a profile of the student that will impact their behavior and academic performance during their university life. Based on these data, we have applied data mining techniques in order to build patterns of behavior that represent correlations between the characteristics of the students. Our results highlight the importance of using pattern mining techniques on data associated with the psychological evaluation of new university students.

Nowadays, the representation of many real word problems needs to use some type of relational model. As a consequence, information used by a wide range of systems has been stored in multi relational tables. However, from a data mining point of view, it has been a problem, since most of the traditional data mining algorithms have not been originally proposed to handle this type of data without discarding relationship information. Aiming to ameliorate this problem, we propose a hierarchical approach for handling relational data. In this approach the relational data is converted into a hierarchical structure (the main table as the root and the relations as the nodes). This hierarchical way to represent relational data can be used either for classification or clustering purposes. In this paper, we will use it in clustering algorithms. In order to do so, we propose a hierarchical distance metric to compute the similarity between the tables. In the empirical analysis, we will apply the proposed approach in two well-known clustering algorithms (k-means and agglomerative hierarchical). Finally, this paper also compares the effectiveness of our approach with one existing relational approach.

Preserving logical and functional dependencies in synthetic tabular data

Over the last decade data warehousing and data mining tools have evolved from research into a unique and popular applications, ranging from data warehousing and data mining for decision support to business intelligence and other kind of applications. The chapter presents and discusses data warehousing methodologies along with the main components of data mining tools and technologies and how they all could be integrated together for knowledge management in a broader sense. Knowledge management refers to the set of processes developed in an organization to create, extract, transfer, store and apply knowledge. The chapter also focuses on how data mining tools and technologies could be used in extracting knowledge from large databases or data warehouses. Knowledge management increases the ability of an organization to learn from its environment and to incorporate knowledge into the business processes by adapting to new tools and technologies. Knowledge management is also about the reusability of the knowledge that is being extracted and stored in the knowledge base. One way to improve the reusability is to use this knowledge base as front-ends to case-based reasoning (CBR) applications. The chapter further focuses on the reusability issues of knowledge management and presents an integrated framework for knowledge management by combining data mining (DM) tools and technologies with CBR methodologies. The purpose of the integrated framework is to discover, validate, retain, reuse and share knowledge in an organization with its internal users as well as its external users. The framework is independent of application domain and would be suitable for uses in areas, such as data mining and knowledge management in e-government.

As an analysis tool either of on-line analytical processing (OLAP) or data mining has its own strong points and weaknesses. It is an effective way to enhance the power and flexibility of data mining in data warehouses and large databases by integrating data mining with OLAP to offset their weaknesses. In the paper, a proposal of integrating data mining and OLAP is put forward. The mechanism of the on-line analytical data mining in data warehouses is described. Aspects which are necessary for developing a successful on-line analytical data mining system is also discussed.

Mining frequent patterns is widely used to discover knowledge from a database. It was originally applied on Market Basket Analysis (MBA) problem which represents the Boolean databases. In those databases, only the existence of an article (item) in a transaction is defined. However, in real-world application, the gathered information generally suffer from imperfections. In fact, a piece of information may contain two types of imperfection: imprecision and uncertainty. Recently, a new database representing and integrating those two types of imperfection were introduced: Evidential Database. Only few works have tackled those databases from a data mining point of view. In this work, we aim to discuss evidential itemset’s support. We improve the complexity of state of art methods for support’s estimation. We also introduce a new support measure gathering fastness and precision. The proposed methods are tested on several constructed evidential databases showing performance improvement.

The explosion of very large databases has created extraordinary opportunities for monitoring, analyzing and predicting global economical, geographical, demographic, medical, political, and other processes in the world. Statistical analysis and data mining techniques have emerged for these purposes. Data mining is the process of discovering previously unknown but potentially useful patterns, rules, or associations from huge quantity of data. Data mining can be performed on different data repositories such as relational databases, data warehouses, transactional databases, sequence databases, spatial databases, spatio-temporal databases, and text databases, etc. Typically, data mining functionalities can be classified into two categories: descriptive and predictive. Descriptive mining tasks aim at characterizing the general properties of the data in the databases, while predictive mining tasks perform inherence on the current data in order to make prediction in future.KeywordsRoot Mean Square ErrorData MiningAssociation RuleIntrusion DetectionAnomaly DetectionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Evidence-based medicine is an important initiative of the U.S. National Institutes of Health. However, published data are limited. Medical knowledge can be advanced through pharmacoepidemiology and data mining research of very large databases, data warehouses. Before Vioxx (rofecoxib) and Celebrex (celecoxib) were publicly suspected of being associated with acute myocardial infarction and stroke, etodolac was evaluated for gastrointestinal safety using a data warehouse constructed from several data sources and data mining tools embedded with pharmacoepidemiologic methods. Results were published in a top tier medical journal, and established gastrointestinal safety of a generic COX-2 analgesic medication. This project demonstrates that the knowledge needed for evidence-based medicine can be augmented through pharmacoepidemiology research using data mining and data warehousing.

Extending classical functional dependencies for physical database design

Schema merging is the process of incorporating data models into an integrated, consistent schema from which query solutions satisfying all incorporated models can be derived. The efficiency of such a process is reliant on the effective semantic representation of the chosen data models, as well as the mapping relationships between the elements of the source data models. Consider a scenario where, as a result of company mergers or acquisitions, a number of related, but possible disparate data marts need to be integrated into a global data warehouse. The ability to retrieve data across these disparate, but related, data marts poses an important challenge. Intuitively, forming an all-inclusive data warehouse includes the tedious tasks of identifying related fact and dimension table attributes, as well as the design of a schema merge algorithm for the integration. Additionally, the evaluation of the combined set of correct answers to queries, likely to be independently posed to such data marts, becomes difficult to achieve. Model management refers to a high-level, abstract programming language designed to efficiently manipulate schemas and mappings. Particularly, model management operations such as match, compose mappings, apply functions and merge, offer a way to handle the above-mentioned data integration problem within the domain of data warehousing. In this research, we introduce a methodology for the integration of star schema source data marts into a single consolidated data warehouse based on model management. In our methodology, we discuss the development of three (3) main streamlined steps to facilitate the generation of a global data warehouse. That is, we adopt techniques for deriving attribute correspondences, and for schema mapping discovery. Finally, we formulate and design a merge algorithm, based on multidimensional star schemas; which is primarily the core contribution of this research. Our approach focuses on delivering a polynomial time solution needed for the expected volume of data and its associated large-scale query processing. The experimental evaluation shows that an integrated schema, alongside instance data, can be derived based on the type of mappings adopted in the mapping discovery step. The adoption of Global-And-Local-As-View (GLAV) mapping models delivered a maximally-contained or exact representation of all fact and dimensional instance data tuples needed in query processing on the integrated data warehouse. Additionally, different forms of conflicts, such as semantic conflicts for related or unrelated dimension entities, and descriptive conflicts for differing attribute data types, were encountered and resolved in the developed solution. Finally, this research has highlighted some critical and inherent issues regarding functional dependencies in mapping models, integrity constraints at the source data marts, and multi-valued dimension attributes. These issues were encountered during the integration of the source data marts, as it has been the case of evaluating the queries processed on the merged data warehouse as against that on the independent data marts.

Named entity recognition (NER) plays a vital role in extracting critical medical entities from health care records, facilitating applications such as clinical decision support and data mining. Developing robust NER models for low-resource languages, such as Estonian, remains a challenge due to the scarcity of annotated data and domain-specific pretrained models. Large language models (LLMs) have proven to be promising in understanding text from any language or domain. This study addresses the development of medical NER models for low-resource languages, specifically Estonian. We propose a novel approach by generating synthetic health care data and using LLMs to annotate them. These synthetic data are then used to train a high-performing NER model, which is applied to real-world medical texts, preserving patient data privacy. Our approach to overcoming the shortage of annotated Estonian health care texts involves a three-step pipeline: (1) synthetic health care data are generated using a locally trained GPT-2 model on Estonian medical records, (2) the synthetic data are annotated with LLMs, specifically GPT-3.5-Turbo and GPT-4, and (3) the annotated synthetic data are then used to fine-tune an NER model, which is later tested on real-world medical data. This paper compares the performance of different prompts; assesses the impact of GPT-3.5-Turbo, GPT-4, and a local LLM; and explores the relationship between the amount of annotated synthetic data and model performance. The proposed methodology demonstrates significant potential in extracting named entities from real-world medical texts. Our top-performing setup achieved an F1-score of 0.69 for drug extraction and 0.38 for procedure extraction. These results indicate a strong performance in recognizing certain entity types while highlighting the complexity of extracting procedures. This paper demonstrates a successful approach to leveraging LLMs for training NER models using synthetic data, effectively preserving patient privacy. By avoiding reliance on human-annotated data, our method shows promise in developing models for low-resource languages, such as Estonian. Future work will focus on refining the synthetic data generation and expanding the method's applicability to other domains and languages.