Data preparation: Art or science?

The knowledge discovery and data mining (KDDM) process models describe the various phases (e.g. business understanding, data understanding, data preparation, modeling, evaluation and deployment) of the KDDM process. They act as a roadmap for implementation of the KDDM process by presenting a list of tasks for executing the various phases. The checklist approach of describing the tasks is not adequately supported by appropriate tools, which specify ‘how’ the particular task can be implemented. This may result in tasks not being implemented. Another disadvantage is that the long checklist does not capture or leverage the dependencies that exist among the various tasks of the same and different phases. This not only makes the process cumbersome to implement, but also hinders possibilities for semi-automation of certain tasks. Given that each task in the process model serves an important goal and even affects the execution of related tasks due to the dependencies, these limitations are likely to negatively affect the efficiency and effectiveness of KDDM projects. This paper proposes an improved KDDM process model that overcomes these shortcomings by prescribing tools for supporting each task as well as identifying and leveraging dependencies among tasks for semi-automation of tasks, wherever possible.

Analysing datasets using data mining techniques can enhance decision making in organizations. However, to ensure that the full potential of these techniques is realised it is important that decision makers understand there are Knowledge Discovery and Data Mining (KDDM) processes that are mature enough to be adopted. This paper demonstrates the benefits of using a KDDM process to evaluate survey data for internet banking users in Jamaica which includes demographic as well as attitudinal and behavioral variables. The major benefits of following this process include the selection of a set of models, rather than a single model, which are more relevant to the business/research objectives and use of a more targeted knowledge discovery process as the data mining analyst is now directed to consider the effects the decisions in each phase will have on subsequent phases. This leads to more relevant knowledge being extracted from the data mining process.

New information and communication technologies are changing the way products are developed, manufactured, serviced and managed over the product’s lifecycle. Today’s smart products not only consist of their physical components, but are also endowed with intelligence. Data and the capabilities to process data into knowledge and eventually decisions have become critical components of the product itself and of the process to develop/operate the product. This paper investigates how engineers and a new functional role, data scientists, can effectively collaborate in a mixed team for new product development with data-driven features (NPD3). We focus on the concept development stage, typically the fuzziest phase of product development. In this paper, an integrated process model is explored by revisiting the traditional new product development (NPD) process model as well as the knowledge discovery and data mining (KDDM) process model. Then a case study of the development of an application-specific unmanned aircraft system (UAS) is used to examine the proposed model.

The development of modern technology allows enterprises to continuously expand the means of financial management. The use of advanced information technology and modern management means makes the financial management of enterprises continue to improve, but there are also many problems. In the face of various types of risks that enterprises may face at present, in addition to strengthening financial accounting, enterprises should also recognize the various risks that the double-edged sword of financial management informatization may bring to enterprise financial management and carefully analyze the reasons for the emergence of risks, from the ideological and technical means to continuously improve the entire process of financial management informatization, which is an important topic for every financial worker to discuss. This paper mainly studies and innovates the data mining process and the support vector machine model and designs the data preprocessing method in the data mining process to perform feature selection and optimize the parameters of the support vector machine model. The specific process is as follows: based on CRISP-DM, the industry-standard model of the data mining process, a series of data preprocessing, eigenvalue extraction, parameter optimization, training set pruning, and other methods are designed from business understanding, data understanding, data preparation, and other aspects to improve data quality. In the traditional support vector machine, when the test sample is located at the boundary point of the hyperplane, the judgment may be wrong. In the aspect of SVM model improvement, according to the discrimination method of SVM, the weighted K-nearest neighbor algorithm is introduced to redistinguish the qualified test samples in the feature space. From the whole process of data mining, a data mining system is designed, which is “data preprocessing standardization + genetic algorithm feature selection + training set pruning + support vector machine classifier discrimination optimization.” Finally, this paper uses a series of data mining optimization methods designed to mine the actual financial data of listed companies and has achieved good results.

The large number of government and industry activities supporting the Unit of Action (UA), with attendant documents, reports and briefings, can overwhelm decision-makers with an overabundance of information that hampers the ability to make quick decisions often resulting in a form of gridlock. In particular, the large and rapidly increasing amounts of data and data formats stored on UA Advanced Collaborative Environment (ACE) servers has led to the realization that it has become impractical and even impossible to perform manual analysis leading to timely decisions. UA Program Management (PM UA) has recognized the need to implement a Decision Support System (DSS) on UA ACE. The objective of this document is to research the commercial Knowledge Discovery and Data Mining (KDDM) market and publish the results in a survey. Furthermore, a ranking mechanism based on UA ACE-specific criteria has been developed and applied to a representative set of commercially available KDDM solutions. In addition, an overview of four R&D areas identified as critical to the implementation of DSS on ACE is provided. Finally, a comprehensive database containing detailed information on surveyed KDDM tools has been developed and is available upon customer request.

A critical issue in data mining and knowledge discovery is the problem of data quality. Quantifying the presence of noise in dataset is often used as an indicator of data quality. While existing wo...

BackgroundThe Coronavirus Disease 2019 (COVID-19) pandemic was a huge shock to society, and the ensuing information problems had a huge impact on society at the same time. The urgent need to understand the Infodemic, i.e., the importance of the spread of false information related to the epidemic, has been highlighted. However, while there is a growing interest in this phenomenon, studies on the topic discovery, data collection, and data preparation phases of the information analysis process have been lacking.ObjectiveSince the epidemic is unprecedented and has not ended to this day, we aimed to examine the existing Infodemic-related literature from January 2019 to December 2022.MethodsWe have systematically searched ScienceDirect and IEEE Xplore databases with some search limitations. From the searched literature we selected titles, abstracts and keywords, and limitations sections. We conducted an extensive structured literature search and analysis by filtering the literature and sorting out the available information.ResultsA total of 47 papers ended up meeting the requirements of this review. Researchers in all of these literatures encountered different challenges, most of which were focused on the data collection step, with few challenges encountered in the data preparation phase and almost none in the topic discovery section. The challenges were mainly divided into the points of how to collect data quickly, how to get the required data samples, how to filter the data, what to do if the data set is too small, how to pick the right classifier and how to deal with topic drift and diversity. In addition, researchers have proposed partial solutions to the challenges, and we have also proposed possible solutions.ConclusionsThis review found that Infodemic is a rapidly growing research area that attracts the interest of researchers from different disciplines. The number of studies in this field has increased significantly in recent years, with researchers from different countries, including the United States, India, and China. Infodemic topic discovery, data collection, and data preparation are not easy, and each step faces different challenges. While there is some research in this emerging field, there are still many challenges that need to be addressed. These findings highlight the need for more articles to address these issues and fill these gaps.

Feature extraction from biological data is a very important discipline in bioinformatics. The aim of this work is to classify protein sequences automatically. To do this, it seemed appropriate to use a data mining process: the process of Knowledge Discovery and Data mining (KDD) from biological data. We are interested in the first phase of the KDD, that consists in the preprocessing and we focus on the step: Feature extraction. Feature extraction is translated by the generation of a set of feature that is presented to a supervised learning algorithm for classification. An extraction method that we have adopted is the method of N-grams. The algorithm of n-grams consists in extracting feature of fixed size of length n. In this paper, we propose a hierarchical algorithm of construction of n-grams to obtain feature of variable sizes. This algorithm of extraction is used to meet the needs of biologists. By using the linear classifier SVM, the experiments on real protein banks show the efficiency of our algorithm while presenting a comparison of our work to previous works.

计算机辅助诊断系统中,使用医学数据库的数据挖掘和知识发现技术具有重要意义和其发展的必然性.文章分析和综合了此类系统研究开发的一般规律及其特殊性,并指出了其中一些需要解决的问题和今后的研究发展方向.

Introduction Harvey J. Miller and Jiawei Han Spatiotemporal Data Mining Paradigms and Methodologies John F. Roddick and Brian G. Lees Fundamentals of Spatial Data Warehousing for Geographic Knowledge Discovery Yvan Bedard and Jiawei Han Analysis of Spatial Data with Map Cubes: Highway Traffic Data Chang-Tien Lu, Arnold P. Boedihardjo, and Shashi Shekhar NEW! Data Quality Issues and Geographic Knowledge Discovery Marc Gervais, Yvan Bedard, Marie-Andree Levesque, Eveline Bernier, and Rodolphe Devillers Spatial Classification and Prediction Models for Geospatial Data Mining Shashi Shekhar, Ranga Raju Vatsavai, and Sanjay Chawla An Overview of Clustering Methods in Geographic Data Analysis Jiawei Han, Jae-Gil Lee, and Micheline Kamber NEW! Computing Medoids in Large Spatial Datasets Kyriakos Mouratidis, Dimitris Papadias, Spiros Papadimitriou NEW! Looking for a Relationship? Try GWR A. Stewart Fotheringham, Martin Charlton, and Urska Demsar Leveraging the Power of Spatial Data Mining to Enhance the Applicability of GIS Technology Donato Malerba, Antonietta Lanza, and Annalisa Appice Visual Exploration and Explanation in Geography: Analysis with Light Mark Gahegan NEW! Multivariate Spatial Clustering and Geovisualization Diansheng Guo NEW! Toward Knowledge Discovery about Geographic Dynamics in Spatiotemporal Databases} May Yuan NEW! The Role of a Multitier Ontological Framework in Reasoning to Discover Meaningful Patterns of Sustainable Mobility Monica Wachowicz, Jose Macedo, Chiara Renso, and Arend Ligtenberg NEW! Periodic Pattern Discovery from Trajectories of Moving Objects Huiping Cao, Nikos Mamoulis, and David W. Cheung NEW! Decentralized Spatial Data Mining for Geosensor Networks Patrick Laube and Matt Duckham NEW! Beyond Exploratory Visualization of Space-Time Paths Menno-Jan Kraak and Otto Huisman

Formal Concept Analysis (FCA) is a valid tool for data mining and knowledge discovery, which identifies conceptual structures from (formal) contexts. As many practical applications involve non-bina...

Data preparation is the most crucial stage before any machine learning algorithm can be applied. Equally important is text data preparation for many areas such as information retrieval, data mining, information extraction and natural language processing. In this paper, how text data preparation for short-text automated assessment grading is implemented in RapidMiner will be discovered. RapidMiner has becoming a very popular tool in field of data exploration, data preparation, modelling, scoring, automation and many other data science related. It is found that RapidMiner graphical visual environment makes it convenient, simple and fast to design a better model for automated grading. With this, an experiment is carried out to explore this option on short-text automated grading. The popularity of studying automated grading in teaching and learning has also becoming a popular area ever since it was introduced. The main challenge in studying this area is to extract important information out of a collection of unstructured text-based data. Thus, this paper will describe in detail how text data can be prepared in RapidMiner tool in order to discover relevant information for machine learning implementation.

Pavement deflection data are often used to evaluate a pavement’s structural condition nondestructively. Pavement layers are important parameters in view of bearing capacity. Pavement layer thickness may be known from the design project or site investigation. At the same time, using backcalculation analysis, flexible pavement layer thicknesses together with in situ material properties can also be backcalculated from the measured field data through appropriate analysis techniques. Data mining (DM) process has not been used as a backcalculation tool before. In this study, DM process is used in backcalculating the pavement layer thickness from deflections measured on the surface of the flexible pavements. Experimental deflection data groups from NDT are used to show the capability of the DM process in backcalculating the pavement layer thickness and compared each other. Performing the study, modeling with Kstar method gives fine results with respect to other DM modeling techniques. Backcalculation of pavement layer thickness using DM process has been carried out for the first time.

Knowledge Discovery (KD) and Data Mining are two well-known and still growing fields that, with the advancements of data collection and storage technologies, emerged and expanded with great strength by the many possibilities and benefits that exploring and analyzing data can bring. However, it is a task that requires great domain expertise to really achieve its full potential. Furthermore, it is an activity which is done mainly by data experts who know little about specific domains, like the healthcare sector, for example. Thus, in this research, we propose means for allowing domain experts from the medical domain (e.g. doctors and nurses) to also be actively part of the Knowledge Discovery process, focusing in the Data Preparation phase, and use the specific domain knowledge that they have in order to start unveiling useful information from the data. Hence, a guideline based on the CRISP-DM framework, in the format of methods fragments is proposed to guide these professionals through the KD process.

Data mining is the focus of big data applications in various fields. Data pre-processing is a crucial step in the data mining process. With the development of the information society and the application of databases, the educational data has seen explosive growth, and the data on poor students has become informative. However, the actual student financial aid management system collects the data on poor students which generally has problems such as missing values, attributes redundancy, and noise. To solve this problem, we proposed a novel method called DPBP to preprocess data. The proposed DPBP approach consists of four stages: the preparation of data, the scoping of characteristics, the combination of characteristics, and the filtering of missing number. Firstly, we prepare the dataset by extracting data. Next, the characteristic range is limited by choosing experimental results of feature selection algorithm. Then, third stage performs feature combination to obtain the feature decomposition sets. Finally, based on accuracy and missing number, we gain the optimal dataset. Series of experiments result show that our proposed method significantly improves the data quality and stability.