Framework For Data Development in Mordern Situation Using Machine Learning Technology

Data preparation is an iterative-agile process for exploring, combining, cleaning and transforming raw data into curated datasets for self-service data integration, data science, data discovery, and BI/analytics. To perform data preparation, data preparation tools are used by analysts, citizen data scientists and data scientists for self- service. The tools are also used by citizen integrators and data engineers for data enablement to reduce the time and complexity of interactively accessing, cataloging, harmonizing, transforming and modeling data for analytics in an agile manner with metadata and lineage support. These tools can provide data access for use in mostly analytical tasks that include storage, logical and physical data modeling, and data manipulation for data visualization, data integration and analytics. Some tools support machine-learning algorithms that can recommend or even automate actions to augment and accelerate data preparation. Keywords: Data Mining, Data preparation, Data Analysis, Big Data Analytics

ABSTRACTMore than 50 years ago, John Tukey called for a reformation of academic statistics. In “The Future of Data Analysis,” he pointed to the existence of an as-yet unrecognized science, whose subject of interest was learning from data, or “data analysis.” Ten to 20 years ago, John Chambers, Jeff Wu, Bill Cleveland, and Leo Breiman independently once again urged academic statistics to expand its boundaries beyond the classical domain of theoretical statistics; Chambers called for more emphasis on data preparation and presentation rather than statistical modeling; and Breiman called for emphasis on prediction rather than inference. Cleveland and Wu even suggested the catchy name “data science” for this envisioned field. A recent and growing phenomenon has been the emergence of “data science” programs at major universities, including UC Berkeley, NYU, MIT, and most prominently, the University of Michigan, which in September 2015 announced a $100M “Data Science Initiative” that aims to hire 35 new faculty. Teaching in these new programs has significant overlap in curricular subject matter with traditional statistics courses; yet many academic statisticians perceive the new programs as “cultural appropriation.” This article reviews some ingredients of the current “data science moment,” including recent commentary about data science in the popular media, and about how/whether data science is really different from statistics. The now-contemplated field of data science amounts to a superset of the fields of statistics and machine learning, which adds some technology for “scaling up” to “big data.” This chosen superset is motivated by commercial rather than intellectual developments. Choosing in this way is likely to miss out on the really important intellectual event of the next 50 years. Because all of science itself will soon become data that can be mined, the imminent revolution in data science is not about mere “scaling up,” but instead the emergence of scientific studies of data analysis science-wide. In the future, we will be able to predict how a proposal to change data analysis workflows would impact the validity of data analysis across all of science, even predicting the impacts field-by-field. Drawing on work by Tukey, Cleveland, Chambers, and Breiman, I present a vision of data science based on the activities of people who are “learning from data,” and I describe an academic field dedicated to improving that activity in an evidence-based manner. This new field is a better academic enlargement of statistics and machine learning than today’s data science initiatives, while being able to accommodate the same short-term goals. Based on a presentation at the Tukey Centennial Workshop, Princeton, NJ, September 18, 2015.

In this chapter we begin with a detailed explanation about data gathering and preparation process such as data acquisition and data collection (Sects. 4.2 and 4.3). Then a temporal coverage of land use change trend in the study area will be presented (Sect. 4.4). It will be followed by temporal change mapping and change visualisation within the presumed time period to measure urban sprawl in that period (Sects. 4.5–4.7). Furthermore, the demographic changes over this time will be taken into account (Sect. 4.8).

Because of the growing involvement of communities from various disciplines, data science is constantly evolving and gaining popularity. The growing interest in data science-based services and applications presents numerous challenges for their development. Therefore, data scientists frequently turn to various forums, particularly domain-specific Q&A websites, to solve difficulties. These websites evolve into data science knowledge repositories over time. Analysis of such repositories can provide valuable insights into the applications, topics, trends, and challenges of data science. In this article, we investigated what data scientists are asking by analyzing all posts to date on DSSE, a data science-focused Q&A website. To discover main topics embedded in data science discussions, we used latent Dirichlet allocation (LDA), a probabilistic approach for topic modeling. As a result of this analysis, 18 main topics were identified that demonstrate the current interests and issues in data science. We then examined the topics' popularity and difficulty. In addition, we identified the most commonly used tasks, techniques, and tools in data science. As a result, "Model Training", "Machine Learning", and "Neural Networks" emerged as the most prominent topics. Also, "Data Manipulation", "Coding Errors", and "Tools" were identified as the most viewed (most popular) topics. On the other hand, the most difficult topics were identified as "Time Series", "Computer Vision", and "Recommendation Systems". Our findings have significant implications for many data science stakeholders who are striving to advance data-driven architectures, concepts, tools, and techniques.

Data preparation (also called “wrangling” or “cleaning”) – the evaluation and manipulation of data prior to formal analysis – is often dismissed as a precursor to meaningful engagement with a dataset. Here, we re-envision data preparation in light of calls to prepare students for a data-rich world. Traditionally, curricular statistics explorations involve data that are derived from observations that students record themselves or that reflect familiar, relatively closed systems. In contrast, pre-constructed public datasets are much larger in scope and involve temporal, geographic, and other dimensions that complicate inference and blur boundaries between “signal” and “noise.” As a result, students have fewer opportunities to consider sources of variability in such datasets. Due to these constraints, we argue that data preparation becomes an important site for students to reason about variability with public data. Through analyses of repeated task-based interviews with five pairs of adolescent participants, we find that specific actions during data preparation, such as filtering data or calculating new measures, presented opportunities to engage leaners with variability as they prepared and analyzed several public socioscientific datasets. More broadly, our study highlights some changes to theory and curriculum in statistics education that are necessitated by a focus on “big data literacy”.

Data Science is associated with new discoveries, the discovery of value from the data. It is a practice of deriving insights and developing business strategies through transformation of data in to useful information. It has been evaluated as a scientific field and research evolution in disciplines like statistics, computing science, intelligence science, and practical transformation in the domains like science, engineering, public sector, business and lifestyle. The field encompasses the larger areas of artificial intelligence, data analytics, machine learning, pattern recognition, natural language understanding, and big data manipulation. It also tackles related new scientific challenges, ranging from data capture, creation, storage, retrieval, sharing, analysis, optimization, and visualization, to integrative analysis across heterogeneous and interdependent complex resources for better decision-making, collaboration, and, ultimately, value creation. In this paper we entitled epicycles of analysis, formal modeling, from data analysis to data science, data analytics -A keystone of data science, The Big data is not a single technology but an amalgamation of old and new technologies that assistance companies gain actionable awareness. The big data is vital because it manages, store and manipulates large amount of data at the desirable speed and time. Big data addresses detached requirements, in other words the amalgamate of multiple un-associated datasets, processing of large amounts of amorphous data and harvesting of unseen information in a time-sensitive generation.
 As businesses struggle to stay up with changing market requirements, some companies are finding creative ways to use Big Data to their growing business needs and increasingly complex problems. As organizations evolve their processes and see the opportunities that Big Data can provide, they struggle to beyond traditional Business Intelligence activities, like using data to populate reports and dashboards, and move toward Data Science- driven projects that plan to answer more open-ended and sophisticated questions.
 Although some organizations are fortunate to have data scientists, most are not, because there is a growing talent gap that makes finding and hiring data scientists in a timely manner is difficult. This paper, aimed to demonstrate a close view about Data science, big data, including big data concepts like data storage, data processing, and data analysis of these technological developments, we also provide brief description about big data analytics and its characteristics , data structures, data analytics life cycle, emphasizes critical points on these issues.

Developments in information and communication technologies offer new opportunities to use and integrate scientific data that have been collected by researchers and scholars from diverse fields of inquiry. Data archives and digital repository systems are being developed to preserve current and legacy scientific data and technical information for use by others. However, capabilities are needed for data producers of various disciplines to easily and efficiently submit their data into archival systems for preservation. Analysis of digital preservation requirements has identified the requirements for services to support the submission and review of scientific data for preservation. Data submission and review processes are segmented into services, which are defined to support efficient preparation of scientific data for ingest into an archive or digital repository system. A model is proposed to inform the design of submission and workflow services for preserving interdisciplinary scientific data. Recommendations are offered for improving the design and evaluation of systems and services to prepare and preserve scientific data for new uses by interdisciplinary communities of users in the future. Improving the infrastructure that enables members of the scientific community to submit their data for archiving contributes to the scientific data stewardship and data curation capabilities needed to preserve scientific data for future generations of users.

Financial distress prediction remains a central topic in corporate finance and risk management, with extensive research devoted to improving classification accuracy through increasingly sophisticated statistical and machine learning techniques. Nevertheless, the influence of data preparation on predictive performance has received comparatively less systematic attention. This study examines how an economically grounded data-preparation process affects the predictive performance of selected statistical and machine-learning models dedicated to predicting corporate financial distress. Using the chosen financial ratios, generally accepted indicators of corporate financial stability and economic performance, financial distress models are estimated on both raw, unprocessed input data and pre-processed data involving the exclusion of economically implausible accounting values, treatment of missing observations, and class balancing. In light of the above, the study adopts a structured methodological approach to assess the predictive performance of selected classification models, namely decision tree algorithms (CART, CHAID, and C5.0), artificial neural networks (ANNs), logistic regression (LR), and linear discriminant analysis (DA), using confusion-matrix–based evaluation and a comprehensive set of evaluation measures. The results suggest that the process of input data preparation is a critical factor, significantly improving the predictive performance of financial distress prediction models across most modelling techniques employed. The most pronounced gains are observed in decision tree models. ANNs also demonstrate marked improvement after input data preparation, whereas LR benefits more moderately, and linear DA remains limited despite preprocessing. The average gain in accuracy across all six modelling techniques, calculated as the difference between pre-processed and raw performance for each method and averaged across methods, was approximately 15.6 percentage points, with specificity improving by approximately 26.9 percentage points on average, amounting to roughly half the performance variation attributable to algorithm choice, which underscores that data preparation is a primary determinant of model reliability alongside algorithm selection. A step-level detailed analysis further shows that missing value imputation is the dominant driver of improvement for tree-based models, while class balancing contributes most for ANNs and logistic regression. The findings highlight that reliable financial distress prediction depends not only on technique selection but also on the consistency and economic plausibility of the input data, underscoring the central role of structured data preparation in developing robust early-warning models.

Creating presentation slides is a critical but time-consuming task for data scientists. While researchers have proposed many AI techniques to lift data scientists’ burden on data preparation and model selection, few have targeted the presentation creation task. Based on the needs identified from a formative study, this paper presents NB2Slides, an AI system that facilitates users to compose presentations of their data science work. NB2Slides uses deep learning methods as well as example-based prompts to generate slides from computational notebooks, and take users’ input (e.g., audience background) to structure the slides. NB2Slides also provides an interactive visualization that links the slides with the notebook to help users further edit the slides. A follow-up user evaluation with 12 data scientists shows that participants believed NB2Slides can improve efficiency and reduces the complexity of creating slides. Yet, participants questioned the future of full automation and suggested a human-AI collaboration paradigm.

Obesity and its related co-morbidities are chronic diseases with significant associated healthcare cost. Although many procedures, diets and medications have been introduced to induce weight loss, Bariatric surgery has been established as the most effective therapy for morbid obesity. However, the outcome of such surgeries is unpredictable. Bariatric Surgery Information Systems (BSIS) can be used to improve the prediction of the outcome. In this paper we will present a robust rule based semantic data integration method (RxSem) for solving the data preparation and combination problems associated with Medical Data (MD) for BSIS. RxSem enables the integration of clinical data from disparate sources by resolving canonical inconsistencies and semantic non-homogeneities. We will present ReSem using an illustrative example and share our observations regarding the difficulties and stumbling points encountered in data preparation and integration in this domain. We present solutions to some of the most pressing problems. Although we concentrate on BSIS, our observations and solutions are general and apply to a vast array of health and medical information systems.

In the fourth industrial revolution, various inter-mediation such as theories, techniques, or implementation would be used. It contains computational intelligence, applied soft computing and fuzzy logic and artificial neural networks, intelligent contents security, model driven architecture and meta-modeling, multimedia contents processing and retrieval, vehicular N/W, big data, intelligence information processing, convergence/complex contents, smart learning, intelligent contents design management, methodology and design theory, intelligent media contents convergence/complex media, social media and collective intelligence, and social media big data analytic. To consider these interesting and significant issues which have related to the fast developing area of smart media and application, the editors have edited this special issue Smart Media and Application, and finally, it selected eleven papers. In Motion data acquisition method for motion analysis in golf by Hwang et al,1 it studied the process a system that obtains the same motion as the actual one by acquiring information on the motion of the subject using 15 inertial sensors, and using the information on the actual joint length and initial direction of the subject extracted from a depth camera. As a result, it shows that measurement error of information on the joint length and the foot stance of subjects extracted through the depth camera ranges from a minimum of 4.4% to a maximum of 6.94%. In NSCT domain–based secure multiple watermarking technique through lightweight encryption for medical images, which is written by Thakur et al,2 it defined non-subsampled contourlet transform (NSCT) domain, in which the proposed scheme first partitions the host image into sub-components and then calculates the entropy values. In Environmental monitoring system for intelligent stations by Li et al,3 it proposed the design of the high-speed rail station environmental monitoring system based on the LoRa communication technology, and it aims to enable the real-time monitoring of temperature, humidity, illuminance, and noise decibels in high-speed railway stations. Efficient dummy generation for considering obstacles and protecting user location is by Song et al.4 This paper was to study an efficient dummy creating techniques to improve user privacy protection, and as a result, it showed that it improved on other recent techniques. Key node selection based on a genetic algorithm for fast patching in social networks by Kim et al5 provided considerable amounts of personal information and shares this information with friends without space-time limitations. It tried to improve the patch propagation speed; it was important to select key nodes that are the starting points of the patch process. In this paper, the authors proposed a key node selection scheme based on a genetic algorithm to find the most significant contribution nodes of the patch propagation. Following the simulation result, it showed the proposed scheme propagates patches more rapidly than the existing one. In Effective computer-assisted pronunciation training based on phone-sensitive word recommendation by Jo et al,6 they proposed a computer-assisted pronunciation training system targeting unacceptable pronunciation due to confusion among contextual allophones, a problem that often emerges from phoneme-based feedback provided during pronunciation training. Finally, it showed that the proposed system results in the improvement of pronunciation skills through training sessions that use recommended words containing phoneme pairs that were initially pronounced incorrectly. In Dataset retrieval system based on automation of data preparation with dataset description model by Mun et al,7 it tried to overcome the problem by proposing a dataset description model that can express the requirements for data processing and dataset retrieval system based on automated data preparation and it could possibly provide good quality datasets for statistical learning applications using data preparation methods such as data acquisition, refinement, and organization. In An enhanced 3DCNN-ConvLSTM for spatiotemporal multimedia data analysis by Wang et al,8 it used 3DCNN in CNN part and ConvLSTM in RNN part. In A web-based group decision support system for multicriteria problems written by Conceição et al,9 it used a multiagent system to combine and process this information, using virtual agents that represent each decision-marker. The high level of usability that the system provides will contribute to an easier acceptance and adoption of this kind of systems. In Intelligent and semantic threshold schemes for security in cloud computing by Ogiela and Snasel,10 it tried to enhanced with new classes of solutions, including semantic protocols of secret sharing, and it is subjected to the process of splitting and hiding that could take various forms including that of services. In Estimation of a physical activity energy expenditure with a patch-type sensor module using artificial neural network by Kang et al,11 it proposed the most accurate method using a wireless patch-type sensor to predict the energy expenditure of physical activities. Through the optimization of the prediction of energy expenditure of physical activities using the neural network algorithm, it achieved RMSE of 0.1893, R2 of 0.91 for the energy expenditures of aerobic and anaerobic exercises. We would like to thank you our sincere appreciation of the valuable contributions to all authors.

As new resolution enhancement techniques (RETs) are adopted to progress beyond 100 nm, the costs and benefits to the overall system of data handling must be viewed as an overall system. A 'state' approach to viewing flows has been very useful for determining insertion points for several RETs. Data preparation is a necessary part of the transformation from layout to mask. The data manipulations of required for RET also fall in this space between layout and mask. When separate point tools are used to carry out these different manipulations, increasingly large amounts of time are required to simply move data from one point tool to the other. As data file sizes increase as expected over the next decade, this problem will increase by orders of magnitude. None of this time is providing any additional productivity for data preparation. The result has been pressure to provide an integrated solution, in which OPC, phase assignment, and data preparation all share a single database. This delays flattening the data until the last necessary moment, presumably in the mask writer itself.

The accuracy of the pressure and rate data utilized for analysis directly affects the interpretation’s results. In well test interpretation, data preparation is essential and usually takes longer than analyzing well pressure responses. The common issues that arise while arranging data for analysis are addressed in detail in the chapter, along with data verification and validation.

Electronic health records (EHRs) have been successfully used in data science and machine learning projects. However, most of these data are collected for clinical use rather than for retrospective analysis. This means that researchers typically face many different issues when attempting to access and prepare the data for secondary use. We aimed to investigate how raw EHRs can be accessed and prepared in retrospective data science projects in a disciplined, effective, and efficient way. We report our experience and findings from a large-scale data science project analyzing routinely acquired retrospective data from the Kepler University Hospital in Linz, Austria. The project involved data collection from more than 150,000 patients over a period of 10 years. It included diverse data modalities, such as static demographic data, irregularly acquired laboratory test results, regularly sampled vital signs, and high-frequency physiological waveform signals. Raw medical data can be corrupted in many unexpected ways that demand thorough manual inspection and highly individualized data cleaning solutions. We present a general data preparation workflow, which was shaped in the course of our project and consists of the following 7 steps: obtain a rough overview of the available EHR data, define clinically meaningful labels for supervised learning, extract relevant data from the hospital’s data warehouses, match data extracted from different sources, deidentify them, detect errors and inconsistencies therein through a careful exploratory analysis, and implement a suitable data processing pipeline in actual code. Only few of the data preparation issues encountered in our project were addressed by generic medical data preprocessing tools that have been proposed recently. Instead, highly individualized solutions for the specific data used in one’s own research seem inevitable. We believe that the proposed workflow can serve as a guidance for practitioners, helping them to identify and address potential problems early and avoid some common pitfalls.

IntroductionModern exoplanetary research increasingly requires combining expertise from astronomy, data science, and research software engineering. The complexity and volume of data from current and future missions demand not only robust processing techniques, but also transparent and reproducible analysis frameworks. In this context, we present a modular, open-source software framework developed to support systematic comparison of different data processing and modelling methods. Its architecture allows for flexibility, extensibility, and collaboration across disciplines.The primary aim of this tool is to improve the reliability and interpretability of data-intensive exoplanet analysis workflows. By enabling the side-by-side application of multiple methodologies, researchers can better understand how specific assumptions and preprocessing decisions affect the outcomes of scientific inference.MethodThe framework allows researchers to create interchangeable modules covering stages such as data preparation, detrending, modelling, and parameter estimation. Each module is designed to operate independently within a defined interface, allowing researchers to substitute components and evaluate their effects on final results.This design supports collaborative development and integration of methods by contributors with varied technical backgrounds. It facilitates comparative workflows and improves reproducibility by preserving the full sequence of processing steps used in each analysis.As a proof of concept, the framework has been applied to the analysis of the long-period exoplanet candidate TOI-4409 b. This target was observed across 21 transits using a combination of ground-based and space-based photometric instruments, including TESS, ASTEP, CHAT, OMES, LCOGT, and PEST. The tool was used to integrate the datasets and test multiple modelling approaches in order to refine the planet’s properties. Additionally, the framework is being used in the analysis of archival TESS survey data to explore the effects of different noise-processing strategies on large-scale population-level studies.ResultsThe application of the framework to TOI-4409 b demonstrated its capacity to manage multi-instrument datasets and compare model outputs under controlled conditions. The combined light curve analysis yielded updated estimates of the planet’s physical and orbital parameters, building on previos findings.In parallel, its use in TESS archive analysis highlights the framework’s scalability and utility in benchmarking data-processing techniques across large datasets, allowing users to evaluate the impact of algorithmic variation on statistical outcomes.ConclusionThis modular framework contributes to current efforts in research software engineering by providing a reproducible and extensible platform for astronomical data analysis. Its structure supports interdisciplinary collaboration by enabling clear communication and integration of alternative methodologies.By presenting this tool within the EPSC-DPS community, particularly in the context of cross-disciplinary planetary science collaboration, we aim to introduce it as a useful addition to researchers' analytical toolkits, and to gather practical feedback from a diverse range of users to guide its development and ensure it aligns with real scientific needs.