VeridicalFlow: a Python package for building trustworthy data science pipelines with PCS

Increasingly larger number of software systems today are including data science components for descriptive, predictive, and prescriptive analytics. The collection of data science stages from acquisition, to cleaning/curation, to modeling, and so on are referred to as data science pipelines. To facilitate research and practice on data science pipelines, it is essential to understand their nature. What are the typical stages of a data science pipeline? How are they connected? Do the pipelines differ in the theoretical representations and that in the practice? Today we do not fully understand these architectural characteristics of data science pipelines. In this work, we present a three-pronged comprehensive study to answer this for the state-of-the-art, data science in-the-small, and data science in-the-large. Our study analyzes three datasets: a collection of 71 proposals for data science pipelines and related concepts in theory, a collection of over 105 implementations of curated data science pipelines from Kaggle competitions to understand data science in-the-small, and a collection of 21 mature data science projects from GitHub to understand data science in-the-large. Our study has led to three representations of data science pipelines that capture the essence of our subjects in theory, in-the-small, and in-the-large.

Data science has become prevalent in a large variety of domains. Inherent in its practice is an exploratory, probing, and fact finding journey, which consists of the assembly, adaptation, and execution of complex data science pipelines. The trustworthiness of the results of such pipelines rests entirely on their ability to be reproduced with fidelity, which is difficult if pipelines are not documented or recorded minutely and consistently. This difficulty has led to a reproducibility crisis and presents a major obstacle to the safe adoption of the pipeline results in production environments. The crisis can be resolved if the provenance for each data science pipeline is captured transparently as pipelines are executed. However, due to the complexity of modern data science pipelines, transparently capturing sufficient provenance to allow for reproducibility is challenging. As a result, most existing systems require users to augment their code or use specific tools to capture provenance, which hinders productivity and results in a lack of adoption. In this paper, we present Ursprung, 1 a transparent provenance collection system designed for data science environments. 2 The Ursprung philosophy is to capture provenance and build lineage by integrating with the execution environment to automatically track static and runtime configuration parameters of data science pipelines. Rather than requiring data scientists to make changes to their code, Ursprung records basic provenance information from system-level sources and combines it with provenance from application-level sources (e.g., log files, stdout), which can be accessed and recorded through a domain-specific language. In our evaluation, we show that Ursprung is able to capture sufficient provenance for a variety of use cases and only adds an overhead of up to 4%.

Since 1989, the first year of the democratic presidential election after a long period of a dictatorship regime, Brazil conducted eight presidential elections. This period was marked by short and long-term shifts of power and two impeachment processes. Such instability is a case of study in electoral studies, e.g., the study of the population voting behavior. Understanding patterns in the population behavior can give us insight into factors and influences that affect the quality of democratic political decisions. In light of this, our paper focuses on analyzing the Brazilian presidential election voting behavior across the years and the Brazilian territory. Following a data science pipeline, we divided the analysis process into five steps: (i) data selection; (ii) data preprocessing; (iii) identification of spatial patterns, in which we seek to understand the role of space in the election results using spatial autocorrelation techniques; (iv) identification of temporal patterns, where we investigate similar trends of votes over the years using a hierarchical clustering method; and (v) evaluation of the results. It is noteworthy that the data in this work represents the election results at the municipal level, from 1994 to 2018, of the two most relevant parties of this period: the Brazilian Social Democracy Party (PSDB) and the Workers’ Party (PT). Through the results obtained, we found the existence of spatial dependence in every electoral year investigated. Moreover, despite the changes in the political-economic context over the years, neighboring cities seem to present similar voting behavior trends.

Background and objective: Despite fast evolution cycles in deep learning methodologies for medical imaging in radiotherapy, auto-segmentation solutions rarely run in clinics due to the lack of open-source frameworks feasible for processing DICOM RT Structure Sets. Besides this shortage, available open-source DICOM RT Structure Set converters rely exclusively on 2D reconstruction approaches leading to pixelated contours with potentially low acceptance by healthcare professionals. PyRaDiSe, an open-source, deep learning framework independent Python package, addresses these issues by providing a framework for building auto-segmentation solutions feasible to operate directly on DICOM data. In addition, PyRaDiSe provides profound DICOM RT Structure Set conversion and processing capabilities; thus, it applies also to auto-segmentation-related tasks, such as dataset construction for deep learning model training.Methods: The PyRaDiSe package follows a holistic approach and provides DICOM data handling, deep learning model inference, pre-processing, and post-processing functionalities. The DICOM data handling allows for highly automated and flexible handling of DICOM image series, DICOM RT Structure Sets, and DICOM registrations, including 2D-based and 3D-based conversion from and to DICOM RT Structure Sets. For deep learning model inference, extending given skeleton classes is straightforwardly achieved, allowing for employing any deep learning framework. Furthermore, a profound set of pre-processing and post-processing routines is included that incorporate partial invertibility for restoring spatial properties, such as image origin or orientation.Results: The PyRaDiSe package, characterized by its flexibility and automated routines, allows for fast deployment and prototyping, reducing efforts for auto-segmentation pipeline implementation. Furthermore, while deep learning model inference is independent of the deep learning framework, it can easily be integrated into famous deep learning frameworks such as PyTorch or Tensorflow. The developed package has successfully demonstrated its capabilities in a research project at our institution for organs-at-risk segmentation in brain tumor patients. Furthermore, PyRaDiSe has shown its conversion performance for dataset construction.Conclusions: The PyRaDiSe package closes the gap between data science and clinical radiotherapy by enabling deep learning segmentation models to be easily transferred into clinical research practice. PyRaDiSe is available on https://github.com/ubern-mia/pyradise and can be installed directly from the Python Package Index using pip install pyradise.

Background: Determining a patient’s response to a specific therapy is a vital step in developing personalized cancer treatment. Personalized treatment relies on two key technological advancements: algorithms tailored to each patient’s molecular profiles, and comprehensive datasets detailing the effects of patient diversity on drug response. Despite recent advancements in machine learning-based models, each new algorithm requires curation of numerous datasets, which poses a significant burden to researchers who need to evaluate and compare various algorithms. Currently, datasets include extensive genomics, transcriptomics, proteomics and metabolomics measurements along with drug response data (i.e., cell viability) for each cell line, collected through the Cancer Cell Line Encyclopedia (CCLE) and other initiatives. Additional omics data is available for tumor organoids and patient samples generated by consortia such as Human Cancer Model Initiative (HCMI) and the Clinical Proteomic Tumor Analysis Consortium (CPTAC). However, to date there has been little public effort to annotate and harmonize these data across repositories. Methods: To address this need, we developed the Python package coderdata that collates the most up-to-date cancer omics and drug response data. We curated four public cancer datasets, establishing reproducible pipelines that standardize data retrieval, formatting, and integration so the datasets can be built on a cluster if needed. The datasets undergo bi-monthly synchronization with FigShare, ensuring the data is up to date and available. Scientists can use coderdata to retrieve and format data on the command line interface as well as within Python scripts. Results: Coderdata is a straightforward Python package that will enable the development and benchmarking of diverse machine learning applications in cancer research. At its core, it represents reproducible data harmonization that enables heterogeneous datasets to be analyzed in bulk. This Python package downloads and standardizes data from multiple consortia-related resources including HCMI, CPTAC, CCLE, and BeatAML, together representing 4931 multi-omics samples from across 250+ cancer types. These datasets include cancer data metrics such as copy number, mutation, transcriptomics, proteomics, miRNA, methylation, and drug response data. Conclusions: The applications of machine learning in cancer biology are hampered by the distributed nature of existing datasets. As such, the collection and standardization of data by coderdata substantially reduces the time investment required for data curation in cancer research. Direct access to benchmark multi-omics and drug response datasets enables scientists to focus on algorithm development for tumor drug response prediction, potentially accelerating the discovery of therapeutic strategies. Citation Format: Jeremy Jacobson, Sydney Schwartz, M. Ryan Weil, Neeraj Kumar, Sara Gosline. The cancer omics and drug experimental response dataset (CODERData): A harmonized benchmark dataset for machine learning models of drug response prediction [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 6210.

<p>As machine learning algorithms are being used more and more prominently in the meteorology and climate domains, the need for reference datasets has been identified as a priority. Moreover, boilerplate code for data handling is ubiquitous in scientific experiments. In order to focus on science, climate/meteorology/data scientists need generic and reusable domain-specific tools. To achieve these goals, we used the plugin based CliMetLab python package along with many packages listed by Pangeo.  </p><p><br>Our use case consists in providing data for machine learning algorithms in the context of the sub-seasonal to seasonal (S2S) prediction challenge 2021. The data size is about 2 Terabytes of model predictions from three different models. We experimented with providing data in multiple formats: Grib, NetCDF, and Zarr. A Pangeo recipe (using the python package pangeo_forge_recipes) was used to generate Zarr data (relying heavily on xarray and dask for parallelisation). All three versions of the S2S data have been stored on an S3 bucket located on the ECMWF European Weather Cloud (ECMWF-EWC). </p><p><br>CliMetLab aims at providing a simple interface to access climate and meteorological datasets, seamlessly downloading and caching data, converting to xarray datasets or panda dataframes, plotting data, feed them into machine learning frameworks such as tensorflow or pytorch. CliMetLab is open-source and still a Beta version (https://climetlab.readthedocs.io). The main target platform of CliMetLab is Jupyter notebooks. Additionally, a CliMetLab plugin allows shipping dataset-specific code along with a well-defined published dataset. Taking advantage of the CliMetLab tools to minimize the boilerplate code, a plugin has been developed for S2S data as a companion python package of the dataset.</p>

This paper presents the Multilingual COVID-19 Analysis Method (CMTA) for detecting and observing the spread of misinformation about this disease within texts. CMTA proposes a data science (DS) pipeline that applies machine learning models for processing, classifying (Dense-CNN) and analyzing (MBERT) multilingual (micro)-texts. DS pipeline data preparation tasks extract features from multilingual textual data and categorize it into specific information classes (i.e., ‘false’, ‘partly false’, ‘misleading’). The CMTA pipeline was experimented with multilingual micro-texts (tweets), showing misinformation spread across different languages. We performed a comparative analysis of CMTA with eight monolingual models used for detecting misinformation. The comparison shows that CMTA has surpassed various monolingual models and suggests that it can be used as a general method for detecting misinformation in multilingual micro-texts.

While glycans are crucial for biological processes, existing analysis modalities make it difficult for researchers with limited computational background to include these diverse carbohydrates into workflows. Here, we present glycowork, an open-source Python package designed for glycan-related data science and machine learning by end users. Glycowork includes functions to, for instance, automatically annotate glycan motifs and analyze their distributions via heatmaps and statistical enrichment. We also provide visualization methods, routines to interact with stored databases, trained machine learning models and learned glycan representations. We envision that glycowork can extract further insights from glycan datasets and demonstrate this with workflows that analyze glycan motifs in various biological contexts. Glycowork can be freely accessed at https://github.com/BojarLab/glycowork/.

My work developed a kernel density estimator that well resolves typical structures of probability densities, which was demonstrated on a newly compiled marine data set of organic carbon-13 isotope ratios (δ13CPOC). All work was conducted within the emerging field of marine data science. I identified classical data science, a general understanding of ocean science, communication skills, and confidence as requirements for marine data scientists. In the beginning of my work, the existing δ13CPOC data consisted of about 500 data points in the global ocean. I expanded the existing data set to 4732 data points in a first version, and additionally to 6952 in a second. Both are published at PANGAEA along with meta information such as measurement location, time, and method, and interpolations. I have published a description of the temporal and geographic distribution of the first version at Earth System Science Data. I designed the development of the kernel density estimator algorithm on the existing concept of computing it as a solution of the diffusion equation. My algorithm uses finite differences in space and equidistant time steps with an implicit Euler method, and approximates the optimal smoothing parameter by two pilot steps. Compared to other well-known kernel density estimators, my algorithm produces reliable approximations of multimodal and boundary-close distributions on artificial and real marine data and is robust to noise. My implementation is published as a Python package on Zenodo, its description is submitted to Geoscientific Model Development. I was able to show that my kernel density estimator reliably evalu- ates ocean data and thus lays a foundation for calibrating Earth system models. At the same time, I was able to contribute to the definition and establishment of the field of Marine Data Science.

MotivationThe identification of predictive biomarker signatures from omics and multi-omics data for clinical applications is an active area of research. Recent developments in assay technologies and machine learning (ML) methods have led to significant improvements in predictive performance. However, most high-performing ML methods suffer from complex architectures and lack interpretability.ResultsWe present the application of a novel symbolic-regression-based algorithm, the QLattice, on a selection of clinical omics datasets. This approach generates parsimonious high-performing models that can both predict disease outcomes and reveal putative disease mechanisms, demonstrating the importance of selecting maximally relevant and minimally redundant features in omics-based machine-learning applications. The simplicity and high-predictive power of these biomarker signatures make them attractive tools for high-stakes applications in areas such as primary care, clinical decision-making and patient stratification.Availability and implementationThe QLattice is available as part of a python package (feyn), which is available at the Python Package Index (https://pypi.org/project/feyn/) and can be installed via pip. The documentation provides guides, tutorials and the API reference (https://docs.abzu.ai/). All code and data used to generate the models and plots discussed in this work can be found in https://github.com/abzu-ai/QLattice-clinical-omics.Supplementary informationSupplementary material is available at Bioinformatics online.

On the global arena, drug and substance abuse (DSA) problem has caused unprecedented suffering. The number of the youth and adult population developing addictions has been on a steady rise, according to the United Nations Office on Drugs and Crime (UNODC) report. A new mathematical model incorporating the impact of policing and rehabilitation on proliferation of DSA in the community is designed and examined in this research. This study postulates that the propagation of DSA is comparable to that of horrendous contagious disease. The effective reproduction number ( R 0 ) of the proposed deterministic model provides the substratum for stability analysis. From the analysis, the DSA‐free equilibrium (DSAFE) is corroborated to be locally asymptotically stable when R 0 < 1 and unstable for R 0 > 1. The local stability of DSA endemic equilibrium (DSAEE) is explored by employing the centre manifold technique. The study rules out the possibility of occurrence of backward bifurcation. Using Python software, various simulation scenarios were run in order to correlate the outcomes with the analytical findings. The simulation results show that increasing the rate at which light drug abusers (LDAs) and heavy drug abusers (HDAs) are identified and placed under structured treatment and therapy, as well as proactive response by intercepting and dismantling trafficking networks via efficacious law enforcement, would adversely diminish drug abuse menace not only in Kenya but in the world at large.

The Large Synoptic Survey Telescope (LSST) will be an 8.4m optical survey telescope sited in Chile and capable of imaging the entire sky twice a week. The data rate of approximately 15TB per night and the requirements to both issue alerts on transient sources within 60 seconds of observing and create annual data releases means that automated data management systems and data processing pipelines are a key deliverable of the LSST construction project. The LSST data management software has been in development since 2004 and is based on a C++ core with a Python control layer. The software consists of nearly a quarter of a million lines of code covering the system from fundamental WCS and table libraries to pipeline environments and distributed process execution. The Astropy project began in 2011 as an attempt to bring together disparate open source Python projects and build a core standard infrastructure that can be used and built upon by the astronomy community. This project has been phenomenally successful in the years since it has begun and has grown to be the de facto standard for Python software in astronomy. Astropy brings with it considerable expectations from the community on how astronomy Python software should be developed and it is clear that by the time LSST is fully operational in the 2020s many of the prospective users of the LSST software stack will expect it to be fully interoperable with Astropy. In this paper we describe the overlap between the LSST science pipeline software and Astropy software and investigate areas where the LSST software provides new functionality. We also discuss the possibilities of re-engineering the LSST science pipeline software to build upon Astropy, including the option of contributing affliated packages.

Because of the language’s simplicity, flexibility and massive libraries supporting artificial intelligence, web development and data science python has become one of the widely used programming language. But this speedy growth has led to software security and vulnerability concerns. This paper examines the security provisions available in python programming and their role in developing secure applications. Herein the analysis is done based on security mechanisms such as validation of inputs, authentication and authorization, libraries available for cryptography, management of dependency tools, static code analysis tools and secure web frameworks. The paper highlights the common security challenges faced in python programming and emphasis on the importance of secure coding practices and comparison of vulnerability detection tools implemented in the past research works. The finding proclaims that python provides several built-in tools and mechanisms which when properly implemented can notably improve reliability and security of software systems. Keywords: Python Security, Secure Coding Practices, Vulnerability Detection, Machine Learning Based Security Analysis, Static Code Analysis, Cryptography in Python, Software Security

A Review on Reddit News Headlines with NLTK tool

Internet of Things (IoT) grows quickly; protecting private data and avoiding unwanted access have become crucial issues. Large volumes of diverse IoT traffic are frequently too much for traditional intrusion detection techniques to handle, which lowers accuracy and increases false detection rates. To tackle this problem, a novel Deep Learning (DL) system that is optimised for identifying unauthorised assaults on IoT devices. Pre-processing procedures including cleaning, normalisation, and transformation are used to refine the raw data after traffic capture and feature extraction. In order to capture intricate connections and delicate patterns in IoT traffic, a Generative Adversarial Network (GAN) is deployed that have been augmented with a self-attention mechanism. Using the Fossa Optimisation Algorithm (FOA), this guarantees effective convergence by tuning the parameters of proposed classifier to further enhance prediction performance and reliable generalisation across various IoT attack situations. The proposed method is a viable strategy for enhancing IoT network security and resilience against changing cyber threats because it shows excellent dependability in detecting unauthorised intrusions proven by the accuracy of 98% and precision and recall of 100 % validated in python software.