XAutoML: A Visual Analytics Tool for Understanding and Validating Automated Machine Learning

In recent years, a wide variety of automated machine learning (AutoML) methods have been proposed to generate end-to-end ML pipelines. While these techniques facilitate the creation of models, given their black-box nature, the complexity of the underlying algorithms, and the large number of pipelines they derive, they are difficult for developers to debug. It is also challenging for machine learning experts to select an AutoML system that is well suited for a given problem. In this paper, we present the Pipeline Profiler, an interactive visualization tool that allows the exploration and comparison of the solution space of machine learning (ML) pipelines produced by AutoML systems. PipelineProfiler is integrated with Jupyter Notebook and can be combined with common data science tools to enable a rich set of analyses of the ML pipelines, providing users a better understanding of the algorithms that generated them as well as insights into how they can be improved. We demonstrate the utility of our tool through use cases where PipelineProfiler is used to better understand and improve a real-world AutoML system. Furthermore, we validate our approach by presenting a detailed analysis of a think-aloud experiment with six data scientists who develop and evaluate AutoML tools.

Currently, there is no therapy targeting septic cardiomyopathy (SC), a key contributor to organ dysfunction in sepsis. In this study, we used a machine learning (ML) pipeline to explore transcriptomic, proteomic, and metabolomic data from patients with septic shock, and prospectively collected measurements of high-sensitive cardiac troponin and echocardiography. The purposes of the study were to suggest an exploratory methodology to identify and characterise the multiOMICs profile of (i) myocardial injury in patients with septic shock, and of (ii) cardiac dysfunction in patients with myocardial injury. The study included 27 adult patients admitted for septic shock. Peripheral blood samples for OMICS analysis and measurements of high-sensitive cardiac troponin T (hscTnT) were collected at two time points during the ICU stay. A ML-based study was designed and implemented to untangle the relations among the OMICS domains and the aforesaid biomarkers. The resulting ML pipeline consisted of two main experimental phases: recursive feature selection (FS) assessing the stability of biomarkers, and classification to characterise the multiOMICS profile of the target biomarkers. The application of a ML pipeline to circulate OMICS data in patients with septic shock has the potential to predict the risk of myocardial injury and the risk of cardiac dysfunction.

In this paper, we detail how damage mechanisms commonly found on marine terminal structures and process equipment can be automatically detected using a machine learning (ML) pipeline. Defects in its varied forms, such as corrosion, steel, wood and concrete damages must be closely monitored in offshore oil and gas platforms and also in marine terminals; the cost of not monitoring and detecting defects can range from unplanned shutdowns, plant damage, to human fatalities. On-site inspection of defects is labor intensive and require frequent travel to offshore and onshore facilities. This process has two shortcomings: firstly, it is costly involving man hours spent on the facility and secondly only a subset of the total defects is monitored during each inspection due to difficulty in obtaining full coverage; specially in this scenario a high-risk defect could be missed completely. We use marine terminals as an example of how defects can be detected using machine learning techniques, however the pipeline for generating the ML models could be extended to other types of defects and structures. Significantly, the output from our ML pipeline can be used as an input to the Risk Based Inspection (RBI) system assisting asset operators in their inspection and maintenance planning. Furthermore, digitized inspections and ML can provide around 95% coverage of the terminal while avoiding problems faced by a human inspector such as producing consistent results across inspections and sites. Our system captured terrestrial spherical imagery and range data (LIDAR), subsea imagery, sonar, and ultrasonic thickness (UT) gauge data using different sensors and equipment on a marine terminal. Equipment used in our data-capture included an unmanned aerial vehicle (UAV), a terrestrial scanner, a remotely operated vehicle for underwater imaging (ROV) and a remotely operated surface vehicle (ROSV). For the purposes of this work, we used the images captured by the terrestrial scanner. We built an automated defect detection pipeline using state of the art machine learning and artificial intelligence (AI) tools taking the cube face images as inputs. The defect detection pipeline was trained with examples of defects labeled by asset integrity engineers to ensure quality of results. We built the pipeline to detect damage mechanisms for steel and wooden structures, however our system could be scaled to incorporate other types of defects and structures. Our AI pipeline was able to achieve a high level of accuracy identifying wood damage with a mean-Average Precision (m-AP) of 0.80 and steel damage with a m-AP of 0.84, where a m-AP of 1.0 indicates perfect detection of damage.

BackgroundMagnetic resonance imaging (MRI) can be used to monitor disease progression in Alzheimer's disease (AD). Portable, low‐field MRI (LF) facilitates point‐of‐care assessment and improves patient‐centered access. We developed a machine learning (ML) pipeline to use in conjunction with LF‐MRI to quantify brain morphometry and enable monitoring of AD patients in the outpatient clinic.MethodPatients with mild cognitive impairment (MCI) or dementia due to Alzheimer's disease (AD) were enrolled from the outpatient Massachusetts General Hospital (MGH) Memory Disorders Unit. A separate cohort of subjects with vascular comorbidities (VC) of comparable age were recruited from the Yale New Haven Hospital. LF‐MRI was acquired on a 0.064 T MRI (Hyperfine Inc), with conventional high‐field MRI (HF‐MRI; 1.5‐3 T) scans obtained within 7±11 months of LF‐MRI. T2 FLAIR sequences from HF and LF counterparts were processed through FreeSurfer‐based pipeline WMH‐SynthSeg for quantification of the following regions: whole brain, white matter, cortex, hippocampus, amygdala, putamen, pallidum, caudate, thalamus, accumbens, cerebellum, and ventricles.ResultThe study population included MCI (N = 31, 72±7 years), AD (N = 24, 72±9 years), and VC (N = 22, 64±8 years). Correlations between HF and LF derived brain volumes was high across all brain regions (all p <0.05). The highest correlations were observed in the cortex and white matter (r=0.91; 95% CI 0.86, 0.94), lateral ventricle (r=0.98; 95% CI 0.97, 0.99), 3rd ventricle (r=0.95; 95% CI 0.92, 0.97), hippocampus (r=0.80; 95% CI 0.70, 0.87), caudate (r=0.82; 95% CI 0.73, 0.88), and amygdala (r=0.89; 95% CI 0.83, 0.93). The MCI and AD cohorts showed regional atrophy relative to the VC cohort (all regions p <0.05) except for the caudate and 4th ventricle (p >0.05). Regions with the most significant atrophy included the cortex, hippocampus, and amygdala (all p <0.001).ConclusionLF‐MRI acquisition at the point‐of‐care for MCI or AD patients is feasible, and application of ML algorithms can generate brain volumes comparable to those derived from conventional counterparts, allowing for differentiation between VC and AD/MCI subgroups. Our findings demonstrate that LF‐MRI could be used as an accessible neuroimaging option for disease monitoring among those with cognitive impairment.

With the increasing availability of large amounts of data in the livestock domain, we face the challenge to store, combine and analyse these data efficiently. With this study, we explored the use of a data lake for storing and analysing data to improve scalability and interoperability. Data originated from a 2-day animal experiment in which the gait score of approximately 200 turkeys was determined through visual inspection by an expert. Additionally, inertial measurement units (IMUs), a 3D-video camera and a force plate (FP) were installed to explore the effectiveness of these sensors in automating the visual gait scoring. We deployed a data lake using the IMU and FP data of a single day of that animal experiment. This encompasses data from 84 turkeys for which we preprocessed by performing an ‘extract, transform and load’ (ETL-) procedure. To test scalability of the ETL-procedure, we simulated increasing volumes of the available data from this animal experiment and computed the ‘wall time’ (elapsed real time) for converting FP data into comma-separated files and storing these files. With a simulated data set of 30 000 turkeys, the wall time reduced from 1 h to less than 15 min, when 12 cores were used compared to 1 core. This demonstrated the ETL-procedure to be scalable. Subsequently, a machine learning (ML) pipeline was developed to test the potential of a data lake to automatically distinguish between two classses, that is, very bad gait scores v. other scores. In conclusion, we have set up a dedicated customized data lake, loaded data and developed a prediction model via the creation of an ML pipeline. A data lake appears to be a useful tool to face the challenge of storing, combining and analysing increasing volumes of data of varying nature in an effective manner.

BackgroundPsychosis poses substantial social and healthcare burdens. The analysis of speech is a promising approach for the diagnosis and monitoring of psychosis, capturing symptoms like thought disorder and flattened affect. Recent advancements in Natural Language Processing (NLP) methodologies enable the automated extraction of informative speech features, which has been leveraged for early psychosis detection and assessment of symptomology. However, critical gaps persist, including the absence of standardized sample collection protocols, small sample sizes, and a lack of multi-illness classification, limiting clinical applicability. Our study aimed to (1) identify an optimal assessment approach for the online and remote collection of speech, in the context of assessing the psychosis spectrum and evaluate whether a fully automated, speech-based machine learning (ML) pipeline can discriminate among different conditions on the schizophrenia-bipolar spectrum (SSD-BD-SPE), help-seeking comparison subjects (MDD), and healthy controls (HC) at varying layers of analysis and diagnostic complexity.Methods:We adopted online data collection methods to collect 20 minutes of speech and demographic information from individuals. Participants were categorized as “healthy” help-seekers (HC), having a schizophrenia-spectrum disorder (SSD), bipolar disorder (BD), major depressive disorder (MDD), or being on the psychosis spectrum with sub-clinical psychotic experiences (SPE). SPE status was determined based on self-reported clinical diagnosis and responses to the PHQ-8 and PQ-16 screening questionnaires, while other diagnoses were determined based on self-report from participants. Linguistic and paralinguistic features were extracted and ensemble learning algorithms (e.g., XGBoost) were used to train models. A 70%-30% train-test split and 30-fold cross-validation was used to validate the model performance.Results:The final analysis sample included 1140 individuals and 22,650 minutes of speech. Using 5-minutes of speech, our model could discriminate between HC and those with a serious mental illness (SSD or BD) with 86% accuracy (AUC = 0.91, Recall = 0.7, Precision = 0.98). Furthermore, our model could discern among HC, SPE, BD and SSD groups with 86% accuracy (F1 macro = 0.855, Recall Macro = 0.86, Precision Macro = 0.86). Finally, in a 5-class discrimination task including individuals with MDD, our model had 76% accuracy (F1 macro = 0.757, Recall Macro = 0.758, Precision Macro = 0.766).Conclusion:Our ML pipeline demonstrated disorder-specific learning, achieving excellent or good accuracy across several classification tasks. We demonstrated that the screening of mental disorders is possible via a fully automated, remote speech assessment pipeline. We tested our model on relatively high number conditions (5 classes) in the literature and in a stratified sample of psychosis spectrum, including HC, SPE, SSD and BD (4 classes). We tested our model on a large sample (N = 1150) and demonstrated best-in-class accuracy with remotely collected speech data in the psychosis spectrum, however, further clinical validation is needed to test the reliability of model performance.

Recent technological advances allowed an exponential expansion of biological sequence data, and the extraction of meaningful information through Machine Learning (ML) algorithms. This knowledge improved the understanding of the mechanisms related to several fatal diseases, e.g., Cancer and COVID-19, helping to develop innovative solutions, such as CRISPR-based gene editing, coronavirus vaccine, and precision medicine. These advances benefit our society and economy, directly impacting peoples lives in various areas, such as health care, drug discovery, forensic analysis, and food analysis. Nevertheless, ML approaches applied to biological data require representative, quantitative, and informative features. Necessarily, as many ML algorithms can handle only numerical data, sequences need to be translated into a feature vector. This process, known as feature extraction, is a fundamental step for the elaboration of high-quality ML-based models in bioinformatics, by allowing the feature engineering stage, with the design and selection of suitable features. Feature engineering, ML algorithm selection, and hyperparameter tuning are often manual and time-consuming processes, requiring extensive domain knowledge, and performed manually by a human expert. To deal with this problem, we developed a new package, BioAutoML, which automatically runs an end-to-end ML pipeline. BioAutoML extracts numerical and informative features from biological sequence databases, automating feature selection, recommendation of ML algorithm(s), and tuning of hyperparameters, using Automated ML (AutoML). BioAutoML has two components, divided into four modules, (1) automated feature engineering (feature extraction and selection modules) and (2) Metalearning (algorithm recommendation and hyperparameter tuning modules). Our experimental results, assessing the relevance of our proposal, indicate robust results for different problem domains, such as SARS-CoV-2, anticancer peptides, HIV sequences, and non-coding RNAs. According to our systematic review, our proposal is innovative compared to available studies in the literature, being the first study to propose automated feature engineering and metalearning for biological sequences. BioAutoML has a high potential to significantly reduce the expertise required to use ML pipelines, aiding researchers in combating diseases, particularly in low- and middle-income countries. This initiative can provide biologists, physicians, epidemiologists, and other stakeholders with an opportunity for widespread use of these techniques to enhance the health and well-being of their communities.

On taking advantage of opportunistic meta-knowledge to reduce configuration spaces for automated machine learning

The emergence of the mobile applications that are driven by machine learning (ML) has brought a breakthrough in the world of healthcare, e-commerce, and entertainment with its personalized and smart user experiences. Despite the wide application of ML pipelines in mobile environments, the security and privacy issues that could be raised have become an issue of high concern on a global scale. The storage and processing of data, combined with the deployment of advanced ML models, expose vulnerabilities to risks such as data leakage, adversarial attacks, and unauthorized access. These threats not only compromise user trust but also incur applications with financial and brand damage. The present article deals with the important problems of the safety of mobile ML pipelines, namely adversarial manipulations, insecure storage, and transmitting of data. The vital techniques that can help to work out those problems are described along with the most popular ones, namely secure model storage based on encryption, federated learning for data transfer reduction, and model robustness tactics that will increase adversarial attacks defense. Besides that, new technologies like homomorphic encryption and blockchain are also discussed as ways to secure model updates. Consequently, this paper is involved in the technical and practical levels of the project, and it is meant to show the importance of building secure and privacy-preserving ML pipelines that do not only keep data for user trust but also maintain efficiency. It serves the purpose of opening the way to moving to secure mobile ML solutions in the time of intelligent mobile systems that have started to be of great interest.

In recent years, Machine Learning (ML) and Automated Machine Learning (Auto-ML) have attracted significant attention. The ML pipeline involves repetitive tasks such as data preprocessing, feature engineering, model selection, and hyperparameter tuning. Developing a machine learning model demands considerable time for development, stress testing, and numerous experiments. Additionally, constructing a model with a limited search space of pipeline steps and various algorithms can take hours. As a result, Auto-ML has become widely adopted to reduce the time and effort required for these tasks. However, most current Auto-ML frameworks primarily concentrate on algorithm selection and hyperparameter optimization, known as CASH, while overlooking other critical ML pipeline steps like data preprocessing and feature engineering. This limited focus often results in suboptimal pipelines for specific datasets. Moreover, a significant number of frameworks overlook the integration of meta-learning, resulting in the promotion of high-performing pipelines customized for individual tasks rather than a universally optimal solution. Consequently, this deficiency necessitates the quest for a new pipeline tailored to each unique task, further underscoring the importance of a more comprehensive approach in Auto-ML frameworks. Additionally, while some Auto-ML frameworks address the entire pipeline, they often overlook the challenges posed by imbalanced datasets. To address these issues, we propose a novel and efficient meta-learning Auto-ML framework that effectively manages imbalanced datasets. The proposed framework outperforms state-of-the-art results in terms of accuracy, precision, recall, and time, demonstrating, on average, more than 5% improvement compared to the existing auto-ML frameworks. This paper also illustrates how our proposed framework outperforms current state-of-the-art solutions.

A comparative study of machine learning methods for predicting the evolution of brain connectivity from a baseline timepoint

In recent years, many incidents have been reported where machine learning models exhibited discrimination among people based on race, sex, age, etc. Research has been conducted to measure and mitigate unfairness in machine learning models. For a machine learning task, it is a common practice to build a pipeline that includes an ordered set of data preprocessing stages followed by a classifier. However, most of the research on fairness has considered a single classifier based prediction task. What are the fairness impacts of the preprocessing stages in machine learning pipeline? Furthermore, studies showed that often the root cause of unfairness is ingrained in the data itself, rather than the model. But no research has been conducted to measure the unfairness caused by a specific transformation made in the data preprocessing stage. In this paper, we introduced the causal method of fairness to reason about the fairness impact of data preprocessing stages in ML pipeline. We leveraged existing metrics to define the fairness measures of the stages. Then we conducted a detailed fairness evaluation of the preprocessing stages in 37 pipelines collected from three different sources. Our results show that certain data transformers are causing the model to exhibit unfairness. We identified a number of fairness patterns in several categories of data transformers. Finally, we showed how the local fairness of a preprocessing stage composes in the global fairness of the pipeline. We used the fairness composition to choose appropriate downstream transformer that mitigates unfairness in the machine learning pipeline.

Performance of Automatic Machine Learning versus Radiologists in the Evaluation of Endometrium on Computed Tomography

Machine Learning (ML) is increasingly being used as an essential component of modern software systems. Also, the maturity of the adopted techniques and the availability of frameworks have changed the way developers approach ML-related development problems. This paper aims at investigating, by analyzing Stack Overflow (SO) posts related to ML, how the questions about ML have been changing over the years, and across six different programming languages. We analyzed 43,950 SO posts in the period 2008-2020, studying (i) how the number of ML-related posts changes over time for each programming language, (ii) how the posts are distributed across different phases of a ML pipeline, and (iii) whether posts belonging to different languages or phases are more or less challenging to address. We found that some programming languages are fading while others are becoming more popular in ML development. While model-building questions are the most discussed in general, the level of challenges posed by the other phases of the ML pipeline appears to be language-dependent. Results of this work could be used to better understand ML challenges in different programming languages, and, possibly, to improve ML tutorials related to different languages.

Part defects in additive manufacturing are more frequent compared to machining or molding. Failures can go unnoticed for hours, wasting resources and extending process cycle times. This paper describes a Machine Learning based method for automated sensing of onset failure in additive manufacturing machinery. Investigations are conducted on a Fused Filament Fabrication (FFF) 3D printer, and the same methods are then applied to a digital light processing 3D printer. The investigation focuses on signal-based analysis, specifically passive sensing of stepper motors relating DC current measurements to the torque on a stepper, as opposed to any active acoustic interrogation of the part. Passive methods are used to characterize the loading on a feeder stepper in an FFF machine, forming a model that can identify early signs of filament-based failure with 85.65% 10-fold cross-validation accuracy. Efforts show filament breakage can be detected minutes before material runout would cause a defect, allowing ample time to pause, correct, or control the print. The machine learning pipeline was not naively conceived but optimized through automated machine learning.