SML-AutoML: A Smart Meta-Learning Automated Machine Learning Framework

Recent technological advances allowed an exponential expansion of biological sequence data, and the extraction of meaningful information through Machine Learning (ML) algorithms. This knowledge improved the understanding of the mechanisms related to several fatal diseases, e.g., Cancer and COVID-19, helping to develop innovative solutions, such as CRISPR-based gene editing, coronavirus vaccine, and precision medicine. These advances benefit our society and economy, directly impacting peoples lives in various areas, such as health care, drug discovery, forensic analysis, and food analysis. Nevertheless, ML approaches applied to biological data require representative, quantitative, and informative features. Necessarily, as many ML algorithms can handle only numerical data, sequences need to be translated into a feature vector. This process, known as feature extraction, is a fundamental step for the elaboration of high-quality ML-based models in bioinformatics, by allowing the feature engineering stage, with the design and selection of suitable features. Feature engineering, ML algorithm selection, and hyperparameter tuning are often manual and time-consuming processes, requiring extensive domain knowledge, and performed manually by a human expert. To deal with this problem, we developed a new package, BioAutoML, which automatically runs an end-to-end ML pipeline. BioAutoML extracts numerical and informative features from biological sequence databases, automating feature selection, recommendation of ML algorithm(s), and tuning of hyperparameters, using Automated ML (AutoML). BioAutoML has two components, divided into four modules, (1) automated feature engineering (feature extraction and selection modules) and (2) Metalearning (algorithm recommendation and hyperparameter tuning modules). Our experimental results, assessing the relevance of our proposal, indicate robust results for different problem domains, such as SARS-CoV-2, anticancer peptides, HIV sequences, and non-coding RNAs. According to our systematic review, our proposal is innovative compared to available studies in the literature, being the first study to propose automated feature engineering and metalearning for biological sequences. BioAutoML has a high potential to significantly reduce the expertise required to use ML pipelines, aiding researchers in combating diseases, particularly in low- and middle-income countries. This initiative can provide biologists, physicians, epidemiologists, and other stakeholders with an opportunity for widespread use of these techniques to enhance the health and well-being of their communities.

In the last 10 years, various automated machine learning (AutoML) systems have been proposed to build end-to-end machine learning (ML) pipelines with minimal human interaction. Even though such automatically synthesized ML pipelines are able to achieve competitive performance, recent studies have shown that users do not trust models constructed by AutoML due to missing transparency of AutoML systems and missing explanations for the constructed ML pipelines. In a requirements analysis study with 36 domain experts, data scientists, and AutoML researchers from different professions with vastly different expertise in ML, we collect detailed informational needs for AutoML. We propose XAutoML , an interactive visual analytics tool for explaining arbitrary AutoML optimization procedures and ML pipelines constructed by AutoML. XAutoML combines interactive visualizations with established techniques from explainable artificial intelligence (XAI) to make the complete AutoML procedure transparent and explainable. By integrating XAutoML with JupyterLab , experienced users can extend the visual analytics with ad-hoc visualizations based on information extracted from XAutoML . We validate our approach in a user study with the same diverse user group from the requirements analysis. All participants were able to extract useful information from XAutoML , leading to a significantly increased understanding of ML pipelines produced by AutoML and the AutoML optimization itself.

A study on the recent trends and progress in the area of Automated Machine Learning (AutoML) is done in detail in this paper. AutoML deals with the end-to-end automation of various steps in a machine-learning pipeline. Some of the steps include feature selection, feature engineering, neural architecture search, hyperparameter optimization, and model selection. The time and the specialized skill set required to perform these tasks may be reduced to some extent with the help of automating all or some of these steps. Thus, a lot of research is going on in the area of AutoML and the recent research articles add justice to the same. A review of existing literature on AutoML with a focus on feature engineering is presented in this paper to assist scientists in building better machine learning models "off the shelf" without extensive data science experience. The use of AutoML in different sectors will also be discussed, as will existing applications of AutoML. A review of published papers, accompanied by describing work in AutoML from a computer science perspective was conducted. Received: 1 February 2023 | Revised: 22 April 2023 | Accepted: 15 May 2023 Conflicts of Interest The authors declare that they have no conflicts of interest to this work. Data Availability Statement Data sharing is not applicable to this article as no new data were created or analyzed in this study. Author Contribution Statement Ravishankar S: Methodology, Software, Validation, Formal analysis, Writing – review & editing, Visualization, Supervision, Project administration. Gopi Battineni: Conceptualization, Investigation, Resources, Data curation, Writing – original draft, Writing – review & editing, Visualization, Supervision, Project administration.

On taking advantage of opportunistic meta-knowledge to reduce configuration spaces for automated machine learning

Predicting survival outcomes in critical accidents has been a focal point in machine learning research. This study addresses several limitations of existing methods, including insufficient management of data imbalance, lack of emphasis on hyperparameter tuning, and proneness to overfitting. Many existing models struggle to generalize effectively on imbalanced datasets or depend on default hyperparameter settings, resulting in biased predictions. By integrating Principal Component Analysis (PCA), hyperparameter optimization, and resampling methods, as well as combining Edited Nearest Neighbors (ENN) with the Synthetic Minority Oversampling Technique (SMOTE), the model significantly improves predictive accuracy and model generalization. An ensemble model combining seven machine learning algorithms—Logistic Regression, Support Vector Machine, KNN, Random Forest, XGBoost, LightGBM, and CatBoost—was applied to predict survival outcomes. Stochastic Weighted Averaging (SWA) was applied to mitigate overfitting and enhance generalization. The accuracy increased from 91.97% to 94.89% after SWA was applied in this specific scenario. The combination of PCA-based dimensionality reduction, hyperparameter tuning, and resampling techniques (ENN + SMOTE) ensured the model handled data imbalance and optimized predictive accuracy. The final model demonstrated excellent performance, with Area Under the Curve (AUC) and Average Precision (AP) values both reaching 0.98, indicating high accuracy and precision. These improvements were validated using the Titanic dataset in a binary classification problem of predicting passenger survival. The results emphasize that ensemble learning, enhanced by SWA, offers a powerful framework for handling imbalanced and complex datasets, providing significant advancements in predictive modeling accuracy. This study provides insights into how machine learning techniques can be effectively combined to solve classification challenges in real-world scenarios.

In recent years, a wide variety of automated machine learning (AutoML) methods have been proposed to generate end-to-end ML pipelines. While these techniques facilitate the creation of models, given their black-box nature, the complexity of the underlying algorithms, and the large number of pipelines they derive, they are difficult for developers to debug. It is also challenging for machine learning experts to select an AutoML system that is well suited for a given problem. In this paper, we present the Pipeline Profiler, an interactive visualization tool that allows the exploration and comparison of the solution space of machine learning (ML) pipelines produced by AutoML systems. PipelineProfiler is integrated with Jupyter Notebook and can be combined with common data science tools to enable a rich set of analyses of the ML pipelines, providing users a better understanding of the algorithms that generated them as well as insights into how they can be improved. We demonstrate the utility of our tool through use cases where PipelineProfiler is used to better understand and improve a real-world AutoML system. Furthermore, we validate our approach by presenting a detailed analysis of a think-aloud experiment with six data scientists who develop and evaluate AutoML tools.

Machine learning solutions have been successfully used to solve many simple and complex problems. However, their development process still relies on human experts to perform tasks such as data preprocessing, feature engineering and model selection. As the complexity of these tasks increases, so does the demand for automated solutions, namely Automated Machine Learning (AutoML). Most algorithms employed in these systems have hyperparameters whose configuration may directly affect their predictive performance. Therefore, hyperparameter tuning is a recurring task in AutoML systems. This thesis investigated how to efficiently automate hyperparameter tuning by means of Meta-learning. To this end, large-scale experiments were performed tuning the hyperparameters of different classification algorithms, and an enhanced experimental methodology was adopted throughout the thesis to explore and learn the hyperparameter profiles for different classification algorithms. The results also showed that in many cases the default hyperparameter settings induced models that are on par with those obtained by tuning. Hence, a new Meta-learning recommender system was proposed to identify when it is better to use default values and when to tune classification algorithms for each new dataset. The proposed system is capable of generalizing several learning processes into a single modular framework, along with the possibility of assigning different algorithms. Furthermore, a descriptive analysis of model predictions is used to identify which data characteristics affect the necessity for tuning in each one of the algorithms investigated in the thesis. Experimental results also demonstrated that the proposed recommender system reduced the time spent on optimization processes, without reducing the predictive performance of the induced models. Depending on the target algorithm, the Meta-learning recommender system can statistically outperform the baselines. The significance of these results opens a number of new avenues for future work.

Currently, there is no therapy targeting septic cardiomyopathy (SC), a key contributor to organ dysfunction in sepsis. In this study, we used a machine learning (ML) pipeline to explore transcriptomic, proteomic, and metabolomic data from patients with septic shock, and prospectively collected measurements of high-sensitive cardiac troponin and echocardiography. The purposes of the study were to suggest an exploratory methodology to identify and characterise the multiOMICs profile of (i) myocardial injury in patients with septic shock, and of (ii) cardiac dysfunction in patients with myocardial injury. The study included 27 adult patients admitted for septic shock. Peripheral blood samples for OMICS analysis and measurements of high-sensitive cardiac troponin T (hscTnT) were collected at two time points during the ICU stay. A ML-based study was designed and implemented to untangle the relations among the OMICS domains and the aforesaid biomarkers. The resulting ML pipeline consisted of two main experimental phases: recursive feature selection (FS) assessing the stability of biomarkers, and classification to characterise the multiOMICS profile of the target biomarkers. The application of a ML pipeline to circulate OMICS data in patients with septic shock has the potential to predict the risk of myocardial injury and the risk of cardiac dysfunction.

In the context of modern industrial systems, efficient failure management is crucial to maintain operational integrity, minimize downtime, and optimize maintenance.This paper explores the application of Automated Machine Learning (AutoML) to enhance both the diagnostic and the prognostic of failure causality in industrial systems.Different failure causes are detected by failure causality diagnostics, and the upcoming failure could be prevented by failure causality prognostics.In fact, future failures could be avoided by preventing their causalities.Traditional machine learning (ML) approaches require significant manual intervention for model selection, hyperparameter tuning, and feature engineering, which can be time-consuming and cost-consuming.AutoML, on the other hand, automates these processes, enabling faster and more accurate predictions while reducing the need for extensive domain expertise.AutoML could be applied for prognostics, predicting the remaining useful life (RUL) of components and foreseeing future failures.This paper integrates AutoML into real-time failure diagnostics, identifying the root causes of system malfunctions using historical and sensor data.The Steel Plates Faults industrial real-world data set is considered to be surveyed for fault detection using AutoML.The run times and accuracy acquired by AutoML are stated to clarify its superiority.

Performance of Automatic Machine Learning versus Radiologists in the Evaluation of Endometrium on Computed Tomography

The emergence of the mobile applications that are driven by machine learning (ML) has brought a breakthrough in the world of healthcare, e-commerce, and entertainment with its personalized and smart user experiences. Despite the wide application of ML pipelines in mobile environments, the security and privacy issues that could be raised have become an issue of high concern on a global scale. The storage and processing of data, combined with the deployment of advanced ML models, expose vulnerabilities to risks such as data leakage, adversarial attacks, and unauthorized access. These threats not only compromise user trust but also incur applications with financial and brand damage. The present article deals with the important problems of the safety of mobile ML pipelines, namely adversarial manipulations, insecure storage, and transmitting of data. The vital techniques that can help to work out those problems are described along with the most popular ones, namely secure model storage based on encryption, federated learning for data transfer reduction, and model robustness tactics that will increase adversarial attacks defense. Besides that, new technologies like homomorphic encryption and blockchain are also discussed as ways to secure model updates. Consequently, this paper is involved in the technical and practical levels of the project, and it is meant to show the importance of building secure and privacy-preserving ML pipelines that do not only keep data for user trust but also maintain efficiency. It serves the purpose of opening the way to moving to secure mobile ML solutions in the time of intelligent mobile systems that have started to be of great interest.

In this paper, we detail how damage mechanisms commonly found on marine terminal structures and process equipment can be automatically detected using a machine learning (ML) pipeline. Defects in its varied forms, such as corrosion, steel, wood and concrete damages must be closely monitored in offshore oil and gas platforms and also in marine terminals; the cost of not monitoring and detecting defects can range from unplanned shutdowns, plant damage, to human fatalities. On-site inspection of defects is labor intensive and require frequent travel to offshore and onshore facilities. This process has two shortcomings: firstly, it is costly involving man hours spent on the facility and secondly only a subset of the total defects is monitored during each inspection due to difficulty in obtaining full coverage; specially in this scenario a high-risk defect could be missed completely. We use marine terminals as an example of how defects can be detected using machine learning techniques, however the pipeline for generating the ML models could be extended to other types of defects and structures. Significantly, the output from our ML pipeline can be used as an input to the Risk Based Inspection (RBI) system assisting asset operators in their inspection and maintenance planning. Furthermore, digitized inspections and ML can provide around 95% coverage of the terminal while avoiding problems faced by a human inspector such as producing consistent results across inspections and sites. Our system captured terrestrial spherical imagery and range data (LIDAR), subsea imagery, sonar, and ultrasonic thickness (UT) gauge data using different sensors and equipment on a marine terminal. Equipment used in our data-capture included an unmanned aerial vehicle (UAV), a terrestrial scanner, a remotely operated vehicle for underwater imaging (ROV) and a remotely operated surface vehicle (ROSV). For the purposes of this work, we used the images captured by the terrestrial scanner. We built an automated defect detection pipeline using state of the art machine learning and artificial intelligence (AI) tools taking the cube face images as inputs. The defect detection pipeline was trained with examples of defects labeled by asset integrity engineers to ensure quality of results. We built the pipeline to detect damage mechanisms for steel and wooden structures, however our system could be scaled to incorporate other types of defects and structures. Our AI pipeline was able to achieve a high level of accuracy identifying wood damage with a mean-Average Precision (m-AP) of 0.80 and steel damage with a m-AP of 0.84, where a m-AP of 1.0 indicates perfect detection of damage.

Modern machine learning services and systems are complicated data systems --- the process of designing such systems is an art of compromising between functionality , performance , and quality . Providing different levels of system supports for different functionalities, such as automatic feature engineering, model selection and ensemble, and hyperparameter tuning, could improve the quality, but also introduce additional cost and system complexity. In this paper, we try to facilitate the process of asking the following type of questions: How much will the users lose if we remove the support of functionality x from a machine learning service? Answering this type of questions using existing datasets, such as the UCI datasets, is challenging. The main contribution of this work is a novel dataset, MLBench, harvested from Kaggle competitions. Unlike existing datasets, MLBench contains not only the raw features for a machine learning task, but also those used by the winning teams of Kaggle competitions. The winning features serve as a baseline of best human effort that enables multiple ways to measure the quality of machine learning services that cannot be supported by existing datasets, such as relative ranking on Kaggle and relative accuracy compared with best-effort systems. We then conduct an empirical study using MLBench to understand example machine learning services from Amazon and Microsoft Azure, and showcase how MLBench enables a comparative study revealing the strength and weakness of these existing machine learning services quantitatively and systematically. The full version of this paper can be found at arxiv.org/abs/1707.09562

Auto-PLD is the automatic design algorithm framework of machine learning pipeline for the whole-process data analysis scenario. First, define a machine learning pipeline with five stages and can support the processing of continuous and discrete type features separately. Then, the automated pipeline design problem is decomposed into two sub problems: structure search and hyper parameter optimization, and an algorithm combining reinforcement learning and Bayesian optimization is proposed to alternately optimize the two subproblems. Finally, in order to improve the efficiency of automatic pipeline design, two parallelized pipeline construction methods are further proposed. Experimental results show that Auto-PLD outperforms auto-sklearn with most datasets. Moreover, with the increase of computing nodes, the parallelized Auto-PLD can further improve the pipeline building performance. The Auto-PLD-random method performed at 1, 4, and 8 hours respectively: 19, 22, 23, 27, 26, and 25. Automated Pipeline Design with Q-learning (Auto-PLD-Q) methods performed at 1, 4, and 8 hours: 18, 20, 26; 23, 27, and 27. The Auto-PLD-DeepC method performed at 1, 4, and 8 hours, respectively: 21,23,27; 22, 26, and 26. Auto-PLD-PG methods performed at 1, 4, and 8 hours, respectively: 19, 21, 27; 26, 26, and 25. Auto-LLE is an automated machine learning algorithm framework for lifelong learning scenarios. For the classification task based on concept drift and data imbalance, an algorithm based on weighted ensemble learning of adaptive model was proposed. Concept types are divided into “long-term concepts” and “short-term concepts,” and different types of concepts are handled separately using incremental learners and adaptive weight update methods, and automatically capture the concept drift and improve the model prediction performance. Based on Auto-PLD and Auto-LLE, we design and implement a system that supports both automated pipeline design and automated lifelong learning. In system design, high system accessibility and scalability are obtained by designing easy high-level programming interface and pluggable module integration. In task type, the common data analysis tasks such as classification, regression, and clustering are supported.

Novel technologies in automated machine learning ease the complexity of algorithm selection and hyper-parameter optimization. However, these are usually restricted to supervised learning tasks such as classification and regression, while unsupervised learning remains a largely unexplored problem. In this paper, we offer a solution for automating machine learning specifically for the case of unsupervised learning with clustering, in a domain-agnostic manner. This is achieved through a combination of state-of-the-art processes based on meta-learning for algorithm and evaluation criteria selection, and evolutionary algorithm for hyper-parameter tuning. We introduce a robust and scalable interactive tool, named cSmartML, built on scikit-learn with 8 clustering algorithms. In order to capture more than a single measure of goodness of the output clustering solution, cSmartML optimizes multiple objective functions. A pareto-approach evaluates each objective simultaneously for each clustering solution. On each of the 27 real and synthetic benchmark datasets, we show that the performance of cSmartML is often much better than using standard selection and hyper-parameter optimization methods. In addition, experimentation reveals that cSmartML takes advantage of the defined objective functions on multi-objective functions framework.