MLbench

Over the past years, Machine Learning has been applied to an increasing number of problems across numerous industries. However, the steady rise in the application of Machine Learning has not come without challenges since companies often lack the expertise or infrastructure to build their own Machine Learning systems. These challenges led to the emergence of a new paradigm, called Machine Learning as a Service. Scientific literature has mainly analyzed this topic in the context of platform solutions that provide ready-to-use environments for companies. We recently have developed a platform-independent approach and labeled it Machine Learning Services. The aim of the present study is to identify and evaluate challenges and opportunities in the application of Machine Learning Services. To do so, we conducted a Delphi Study with a panel of machine learning experts. The study consisted of three rounds and was structured according to the five steps of the Data Science Lifecycle. A variety of challenges from the areas “Communication”, “Environment”, “Approach”, “Data”, “Retraining, Testing, Monitoring and Updating”, “Model Training and Evaluation” were identified. Subsequently, the challenges revealed by the Delphi Study were compared with previous work on Machine Learning as a Service, which resulted from a structured literature review. The identified areas serve as possible future research fields and give further implications for practice. Alleviating communication issues and assessing the business IT infrastructure prior to the machine learning project are among the key findings of our study.

In recent years, Machine Learning (ML) and Automated Machine Learning (Auto-ML) have attracted significant attention. The ML pipeline involves repetitive tasks such as data preprocessing, feature engineering, model selection, and hyperparameter tuning. Developing a machine learning model demands considerable time for development, stress testing, and numerous experiments. Additionally, constructing a model with a limited search space of pipeline steps and various algorithms can take hours. As a result, Auto-ML has become widely adopted to reduce the time and effort required for these tasks. However, most current Auto-ML frameworks primarily concentrate on algorithm selection and hyperparameter optimization, known as CASH, while overlooking other critical ML pipeline steps like data preprocessing and feature engineering. This limited focus often results in suboptimal pipelines for specific datasets. Moreover, a significant number of frameworks overlook the integration of meta-learning, resulting in the promotion of high-performing pipelines customized for individual tasks rather than a universally optimal solution. Consequently, this deficiency necessitates the quest for a new pipeline tailored to each unique task, further underscoring the importance of a more comprehensive approach in Auto-ML frameworks. Additionally, while some Auto-ML frameworks address the entire pipeline, they often overlook the challenges posed by imbalanced datasets. To address these issues, we propose a novel and efficient meta-learning Auto-ML framework that effectively manages imbalanced datasets. The proposed framework outperforms state-of-the-art results in terms of accuracy, precision, recall, and time, demonstrating, on average, more than 5% improvement compared to the existing auto-ML frameworks. This paper also illustrates how our proposed framework outperforms current state-of-the-art solutions.

Machine learning solutions have been successfully used to solve many simple and complex problems. However, their development process still relies on human experts to perform tasks such as data preprocessing, feature engineering and model selection. As the complexity of these tasks increases, so does the demand for automated solutions, namely Automated Machine Learning (AutoML). Most algorithms employed in these systems have hyperparameters whose configuration may directly affect their predictive performance. Therefore, hyperparameter tuning is a recurring task in AutoML systems. This thesis investigated how to efficiently automate hyperparameter tuning by means of Meta-learning. To this end, large-scale experiments were performed tuning the hyperparameters of different classification algorithms, and an enhanced experimental methodology was adopted throughout the thesis to explore and learn the hyperparameter profiles for different classification algorithms. The results also showed that in many cases the default hyperparameter settings induced models that are on par with those obtained by tuning. Hence, a new Meta-learning recommender system was proposed to identify when it is better to use default values and when to tune classification algorithms for each new dataset. The proposed system is capable of generalizing several learning processes into a single modular framework, along with the possibility of assigning different algorithms. Furthermore, a descriptive analysis of model predictions is used to identify which data characteristics affect the necessity for tuning in each one of the algorithms investigated in the thesis. Experimental results also demonstrated that the proposed recommender system reduced the time spent on optimization processes, without reducing the predictive performance of the induced models. Depending on the target algorithm, the Meta-learning recommender system can statistically outperform the baselines. The significance of these results opens a number of new avenues for future work.

Recent technological advances allowed an exponential expansion of biological sequence data, and the extraction of meaningful information through Machine Learning (ML) algorithms. This knowledge improved the understanding of the mechanisms related to several fatal diseases, e.g., Cancer and COVID-19, helping to develop innovative solutions, such as CRISPR-based gene editing, coronavirus vaccine, and precision medicine. These advances benefit our society and economy, directly impacting peoples lives in various areas, such as health care, drug discovery, forensic analysis, and food analysis. Nevertheless, ML approaches applied to biological data require representative, quantitative, and informative features. Necessarily, as many ML algorithms can handle only numerical data, sequences need to be translated into a feature vector. This process, known as feature extraction, is a fundamental step for the elaboration of high-quality ML-based models in bioinformatics, by allowing the feature engineering stage, with the design and selection of suitable features. Feature engineering, ML algorithm selection, and hyperparameter tuning are often manual and time-consuming processes, requiring extensive domain knowledge, and performed manually by a human expert. To deal with this problem, we developed a new package, BioAutoML, which automatically runs an end-to-end ML pipeline. BioAutoML extracts numerical and informative features from biological sequence databases, automating feature selection, recommendation of ML algorithm(s), and tuning of hyperparameters, using Automated ML (AutoML). BioAutoML has two components, divided into four modules, (1) automated feature engineering (feature extraction and selection modules) and (2) Metalearning (algorithm recommendation and hyperparameter tuning modules). Our experimental results, assessing the relevance of our proposal, indicate robust results for different problem domains, such as SARS-CoV-2, anticancer peptides, HIV sequences, and non-coding RNAs. According to our systematic review, our proposal is innovative compared to available studies in the literature, being the first study to propose automated feature engineering and metalearning for biological sequences. BioAutoML has a high potential to significantly reduce the expertise required to use ML pipelines, aiding researchers in combating diseases, particularly in low- and middle-income countries. This initiative can provide biologists, physicians, epidemiologists, and other stakeholders with an opportunity for widespread use of these techniques to enhance the health and well-being of their communities.

Machine Learning can be used with cybersecurity data to protect organizations by using artificial intelligence (AI) to generate rules and models for thread detection. Cloud platforms offer the ability to scale AI efforts as well as automatically generate machine learning models (Auto ML). Adoption of Auto-ML is increasing which is resulting in rapid improvements of the technology. The objective of this paper is to demonstrate the Auto ML functionalities against cyber security threat detection using available tools in free tier accounts created on three different cloud platforms (Microsoft Azure, Google, and IBM). We determined the performance of these tools by the evaluating the optimization speed and accuracy results. A comparison of the advantages of each of the results from the different platforms are presented. Overall, all three platforms performed greater than 70% accuracy with the IBM Cloud Platform having the strongest performance.

The last decade has witnessed the explosion of machine learning research studies with the inception of several algorithms proposed and successfully adopted in different application domains. However, the performance of multiple machine learning algorithms is very sensitive to multiple ingredients (e.g., hyper-parameters tuning and data cleaning) where a significant human effort is required to achieve good results. Thus, building well-performing machine learning algorithms requires domain knowledge and highly specialized data scientists. Automated machine learning (autoML) aims to make easier and more accessible the use of machine learning algorithms for researchers with varying levels of expertise. Besides, research effort to date has mainly been devoted to autoML for supervised learning, and only a few research proposals have been provided for the unsupervised learning. In this paper, we present an overview of the autoML field with a particular emphasis on the automated methods and strategies that have been proposed for unsupervised anomaly detection.

Working together is important aspect of future industry. Therefore, technologies like Internet of Things (IoT), cloud computing, SOA give rise to another industrial revolution. We propose here a concept definition, which focuses on data acquisition, integration and predictive control in the industry. The concept consists of industrial IoT gateway, cloud services and machine learning services. We used machine learning to verify our data acquisition solution and we implemented prediction control as a cloud service. Finally, proposed solution will exceed boundaries inside ICS (Information and Control System), improve flexibility, interoperability and test plant prediction control in smart manufacturing.

In the context of modern industrial systems, efficient failure management is crucial to maintain operational integrity, minimize downtime, and optimize maintenance.This paper explores the application of Automated Machine Learning (AutoML) to enhance both the diagnostic and the prognostic of failure causality in industrial systems.Different failure causes are detected by failure causality diagnostics, and the upcoming failure could be prevented by failure causality prognostics.In fact, future failures could be avoided by preventing their causalities.Traditional machine learning (ML) approaches require significant manual intervention for model selection, hyperparameter tuning, and feature engineering, which can be time-consuming and cost-consuming.AutoML, on the other hand, automates these processes, enabling faster and more accurate predictions while reducing the need for extensive domain expertise.AutoML could be applied for prognostics, predicting the remaining useful life (RUL) of components and foreseeing future failures.This paper integrates AutoML into real-time failure diagnostics, identifying the root causes of system malfunctions using historical and sensor data.The Steel Plates Faults industrial real-world data set is considered to be surveyed for fault detection using AutoML.The run times and accuracy acquired by AutoML are stated to clarify its superiority.

Machine learning algorithms have been widely employed in various applications and fields. Novel technologies in automated machine learning (AutoML) ease algorithm selection and hyperparameter optimization complexity. AutoML frame-works have achieved notable success in hyperparameter tuning and surpassed the performance of human experts. However, depending on such frameworks as black-box can leave machine learning practitioners without insights into the inner working of the AutoML process and hence influence their trust in the models produced. In addition, excluding humans from the loop creates several limitations. For example, most of the current AutoML frameworks ignore the user preferences on defining or controlling the search space, which consequently can impact the performance of the models produced and the acceptance of these models by the end-users. The research in the area of transparency and controllability of AutoML has attracted much interest lately, both in academia and industry. However, existing tools are usually restricted to supervised learning tasks such as classification and regression, while unsupervised learning, particularly clustering, remains a largely unexplored problem. Motivated by these shortcomings, we design and implement cSmartML-GlassBox, an interactive visualization tool that en-ables users to refine the search space of AutoML and analyze the results. cSmartML-GlassBox is equipped with a recommendation engine to recommend a time budget that is likely adequate for a new dataset to obtain well-performing pipeline. In addition, the tool supports multi-granularity visualization to enable machine learning practitioners to monitor the AutoML process, analyze the explored configurations and refine/control the search space. Furthermore, cSmartML-GlassBox is equipped with a logging mechanism such that repeated runs on the same dataset can be more effective by avoiding evaluating the same previously considered configurations. We demonstrate the effectiveness and usability of the cSmartML-GlassBox through a user evaluation study with 23 participants and an expert-based usability study based on four experts. We find that the proposed tool increases users' understanding and trust in the AutoML frameworks.

To relieve the pain of manually selecting machine learning algorithms and tuning hyperparameters, automated machine learning (AutoML) methods have been developed to automatically search for good models. Due to the huge model search space, it is impossible to try all models. Users tend to distrust automatic results and increase the search budget as much as they can, thereby undermining the efficiency of AutoML. To address these issues, we design and implement ATMSeer, an interactive visualization tool that supports users in refining the search space of AutoML and analyzing the results. To guide the design of ATMSeer, we derive a workflow of using AutoML based on interviews with machine learning experts. A multi-granularity visualization is proposed to enable users to monitor the AutoML process, analyze the searched models, and refine the search space in real time. We demonstrate the utility and usability of ATMSeer through two case studies, expert interviews, and a user study with 13 end users.

Automated machine learning (AutoML) represents one of the fastest-growing fields in both industry and academia. Its popularity has increased over the last few years due to the rising demand for machine learning systems usable by non-experts. Existing open-source systems fail to fully automate the machine learning tasks, being more like a black box that generates non-reusable pipelines. We propose an end-to-end system accessible to non-specialists and data scientists who wish to effortlessly create reusable and easy to understand pipelines for any regression and classification problems. The proposed solution increases the automation level by performing an advanced data cleaning followed by a complete feature engineering process without any domain knowledge about the data set. These processes are integrated into a search that generates for each model the optimum instance of preprocessed data. Bayesian optimization (BO) with a Gaussian process (GP) maximizes the performance of these models that will be ensembled using standard and probability-based stacking. We prove that the system performs better than 43.47% of practitioners on real-world data sets, being robust to any variation of tabular data. The true potential is revealed by surpassing at least one similar tool in 50% of performed benchmarks. These results were achieved for any tested data set, regardless of its dimensionality, after a maximum of 30 minutes.

Recently, an increasing number of Artificial Intelligence services have been developed for a variety of domains. Machine Learning and especially Deep Learning services require a large amount of data to provide their functionality. Since data collection is typically complex and difficult, there is often not enough data available. Machine learning services such as anomaly detection or disaggregation algorithms are also being developed in the smart living domain. In practice, however, only a few energy datasets are publicly available, as the collection of such data is expensive and time-consuming due to the equipment required. One way to generate more smart meter data is to use a simulation. Developing such a simulation that is capable of generating meaningful data is a complex task. Therefore, in this paper, we present the Synthetic Time Series Data Generator (SynTiSeD), a multi-agent-based simulation tool that generates meaningful synthetic energy data based on real-world data. Furthermore, SynTiSeD allows generating data of critical situations, which are important for the development of such services, but which cannot be provoked in the real world. For transferability, we demonstrate that Nonintrusive Load Monitoring algorithms trained on synthetic data generated by SynTiSeD provide meaningful results that are even better than those of models trained on real data.

IoT data analytics in dynamic environments: From an automated machine learning perspective

In order to avoid extensive machine learning models selection and optimizations, Automated Machine Learning (AutoML) has arisen as a practical and efficient way to apply machine learning to many different application areas. Data poisoning is a real threat to the accuracy of machine learning models in different settings, and it has in recent research studies been shown that the usage of AutoML can be even more sensitive to data poisoning than is the case for non-AutoML generated models. On the other hand, the usage of AutoML also has the potential of improving the robustness of a model by adapting the model to adversarial patterns. In this way, good accuracy can be maintained despite the attacker’s efforts to poison the data. However, no previous studies have investigated these effects. In this paper, we examine the risks associated with adversarial trajectory attacks in mobile systems, specifically looking into mobility prediction problems. By using mobility data from two different simulation frameworks: a simulator developed by Ericsson, which is based on a real-world deployment of Airtel’s open-network topology, and the ONE framework, we investigate three different AutoML frameworks and how the mobility accuracy for the frameworks is affected by a mobile trajectory attack. Our results show that re-running AutoML at every retraining is vulnerable to adversarial mobility poisoning and shows high accuracy variance. By contrast, using a single, well-chosen model from an initial AutoML search achieves more stable performance across adversarial conditions, even when the training set includes up to 10% adversarial mobility data.

Breast cancer, the most common cancer type among women worldwide, requires early detection and accurate diagnosis for improved treatment outcomes. Segmenting fat and fibroglandular tissue (FGT) in magnetic resonance imaging (MRI) is essential for creating volumetric models, enhancing surgical workflow, and improving clinical outcomes. Manual segmentation is time-consuming and subjective, prompting the development of automated deep-learning algorithms to perform this task. However, configuring these algorithms for 3D medical images is challenging due to variations in image features and preprocessing distortions. Automated machine learning (AutoML) frameworks automate model selection, hyperparameter tuning, and architecture optimization, offering a promising solution by reducing reliance on manual intervention and expert knowledge. We compare nnU-Net and Auto3Dseg, two AutoML frameworks, in segmenting fat and FGT on T1-weighted MRI images from the Duke breast MRI dataset (100 patients). We used threefold cross-validation, employing the Dice similarity coefficient (DSC) and Hausdorff distance (HD) metrics for evaluation. The -test and Tukey honestly significant difference analysis were used to assess statistical differences across methods. nnU-Net achieved DSC scores of (fat) and (FGT), whereas Auto3DSeg achieved (fat) and (FGT). Significant differences in fat HD ( , ) originated from the full resolution and the 3D cascade U-Net. No evidence of significant differences was found in FGT HD or DSC metrics. Ensemble approaches of Auto3Dseg and nnU-Net demonstrated comparable performance in segmenting fat and FGT on breast MRI. The significant differences in fat HD underscore the importance of boundary-focused metrics in evaluating segmentation methods.