Interactive machine learning for health informatics: when do we need the human-in-the-loop?

With the Internet of Things paradigm, the data generated by the rapidly increasing number of connected devices lead to new possibilities, such as using machine learning for activity recognition in smart environments. However, it also introduces several challenges. The sensors of different devices might be of different types, making the fusion of data non-trivial. Moreover, the devices are often mobile, resulting in that data from a particular sensor is not always available, i.e. there is a need to handle data from a dynamic set of sensors. From a machine learning perspective, the data from the sensors arrives in a streaming fashion, i.e., online learning, as compared to many learning problems where a static dataset is assumed. Machine learning is in many cases a good approach for classification problems, but the performance is often linked to the quality of the data. Having a good data set to train a model can be an issue in general, due to the often costly process of annotating the data. With dynamic and heterogeneous data, annotation can be even more problematic, because of the ever-changing environment. This means that there might not be any, or a very small amount of, annotated data to train the model on at the start of learning, often referred to as the cold start problem. To be able to handle these issues, adaptive systems are needed. With adaptive we mean that the model is not static over time, but is updated if there for instance is a change in the environment. By including human-in-the-loop during the learning process, which we refer to as interactive machine learning, the input from users can be utilized to build the model. The type of input used is typically annotations of the data, i.e. user input in the form of correctly labelled data points. Generally, it is assumed that the user always provides correct labels in accordance with the chosen interactive learning strategy. In many real-world applications these assumptions are not realistic however, as users might provide incorrect labels or not provide labels at all in line with the chosen strategy. In this thesis we explore which interactive learning strategies are possible in the given scenario and how they affect performance, as well as the effect of machine learning algorithms on performance. We also study how a user who is not always reliable, i.e. that does not always provide a correct label when expected to, can affect performance. We propose a taxonomy of interactive online machine learning strategies and test how the different strategies affect performance through experiments on multiple datasets. The findings show that the overall best performing interactive learning strategy is one where the user provides labels when previous estimations have been incorrect, but that the best performing machine learning algorithm depends on the problem scenario. The experiments also show that a decreased reliability of the user leads to decreased performance, especially when there is a limited amount of labelled data.

Recent advances in automatic machine learning (aML) allow solving problems without any human intervention. However, sometimes a human-in-the-loop can be beneficial in solving computationally hard problems. In this paper we provide new experimental insights on how we can improve computational intelligence by complementing it with human intelligence in an interactive machine learning approach (iML). For this purpose, we used the Ant Colony Optimization (ACO) framework, because this fosters multi-agent approaches with human agents in the loop. We propose unification between the human intelligence and interaction skills and the computational power of an artificial system. The ACO framework is used on a case study solving the Traveling Salesman Problem, because of its many practical implications, e.g. in the medical domain. We used ACO due to the fact that it is one of the best algorithms used in many applied intelligence problems. For the evaluation we used gamification, i.e. we implemented a snake-like game called Traveling Snakesman with the MAX–MIN Ant System (MMAS) in the background. We extended the MMAS–Algorithm in a way, that the human can directly interact and influence the ants. This is done by “traveling” with the snake across the graph. Each time the human travels over an ant, the current pheromone value of the edge is multiplied by 5. This manipulation has an impact on the ant’s behavior (the probability that this edge is taken by the ant increases). The results show that the humans performing one tour through the graphs have a significant impact on the shortest path found by the MMAS. Consequently, our experiment demonstrates that in our case human intelligence can positively influence machine intelligence. To the best of our knowledge this is the first study of this kind.

In this paper, we highlight three issues that limit performance of machine learning on biomedical images, and tackle them through 3 case studies: 1) Interactive Machine Learning (IML): we show how IML can drastically improve exploration time and quality of direct volume rendering. 2) transfer learning: we show how transfer learning along with intelligent pre-processing can result in better Alzheimer's diagnosis using a much smaller training set 3) data imbalance: we show how our novel focal Tversky loss function can provide better segmentation results taking into account the imbalanced nature of segmentation datasets. The case studies are accompanied by in-depth analytical discussion of results with possible future directions.

Proposed system does analysis and prediction of with Regression Analysis. Conceptually the Intelligence is a combination of technologies and processing (i.e. Virtualization, Cloud Computing, and Machine Learning etc.). Now a day Machine learning (ML) is one of the fastest growing fields in computer science and health care informatics. ML’s goal is to develop algorithm which can learn as time passes and improve also and can be used for prediction algorithm that can interact with agent and can optimize their learning behavior through these interactions where the agent can also be a human. This paper discusses about working Application of Pseudo intelligence using health care. The result of pseudo intelligence improves the efficiency in providing better survivability of people. Intelligence system is helpful to find out need requirement of a particular health care. Simple linear multivariate regression technique is used for prediction. Dataset is taken from UCI repository. We have achieved maximum accuracy using regression. Aggregation is used to convert uncertain probabilistic data into a certain data. Data mining is used for future power requirement analysis.

At present, the people monitoring, in particular outside clinical sites, has become a major research concern. With the rapid technological advances, remote monitoring is considered as a vital component of future e-Health systems. The techniques of the Internet of Things (IoT) offer the opportunity to change the nature of health services in an omnipresent way, and to create them based on the patients’ physical condition rather than their feelings. Nevertheless, these new technologies have also made this area more difficult to manage and manipulate. In this paper, we proposed a monitoring system that can send and analyze the collected real-time data for effective decision-making in the case of Remote Cardiac Patients. We propose to design and develop a Remote Heart Monitoring System (RHMS) that combines two complementary technologies (1) machine learning for generating intelligent valuable information, and (2) visual analytics for gaining insight the collected real-time data. The success of these systems is based on the quality of their design and development. For this reason, we propose to use the multi-agent architecture. The developed prototype was evaluated by 30 participants to verify its feasibility. It has been validated in terms of utility and usability, RHMS time of response and multi-agent modeling quality. The multi-agent modeling of a RHMS based on automatic and interactive machine learning (visual data mining) presented interesting results. It is a distributed architecture of a real-time monitoring system for cardiac patients helping remote monitors to improve the management of their out-of-hospital data.

Interactive Machine Learning (IML) can enable intelligent systems to interactively learn from their end-users, and is quickly becoming more and more relevant to many application domains. Although it places the human in the loop, interactions are mostly performed via mutual explanations that miss contextual information. Furthermore, current model-agnostic IML strategies such as CAIPI are limited to ’destructive’ feedback, meaning that they solely allow an expert to prevent a learner from using irrelevant features. In this work, we propose a novel interaction framework called Semantic Interactive Learning for the domain of document classification, located at the intersection between Natural Language Processing (NLP) and Machine Learning (ML). We frame the problem of incorporating constructive and contextual feedback into the learner as a task involving finding an architecture that enables more semantic alignment between humans and machines while at the same time helping to maintain the statistical characteristics of the input domain when generating user-defined counterexamples based on meaningful corrections. Therefore, we introduce a technique called SemanticPush that is effective for translating conceptual corrections of humans to non-extrapolating training examples such that the learner’s reasoning is pushed towards the desired behavior. Through several experiments we show how our method compares to CAIPI, a state of the art IML strategy, in terms of Predictive Performance and Local Explanation Quality in downstream multi-class classification tasks. Especially in the early stages of interactions, our proposed method clearly outperforms CAIPI while allowing for contextual interpretation and intervention. Overall, SemanticPush stands out with regard to data efficiency, as it requires fewer queries from the pool dataset to achieve high accuracy.

Recent technological advances allowed an exponential expansion of biological sequence data, and the extraction of meaningful information through Machine Learning (ML) algorithms. This knowledge improved the understanding of the mechanisms related to several fatal diseases, e.g., Cancer and COVID-19, helping to develop innovative solutions, such as CRISPR-based gene editing, coronavirus vaccine, and precision medicine. These advances benefit our society and economy, directly impacting peoples lives in various areas, such as health care, drug discovery, forensic analysis, and food analysis. Nevertheless, ML approaches applied to biological data require representative, quantitative, and informative features. Necessarily, as many ML algorithms can handle only numerical data, sequences need to be translated into a feature vector. This process, known as feature extraction, is a fundamental step for the elaboration of high-quality ML-based models in bioinformatics, by allowing the feature engineering stage, with the design and selection of suitable features. Feature engineering, ML algorithm selection, and hyperparameter tuning are often manual and time-consuming processes, requiring extensive domain knowledge, and performed manually by a human expert. To deal with this problem, we developed a new package, BioAutoML, which automatically runs an end-to-end ML pipeline. BioAutoML extracts numerical and informative features from biological sequence databases, automating feature selection, recommendation of ML algorithm(s), and tuning of hyperparameters, using Automated ML (AutoML). BioAutoML has two components, divided into four modules, (1) automated feature engineering (feature extraction and selection modules) and (2) Metalearning (algorithm recommendation and hyperparameter tuning modules). Our experimental results, assessing the relevance of our proposal, indicate robust results for different problem domains, such as SARS-CoV-2, anticancer peptides, HIV sequences, and non-coding RNAs. According to our systematic review, our proposal is innovative compared to available studies in the literature, being the first study to propose automated feature engineering and metalearning for biological sequences. BioAutoML has a high potential to significantly reduce the expertise required to use ML pipelines, aiding researchers in combating diseases, particularly in low- and middle-income countries. This initiative can provide biologists, physicians, epidemiologists, and other stakeholders with an opportunity for widespread use of these techniques to enhance the health and well-being of their communities.

Software designers are coming up with tools having machine learning (ML) capabilities integrated into them. Even naive users can build ML-based models for their respective problem domains using these tools. However, a majority of the advanced ML models behave like a black box in the sense that their behavior is not easily interpretable by humans. This lack of human interpretability acts as a hurdle toward acceptance and subsequent deployment of ML-based solutions. This research work is based on an intuitive idea that in an ideal ML-based solution, the provided dataset, the ML model learned using this dataset, and human domain experts must agree in terms of their perception regarding the problem domain. The objective of this work is to propose a framework that first listens to the dataset, the ML model, and human domain experts individually, and then measure the degree of agreement between them. Listening to the dataset refers identifying important characteristics by computing information gain using entropy and Gini index. Listening to the ML model refers interpreting its decision-making behavior by computing variable importance measures. Listening to human experts refers taking feedback in terms of important features as per their perception regarding the problem domain. The task of measuring agreement between dataset, model, and human experts has been modeled as a “two-judges n-participants” rank correlation problem. The proposed approach has been evaluated on a novel problem domain of predicting joining behavior of freshmen students. A positive degree of agreement was observed between the dataset, the ML model, and human experts in terms of their perception regarding the problem domain. The agreement between the provided dataset, the learned model, and human domain experts helps verify learning acquired by the ML model against the prevailing domain knowledge. This work has the potential to form the basis for developing formal quantitative metrics for evaluating ML models in terms of reliable learning and capability to facilitate trust of human users.

Chapter 15 - Clinical decision support systems to improve the diagnosis and management of respiratory diseases

Technology drives advances in science. Giving scientists access to more powerful tools for collecting and understanding data enables them to both ask and answer new kinds questions that were previously beyond their reach. Of these new tools at their disposal, machine learning offers the opportunity to understand and analyze data at unprecedented scales and levels of detail. The standard machine learning pipeline consists of data labeling, feature extraction, training, and evaluation. However, without expert machine learning knowledge, it is difficult for scientists to optimally construct this pipeline to fully leverage machine learning in their work. Using ecology as a motivating example, we analyze a typical scientist's data collection and processing workflow and highlight many problems facing practitioners when attempting to capitalize on advances in machine learning and pattern recognition. Understanding these shortcomings allows us to outline several novel and underexplored research directions. We end with recommendations to motivate progress in future cross-disciplinary work.

Image grouping in knowledge-rich domains is challenging, since domain knowledge and human expertise are key to transform image pixels into meaningful content. Manually marking and annotating images is not only labor-intensive but also ineffective. Furthermore, most traditional machine learning approaches cannot bridge this gap for the absence of experts' input. We thus present an interactive machine learning paradigm that allows experts to become an integral part of the learning process. This paradigm is designed for automatically computing and quantifying interpretable grouping of dermatological images. In this way, the computational evolution of an image grouping model, its visualization, and expert interactions form a loop to improve image grouping. In our paradigm, dermatologists encode their domain knowledge about the medical images by grouping a small subset of images via a carefully designed interface. Our learning algorithm automatically incorporates these manually specified connections as constraints for reorganizing the whole image dataset. Performance evaluation shows that this paradigm effectively improves image grouping based on expert knowledge.

This open access book presents the first comprehensive overview of general methods in Automatic Machine Learning (AutoML), collects descriptions of existing systems based on these methods, and discusses the first international challenge of AutoML systems. The book serves as a point of entry into this quickly-developing field for researchers and advanced students alike, as well as providing a reference for practitioners aiming to use AutoML in their work. The recent success of commercial ML applications and the rapid growth of the field has created a high demand for off-the-shelf ML methods that can be used easily and without expert knowledge. Many of the recent machine learning successes crucially rely on human experts, who select appropriate ML architectures (deep learning architectures or more traditional ML workflows) and their hyperparameters; however the field of AutoML targets a progressive automation of machine learning, based on principles from optimization and machine learning itself.

Network alarm triage refers to grouping and prioritizing a stream of low-level device health information to help operators find and fix problems. Today, this process tends to be largely manual because existing rule-based tools cannot easily evolve with the network. We present CueT, a system that uses interactive machine learning to constantly learn from the triaging decisions of operators. It then uses that learning in novel visualizations to help them quickly and accurately triage alarms. Unlike prior interactive machine learning systems, CueT handles a highly dynamic environment where the groups of interest are not known a priori and evolve constantly. Our evaluations with real operators and data from a large network show that CueT significantly improves the speed and accuracy of alarm triage.

Benchmarking AutoML solutions for concrete strength prediction: Reliability, uncertainty, and dilemma

PurposeDental implant abutments are defined as medical devices by their intended use. Surfaces of custom-made CAD/CAM two-piece abutments may become contaminated during the manufacturing process in the dental lab. Inadequate reprocessing prior to patient care may contribute to implant-associated complications. Risk-adapted hygiene management is required to meet the requirements for medical devices.MethodsA total of 49 CAD/CAM-manufactured zirconia copings were bonded to prefabricated titanium bases. One group was bonded, polished, and cleaned separately in dental laboratories throughout Germany (LA). Another group was left untreated (NC). Five groups received the following hygiene regimen: three-stage ultrasonic cleaning (CP and FP), steam (SC), argon–oxygen plasma (PL), and simple ultrasonic cleaning (UD). Contaminants were detected using scanning electron microscopy (SEM) and energy-dispersive X-ray spectroscopy (EDS) and segmented and quantified using interactive machine learning (ML) and thresholding (SW). The data were statistically analysed using non-parametric tests (Kruskal–Wallis test, Dunn’s test).ResultsSignificant differences in contamination levels with the different cleaning procedures were found (p ≤ 0.01). The FP–NC/LA groups showed the most significant difference in contamination levels for both measurement methods (ML, SW), followed by CP–LA/NC and UD–LA/NC for SW and CP–LA/NC and PL–LA/NC for ML (p ≤ 0.05). EDS revealed organic contamination in all specimens; traces of aluminum, silicon, and calcium were detected.ConclusionsChemothermal cleaning methods based on ultrasound and argon–oxygen plasma effectively removed process-related contamination from zirconia surfaces. Machine learning is a promising assessment tool for quantifying and monitoring external contamination on zirconia abutments.Graphical