P-225 Trustworthy AI algorithm for embryo ranking

Automated decision-making by AI algorithms is increasingly likely to cause civil liability. However, AI algorithms based on machine learning techniques are less explainable due to technical inscrutability arising from the nature of the learning methods itself, legal opacity due to protection of trade secrets or intellectual property rights, or incomprehensibility of the general public or judges due to the complexity and counterintuitiveness of algorithms, thus making its judicial review difficult. When the mechanism of an AI algorithm is at issue in a lawsuit, the question is how an opaque AI algorithm can be evaluated by experts and reviewed by a court. This article first introduces ACCC v. Trivago decision made by the Federal Court of Australia in 2020, focusing on how the experts appointed by each party presented their opinions on the AI algorithm and how the court drew its conclusion based on their opinions. It then studies issues and solutions that may arise in examining opaque AI algorithms in the Korean civil litigation procedures, making comparison with the Trivago decision. It explains basic features of Explainable AI(XAI) methods including Ante hoc methods and post hoc methods. It then points out problems of the Civil Procedure Act and Intellectual Property laws of South Korea in disclosing data necessary for the experts’ analysis on the AI algorithms and in protecting the trade secrets contained in the produced data, and makes suggestions on how to solve those problems. Lastly, it recommends a more active use of party-appointed experts in the judicial review of the AI algorithms by allowing their active participations throughout the litigation procedure in order to clarify issues and deepen the court’s scientific understanding.

Background and Objectives: This study investigated the usefulness of deep neural network (DNN) models based on 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) and blood inflammatory markers to assess the therapeutic response in pyogenic vertebral osteomyelitis (PVO). Materials and Methods: This was a retrospective study with prospectively collected data. Seventy-four patients diagnosed with PVO underwent clinical assessment for therapeutic responses based on clinical features during antibiotic therapy. The decisions of the clinical assessment were confirmed as ‘Cured’ or ‘Non-cured’. FDG-PETs were conducted concomitantly regardless of the decision at each clinical assessment. We developed DNN models depending on the use of attributes, including C-reactive protein (CRP), erythrocyte sedimentation ratio (ESR), and maximum standardized FDG uptake values of PVO lesions (SUVmax), and we compared their performances to predict PVO remission. Results: The 126 decisions (80 ‘Cured’ and 46 ‘Non-cured’ patients) were randomly assigned with training and test sets (7:3). We trained DNN models using a training set and evaluated their performances for a test set. DNN model 1 had an accuracy of 76.3% and an area under the receiver operating characteristic curve (AUC) of 0.768 [95% confidence interval, 0.625–0.910] using CRP and ESR, and these values were 79% and 0.804 [0.674–0.933] for DNN model 2 using ESR and SUVmax, 86.8% and 0.851 [0.726–0.976] for DNN model 3 using CRP and SUVmax, and 89.5% and 0.902 [0.804–0.999] for DNN model 4 using ESR, CRP, and SUVmax, respectively. Conclusions: The DNN models using SUVmax showed better performances when predicting the remission of PVO compared to CRP and ESR. The best performance was obtained in the DNN model using all attributes, including CRP, ESR, and SUVmax, which may be helpful for predicting the accurate remission of PVO.

Despite the plethora of AI-based algorithms developed for anomaly detection in radiology, subsequent integration into clinical setting is rarely evaluated. In this work, we assess the applicability and utility of an AI-based model for brain aneurysm detection comparing the performance of two readers with different levels of experience (2 and 13years). We aim to answer the following questions: 1) Do the readers improve their performance when assisted by the AI algorithm? 2) How much does the AI algorithm impact routine clinical workflow? We reuse and enlarge our open-access, Time-Of-Flight Magnetic Resonance Angiography dataset (N=460). We use 360 subjects for training/validating our algorithm and 100 as unseen test set for the reading session. Even though our model reaches state-of-the-art results on the test set (sensitivity=74%, false positive rate=1.6), we show that neither the junior nor the senior reader significantly increase their sensitivity (p=0.59, p=1, respectively). In addition, we find that reading time for both readers is significantly higher in the "AI-assisted" setting than in the "Unassisted" (+15s, on average; p=3×10-4 junior, p=3×10-5 senior). The confidence reported by the readers is unchanged across the two settings, indicating that the AI assistance does not influence the certainty of the diagnosis. Our findings highlight the importance of clinical validation of AI algorithms in a clinical setting involving radiologists. This study should serve as a reminder to the community to always examine the real-word effectiveness and workflow impact of proposed algorithms.

In the world of trading and complex market transactions, technology has greatly revolutionized and adapted to the fast-growing data and complexities of modern trading strategies. Computers and advanced algorithms have paved the way for a more digitalized world where virtual transactions are common place. Not only can people buy and sell products online, but can also do these transactions automatically. However, thanks to complex AI and ML algorithms; high-frequency trading or HFT strategies can be formulated for highly efficient and optimized automated trading. There are various HFT firms that uses complex ML and AI algorithms to identify and dynamically adjust buying or selling prices according to available market data to maximize profit and save time. Various businesses can take advantage of HFT strategies to optimize selling and buying frequencies. Relevant data is required for executing a HFT strategy properly, along with a sophisticated AI algorithm which is capable of handling vast amounts of data dynamically. The key features of HFT makes it a compelling route for optimized trading and buying. These key features include: lightning-fast transactions, dynamic market data analysis and execution of numerous transactions at once. This research paper aims to understand how AI and ML algorithms can dynamically optimize HFT strategies to further take leverage of available market data. The research paper discusses the implementation of Deep Reinforcement Learning or DLL framework for optimizing HFT strategies. In particular, the implementation discussed adopts a multi-time scale DRL for improved dynamic data processing. This research paper aims to shed light into the effectiveness of an AI-driven solution for improving HFT efficiency for better and faster trading using a simple deep reinforcement learning or DRL machine learning algorithm.

Bots can be broadly divided into two categories: chat bots and smart bots. Chat bots respond using a predefined set of rules and hence the responses are limited. For example, if you build a bot for reserving a table in a restaurant, the bot would always ask the basic questions of date and time, number of people, and seating preference (indoor/outdoor). If you throw a random request at the chat bot, it might not respond with a meaningful message or might just respond with a generic message. Smart bots are more intelligent. They work with a wide variety of information and generate more human-like responses. Smart bots are designed to learn from the conversation and provide more-useful answers as the conversation progresses, leaving the impression of a human-to-human conversation. Bots can be designed to create smarter responses by using cutting-edge artificial intelligence algorithms. Authoring AI algorithms is a complex task, requiring a varied skill set and lots of analysis to build an AI algorithm that can perform tasks like natural language processing and sentiment analysis or generate recommendations. Microsoft Cognitive Services provides a basket of AI algorithms that can be integrated into any application. These algorithms were developed by an expert team and cater across the fields of computer vision, speech, text analysis, natural language processing, knowledge extraction, and web search. In this chapter, we will learn to build smarter bots using Microsoft Cognitive Services and come to understand the capabilities of the evolving list of powerful AI algorithms.

Statistical considerations for testing an AI algorithm used for prescreening lung CT images

Bone age is considered an indicator for the diagnosis of precocious or delayed puberty and a predictor of adult height. We aimed to evaluate the performance of a deep neural network model in assessing rapidly advancing bone age during puberty using elbow radiographs. In all, 4437 anteroposterior and lateral pairs of elbow radiographs were obtained from pubertal individuals from two institutions to implement and validate a deep neural network model. The reference standard bone age was established by five trained researchers using the Sauvegrain method, a scoring system based on the shapes of the lateral condyle, trochlea, olecranon apophysis, and proximal radial epiphysis. A test set (n = 141) was obtained from an external institution. The differences between the assessment of the model and that of reviewers were compared. The mean absolute difference (MAD) in bone age estimation between the model and reviewers was 0.15 years on internal validation. In the test set, the MAD between the model and the five experts ranged from 0.19 to 0.30 years. Compared with the reference standard, the MAD was 0.22 years. Interobserver agreement was excellent among reviewers (ICC: 0.99) and between the model and the reviewers (ICC: 0.98). In the subpart analysis, the olecranon apophysis exhibited the highest accuracy (74.5%), followed by the trochlea (73.7%), lateral condyle (73.7%), and radial epiphysis (63.1%). Assessment of rapidly advancing bone age during puberty on elbow radiographs using our deep neural network model was similar to that of experts. • Bone age during puberty is particularly important for patients with scoliosis or limb-length discrepancy to determine the phase of the disease, which influences the timing and method of surgery. • The commonly used hand radiographs-based methods have limitations in assessing bone age during puberty due to the less prominent morphological changes of the hand and wrist bones in this period. • A deep neural network model trained with elbow radiographs exhibited similar performance to human experts on estimating rapidly advancing bone age during puberty.

Introduction : Acute cerebrovascular accidents (CVA), including ischemic stroke and intracranial hemorrhage, remain among the leading causes of mortality and long-term disability worldwide. The substantial workload placed on radiologists, combined with the necessity for prompt decision-making under time pressure, underscores the importance of integrating artificial intelligence (AI) technologies into the diagnostic process. Objectives : To assess the diagnostic performance of an artificial intelligence model developed for the detection of acute ischemic stroke and intracranial hemorrhage on non-contrast brain computed tomography (CT). Materials and Methods : This paper represents the results of a retrospective study. The test dataset comprised 263 anonymized non-contrast brain CT examinations of patients aged over 18 years, performed under clinical suspicion of acute cerebrovascular accident. Ground truth was established by two independent radiologists. The performance of the AI model was evaluated against the ground truth dataset using sensitivity, specificity, accuracy, and ROC AUC metrics. In addition, the accuracy of lesion localization and segmentation was analyzed. Results : For ischemic stroke detection, the AI model achieved a sensitivity of 0.85, specificity of 0.82, and overall accuracy of 0.83 (ROC AUC=0.84). For intracranial hemorrhage detection, sensitivity was 0.82, specificity 0.81, and accuracy 0.81 (ROC AUC=0.81). Agreement between radiologists and the model’s proposed lesion contours was observed in 94.2% of cases, while concordance on lesion volume estimation reached 95.7%. Discussion : The findings demonstrate that the AI model provides high diagnostic accuracy and may serve as a valuable tool for clinical decision-making. Nonetheless, the limited positive predictive value highlights the necessity of employing the model in conjunction with clinical context and expert interpretation. Conclusion : The «Brain CT” AI model demonstrated strong potential for automated detection of ischemic stroke and intracranial hemorrhage. Its implementation could contribute to reducing the workload of radiologists and improving diagnostic accuracy in routine practice, contingent upon further validation and model retraining on larger and more diverse datasets.

IntroductionA deep learning-based automatic bone age identification system (ABAIs) was introduced in medical imaging. This ABAIs enhanced accurate, consistent, and timely clinical diagnostics and enlightened research fields of deep learning and artificial intelligence (AI) in medical imaging.AimThe goal of this study was to use the Deep Neural Network (DNN) model to assess bone age in months based on a database of pediatric left-hand radiographs.MethodsThe Inception Resnet V2 model with a Global Average Pooling layer to connect to a single fully connected layer with one neuron using the Rectified Linear Unit (ReLU) activation function consisted of the DNN model for bone age assessment (BAA) in this study. The medical data in each case contained posterior view of X-ray image of left hand, information of age, gender and weight, and clinical skeletal bone assessment.ResultsA database consisting of 8,061 hand radiographs with their gender and age (0–18 years) as the reference standard was used. The DNN model’s accuracies on the testing set were 77.4%, 95.3%, 99.1% and 99.7% within 0.5, 1, 1.5 and 2 years of the ground truth respectively. The MAE for the study subjects was 0.33 and 0.25 year for male and female models, respectively.ConclusionIn this study, Inception Resnet V2 model was used for automatic interpretation of bone age. The convolutional neural network based on feature extraction has good performance in the bone age regression model, and further improves the accuracy and efficiency of image-based bone age evaluation. This system helps to greatly reduce the burden on clinical personnel.

Predicting Caregiver Burden in Multiple Myeloma: Insights from the Carmma Study Using Machine Learning Models

There have been several attempts to develop machine learning (ML) models to identify human infecting viruses from their genomic sequences, with varying degrees of success. Direct comparison between models is problematic, because these models are typically trained and evaluated on different datasets with alternative data splitting schemes, features, and model performance metrics. In this paper we present a standardized dataset of mammal infecting and non-infecting viral pathogens, refined from the previous work of Mollentze et al. to include the latest literature evidence, roughly doubling the number of curated host-virus records available to the community, and new host target labels, primate and mammal. The new host labels were included for several reasons, including previous reports that classification performance is better at broader taxonomic ranks and the idea that there may be more data for primate infection that might serve as a suitable proxy for zoonotic potential and avoidance of false positives for human infection due to absence of evidence. On this dataset, we report the performance of eight machine learning models for predicting mammal-infecting viruses from their genomic sequences. We find that randomly assigning cases in our improved dataset to training/testing sets, when compared to the original assignments into training/testing in Mollentze et al., increases the overall average ROC AUC of prediction of human infection from 0.663 ± 0.070 to 0.784 ± 0.013, consistent with the reduction in phylogenetic distance between train and test sets (relative entropy change from 3.00 to 0.08). The broadest host category of mammal infection can be predicted most reliably at 0.850 ± 0.020. We share our improved dataset and code to enable standardized comparisons of machine learning methods to predict human host infections. Overall, we have presented preliminary evidence that classification of virus host infection is more tractable at higher taxonomic ranks, that unsurprisingly reducing the phylogenetic distance between training and test sets can improve predictive performance, that peptide kmer features appear to be harmful to out of sample model performance, and we are left with the question of whether models for virus host prediction can reasonably be expected to perform well in out of sample scenarios given the likelihood that viruses do not share a common ancestor. Consistent with this concern, when the data is resampled such that there is no overlap between viral families in training and test sets (relative entropy > 24), models perform no better than random chance at prediction of human infection regardless of whether kmers are included (ROC AUC 0.50 ± 0.08) or not (ROC AUC 0.50 ± 0.04).

There have been several attempts to develop machine learning (ML) models to identify human infecting viruses from their genomic sequences, with varying degrees of success. Direct comparison between models is problematic, because these models are typically trained and evaluated on different datasets with alternative data splitting schemes, features, and model performance metrics. In this paper we present a standardized dataset of mammal infecting and non-infecting viral pathogens, refined from the previous work of Mollentze et al. to include the latest literature evidence, roughly doubling the number of curated host-virus records available to the community, and new host target labels, primate and mammal. The new host labels were included for several reasons, including previous reports that classification performance is better at broader taxonomic ranks and the idea that there may be more data for primate infection that might serve as a suitable proxy for zoonotic potential and avoidance of false positives for human infection due to absence of evidence. On this dataset, we report the performance of eight machine learning models for predicting mammal-infecting viruses from their genomic sequences. We find that randomly assigning cases in our improved dataset to training/testing sets, when compared to the original assignments into training/testing in Mollentze et al., increases the overall average ROC AUC of prediction of human infection from 0.663 ± 0.070 to 0.784 ± 0.013, consistent with the reduction in phylogenetic distance between train and test sets (relative entropy change from 3.00 to 0.08). The broadest host category of mammal infection can be predicted most reliably at 0.850 ± 0.020. We share our improved dataset and code to enable standardized comparisons of machine learning methods to predict human host infections. Overall, we have presented preliminary evidence that classification of virus host infection is more tractable at higher taxonomic ranks, that unsurprisingly reducing the phylogenetic distance between training and test sets can improve predictive performance, that peptide kmer features appear to be harmful to out of sample model performance, and we are left with the question of whether models for virus host prediction can reasonably be expected to perform well in out of sample scenarios given the likelihood that viruses do not share a common ancestor. Consistent with this concern, when the data is resampled such that there is no overlap between viral families in training and test sets (relative entropy > 24), models perform no better than random chance at prediction of human infection regardless of whether kmers are included (ROC AUC 0.50 ± 0.08) or not (ROC AUC 0.50 ± 0.04).

In this study, we investigate the performance of computer vision AI algorithms in predicting patient disposition from the emergency department (ED) using short video clips. Clinicians often use “eye-balling” or clinical gestalt to aid in triage, based on brief observations. We hypothesize that AI can similarly use patient appearance for disposition prediction. Data were collected from adult patients at an academic ED, with mobile phone videos capturing patients performing simple tasks. Our AI algorithm, using video alone, showed better performance in predicting hospital admissions (AUROC = 0.693 [95% CI 0.689, 0.696]) compared to models using triage clinical data (AUROC = 0.678 [95% CI 0.668, 0.687]). Combining video and triage data achieved the highest predictive performance (AUROC = 0.714 [95% CI 0.709, 0.719]). This study demonstrates the potential of video AI algorithms to support ED triage and alleviate healthcare capacity strains during periods of high demand.

PD50-06 PERFORMANCE OF A PROSTATE CANCER DETECTION SYSTEM ON IHC-REQUESTED CASES

Introduction: In digital pathology, large amounts of time and money are spent annotating whole slide images (WSIs). By using an AI algorithm trained on thousands of WSIs, it is possible to remove the process of annotating from the workflow whilst simultaneously finding potential imaging features that correlate with drug response, molecular phenotypes, histologic features (e.g. Tertiary Lymphoid Structures), inflammation at the tumor-stroma interface, artifacts, etc. Method: A pipeline was constructed for ingesting WSI from the set of diagnostic slides available in the cancer genome archive (TCGA). The WSIs were masked for tissue areas and divided into patches extracted at multiple magnification levels. These patches were used to train a U-Net (initialised with a pre-trained ResNet34) to complete a self-supervised in-painting task and evaluated with perceptual loss using VGG16. After training, embeddings were extracted from the bottleneck layer of the U-Net, and these were evaluated for their capacity to cluster according to tissue type and magnification level. Results: The trained embeddings showed strong clustering by tissue type, magnification level, and separation of artefacts. The results were qualitatively evaluated by pathologist with consensus and exceeded the performance of a baseline pre-trained ImageNet model. Conclusion: These results highlight a novel methodology for AI algorithm development that removes the need for numerous pathologists to annotate pathology images by hand and converts their role to reviewing and quality-checking images. AI algorithms based on this approach can have a significant impact in the pathologic evaluation of tissue for oncology clinical trials by extracting a richer set of features than glass microscopy alone would permit. Furthermore, self-supervised methods, such as in-painting allow for the training of embeddings that encapsulate a richer understanding of the data and are consequently more repurposable than supervised/labelled approaches. Citation Format: Jason Hipp, Mona Xu, Lucas Bordeaux, Feng Gu, Carlos Pedrinaci, Khan Baykaner. Unsupervised learning of image embeddings enables new opportunities to extract novel information from digital pathology H&E images [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-076.