COVID-19 classification of X-ray images using deep neural networks

The immense spread of coronavirus disease 2019 (COVID-19) has left healthcare systems incapable to diagnose and test patients at the required rate. Given the effects of COVID-19 on pulmonary tissues, chest radiographic imaging has become a necessity for screening and monitoring the disease. Numerous studies have proposed Deep Learning approaches for the automatic diagnosis of COVID-19. Although these methods achieved outstanding performance in detection, they have used limited chest X-ray (CXR) repositories for evaluation, usually with a few hundred COVID-19 CXR images only. Thus, such data scarcity prevents reliable evaluation of Deep Learning models with the potential of overfitting. In addition, most studies showed no or limited capability in infection localization and severity grading of COVID-19 pneumonia. In this study, we address this urgent need by proposing a systematic and unified approach for lung segmentation and COVID-19 localization with infection quantification from CXR images. To accomplish this, we have constructed the largest benchmark dataset with 33,920 CXR images, including 11,956 COVID-19 samples, where the annotation of ground-truth lung segmentation masks is performed on CXRs by an elegant human-machine collaborative approach. An extensive set of experiments was performed using the state-of-the-art segmentation networks, U-Net, U-Net++, and Feature Pyramid Networks (FPN). The developed network, after an iterative process, reached a superior performance for lung region segmentation with Intersection over Union (IoU) of 96.11% and Dice Similarity Coefficient (DSC) of 97.99%. Furthermore, COVID-19 infections of various shapes and types were reliably localized with 83.05% IoU and 88.21% DSC. Finally, the proposed approach has achieved an outstanding COVID-19 detection performance with both sensitivity and specificity values above 99%.

Coronavirus disease-19 (COVID-19) has rapidly spread all over the world. It is found that the low sensitivity of reverse transcription-polymerase chain reaction (RT-PCR) examinations during the early stage of COVID-19 disease. Thus, efficient models are desirable for early-stage testing of COVID-19 infected patients. Chest X-ray (CXR) images of COVID-19 infected patients have shown some bilateral changes. In this paper, deep transfer learning and a deep forest-based model are proposed to diagnose COVID-19 infection from CXR images. Initially, features of X-ray images are extracted using the well-known deep transfer learning model (i.e., ResNet101), which does not require tuning many parameters compared to the deep convolutional neural network (CNN). After that, the deep forest model is utilised to predict COVID-19 infected patients. The deep forest is based upon ensemble learning and requires a small number of hyper-parameters. Additionally, the proposed model is trained on a multi-class dataset that contains four different classes as COVID-19 (+), pneumonia, tuberculosis, and healthy patients. The comparisons are drawn among the proposed deep transfer learning and deep forest-based models, the competitive models. The obtained results show that the proposed model effectively diagnoses COVID-19 infection with an accuracy of 99.4%.

The coronavirus disease 2019 (COVID-19) led to a dramatic increase in the number of cases of patients with pneumonia worldwide. In this study, we aimed to develop an AI-assisted multistrategy image enhancement technique for chest X-ray (CXR) images to improve the accuracy of COVID-19 classification. Our new classification strategy consisted of 3 parts. First, the improved U-Net model with a variational encoder segmented the lung region in the CXR images processed by histogram equalization. Second, the residual net (ResNet) model with multidilated-rate convolution layers was used to suppress the bone signals in the 217 lung-only CXR images. A total of 80% of the available data were allocated for training and validation. The other 20% of the remaining data were used for testing. The enhanced CXR images containing only soft tissue information were obtained. Third, the neural network model with a residual cascade was used for the super-resolution reconstruction of low-resolution bone-suppressed CXR images. The training and testing data consisted of 1,200 and 100 CXR images, respectively. To evaluate the new strategy, improved visual geometry group (VGG)-16 and ResNet-18 models were used for the COVID-19 classification task of 2,767 CXR images. The accuracy of the multistrategy enhanced CXR images was verified through comparative experiments with various enhancement images. In terms of quantitative verification, 8-fold cross-validation was performed on the bone suppression model. In terms of evaluating the COVID-19 classification, the CXR images obtained by the improved method were used to train 2 classification models. Compared with other methods, the CXR images obtained based on the proposed model had better performance in the metrics of peak signal-to-noise ratio and root mean square error. The super-resolution CXR images of bone suppression obtained based on the neural network model were also anatomically close to the real CXR images. Compared with the initial CXR images, the classification accuracy rates of the internal and external testing data on the VGG-16 model increased by 5.09% and 12.81%, respectively, while the values increased by 3.51% and 18.20%, respectively, for the ResNet-18 model. The numerical results were better than those of the single-enhancement, double-enhancement, and no-enhancement CXR images. The multistrategy enhanced CXR images can help to classify COVID-19 more accurately than the other existing methods.

Although the range of immune responses to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is variable, cytokine storm is observed in a subset of symptomatic individuals. To further understand the disease pathogenesis and, consequently, to develop an additional tool for clinicians to evaluate patients for presumptive intervention, we sought to compare plasma cytokine levels between a range of donor and patient samples grouped by a COVID-19 Severity Score (CSS) based on the need for hospitalization and oxygen requirement. Here we utilize a mutual information algorithm that classifies the information gain for CSS prediction provided by cytokine expression levels and clinical variables. Using this methodology, we found that a small number of clinical and cytokine expression variables are predictive of presenting COVID-19 disease severity, raising questions about the mechanism by which COVID-19 creates severe illness. The variables that were the most predictive of CSS included clinical variables such as age and abnormal chest x-ray as well as cytokines such as macrophage colony-stimulating factor, interferon-inducible protein 10, and interleukin-1 receptor antagonist. Our results suggest that SARS-CoV-2 infection causes a plethora of changes in cytokine profiles and that particularly in severely ill patients, these changes are consistent with the presence of macrophage activation syndrome and could furthermore be used as a biomarker to predict disease severity.

Novel coronavirus disease (COVID-19) is an extremely contagious and quickly spreading coronavirus infestation. Severe acute respiratory syndrome (SARS) and Middle East respiratory syndrome (MERS), which outbreak in 2002 and 2011, and the current COVID-19 pandemic are all from the same family of coronavirus. This work aims to classify COVID-19, SARS, and MERS chest X-ray (CXR) images using deep convolutional neural networks (CNNs). To the best of our knowledge, this classification scheme has never been investigated in the literature. A unique database was created, so-called QU-COVID-family, consisting of 423 COVID-19, 144 MERS, and 134 SARS CXR images. Besides, a robust COVID-19 recognition system was proposed to identify lung regions using a CNN segmentation model (U-Net), and then classify the segmented lung images as COVID-19, MERS, or SARS using a pre-trained CNN classifier. Furthermore, the Score-CAM visualization method was utilized to visualize classification output and understand the reasoning behind the decision of deep CNNs. Several deep learning classifiers were trained and tested; four outperforming algorithms were reported: SqueezeNet, ResNet18, InceptionV3, and DenseNet201. Original and preprocessed images were used individually and all together as the input(s) to the networks. Two recognition schemes were considered: plain CXR classification and segmented CXR classification. For plain CXRs, it was observed that InceptionV3 outperforms other networks with a 3-channel scheme and achieves sensitivities of 99.5%, 93.1%, and 97% for classifying COVID-19, MERS, and SARS images, respectively. In contrast, for segmented CXRs, InceptionV3 outperformed using the original CXR dataset and achieved sensitivities of 96.94%, 79.68%, and 90.26% for classifying COVID-19, MERS, and SARS images, respectively. The classification performance degrades with segmented CXRs compared to plain CXRs. However, the results are more reliable as the network learns from the main region of interest, avoiding irrelevant non-lung areas (heart, bones, or text), which was confirmed by the Score-CAM visualization. All networks showed high COVID-19 detection sensitivity (> 96%) with the segmented lung images. This indicates the unique radiographic signature of COVID-19 cases in the eyes of AI, which is often a challenging task for medical doctors.

Introduction: Every area of our lives has been devastated by the worldwide Coronavirus disease 2019 (COVID-19) epidemic. However, the development of artificial intelligence has made it possible to build advanced applications that can fulfill this level of clinical accuracy. This study aimed to create a deep learning model that can detect COVID-19 from a chest image dataset of confirmed patients treated at the provincial hospital in Aceh. Methods: Eight hundred confirmed COVID-19 patients' chest X-ray photos were gathered locally from Dr. Zainoel Abidin General Hospital, Banda Aceh. Performance was evaluated in several ways. First, the dataset was used for training and testing. Second, the data was used to train and test the model. VGG16 is a robust network adapted to an enhanced dataset constructed from a confirmed COVID-19 chest X-ray pool. To artificially produce a huge number of chest X-ray pictures, this study used data augmentation techniques such as random rotation at an angle between 10 and 10°, random noise, and horizontal flips. Results: The experimental results were encouraging: the proposed models classified chest X-ray pictures as normal or COVID-19 with an accuracy of 97.20% for Resnet50, 98.10% for InceptionV3, and 98.30% for VGG16. The results showed the outstanding performance of straightforward COVID-19 diagnosis with the classification of COVID-19 severity, such as mild, severe, and very severe. Conclusion: These made it possible to automate the X-ray image interpretation process accurately and could also be applied when materials and reverse transcription polymerase chain reaction (RT-PCR) tests are scarce.

.Purpose: Coronavirus disease 2019 (COVID-19) is a new infection that has spread worldwide and with no automatic model to reliably detect its presence from images. We aim to investigate the potential of deep transfer learning to predict COVID-19 infection using chest computed tomography (CT) and x-ray images.Approach: Regions of interest (ROI) corresponding to ground-glass opacities (GGO), consolidations, and pleural effusions were labeled in 100 axial lung CT images from 60 COVID-19-infected subjects. These segmented regions were then employed as an additional input to six deep convolutional neural network (CNN) architectures (AlexNet, DenseNet, GoogleNet, NASNet-Mobile, ResNet18, and DarkNet), pretrained on natural images, to differentiate between COVID-19 and normal CT images. We also explored the model’s ability to classify x-ray images as COVID-19, non-COVID-19 pneumonia, or normal. Performance on test images was measured with global accuracy and area under the receiver operating characteristic curve (AUC).Results: When using raw CT images as input to the tested models, the highest accuracy of 82% and AUC of 88.16% is achieved. Incorporating the three ROIs as an additional model inputs further boosts performance to an accuracy of 82.30% and an AUC of 90.10% (DarkNet). For x-ray images, we obtained an outstanding AUC of 97% for classifying COVID-19 versus normal versus other. Combing chest CT and x-ray images, DarkNet architecture achieves the highest accuracy of 99.09% and AUC of 99.89% in classifying COVID-19 from non-COVID-19. Our results confirm the ability of deep CNNs with transfer learning to predict COVID-19 in both chest CT and x-ray images.Conclusions: The proposed method could help radiologists increase the accuracy of their diagnosis and increase efficiency in COVID-19 management.

Cardiac Surgery-Enhanced Recovery Programs Modified for COVID-19: Key Steps to Preserve Resources, Manage Caseload Backlog, and Improve Patient Outcomes

PurposeThis research was designed to investigate the application of artificial intelligence (AI) in the rapid and accurate diagnosis of coronavirus disease 2019 (COVID-19) using digital chest X-ray images, and to develop a robust computer-aided application for the automatic classification of COVID-19 pneumonia from other pneumonia and normal images.Materials and MethodsA total of 1100 chest X-ray images were randomly selected from three different open sources, containing 300 X-ray images of confirmed COVID-19 patients, 400 images of other pneumonia patients, and 400 normal X-ray images. In this study, a classical machine learning approach was employed. The model was built using the support vector machine (SVM) classifier algorithm. The SVM was trained by 630 features obtained from the HOG descriptor, which was quantized into 30 orientation bins in the range between 0 and 360. The model was validated using a 10-fold cross-validation method. The performance of the model was evaluated using appropriate classification metrics, including sensitivity, specificity, area under the curve, positive predictive value, negative predictive value, kappa, and accuracy.ResultsThe multi-level classification model was able to distinguish COVID-19 patients with a sensitivity of 97.92% and specificity of 98.91%, for the internal testing or cross-validation. For the independent external testing, the model showed sensitivity of 95% and specificity of 98.13%, for distinguishing COVID-19 from other pneumonia and no-findings. The binary classification model was able to distinguish COVID-19 patients with a sensitivity of 99.58% and specificity of 99.69%, for the internal testing. For the independent external testing, the model showed a sensitivity of 98.33% and specificity of 100%, for distinguishing COVID-19 from normal images.ConclusionThe model can achieve the rapid and accurate identification of COVID-19 patients from chest X-rays with more than 97% accuracy. This high accuracy and very rapid computer-aided diagnostic approach would be very helpful to control the pandemic.

To the Editor: Chronic subdural hematomas (CSDHs) are encapsulated collections of blood breakdown products and fluid between the dura mater and the arachnoid.1 Their global incidence is estimated to be about 5/100 000/yr in patients younger than 65 yr,2 increasing to 58/100 000/yr in patients over 65 yr.1 Surgery usually allows a good outcome even in elderly. The clinical outcome of CSDH is conditioned by comorbidities including pulmonary diseases. Nowadays, we are facing a new possible condition related to coronavirus disease 2019 (COVID-19) that could affect the outcome. The spread of this infection started in 2019, December 31st when China alerted WHO about cases of unusual amount of interstitial pneumonia (IP) in Wuhan. After 7 d, on January 7th, scientists announced they had identified a new virus named COVID-19 similar to SARS-CoV. Italy is dealing with pandemic spread of COVID-19. As a matter of fact, on 19th March, it became the country with the highest number of confirmed deaths in the world. On March 23rd 2020 Italy was the world's centre of the epidemic with 69.176 confirmed cases, 6.820 deaths, and 21.137 recoveries.3 Clinically COVID-19 could lead to a severe IP and acute respiratory distress syndrome (ARDS), especially in older patients with multiple comorbidities.4 No studies about surgery in COVID-19 patients have been reported so far. We aim to describe our surgical experience with CSDH in a COVID-19 neurosurgical center. We admitted 5 patients with CSDH to the Neurosurgical Department of Brescia University Hospital between February 21, 2020 (first COVID-19 case reported in Italy) and March 23, 2020. The patients were positive for COVID-19 real-time polymerase chain reaction obtained by nasopharyngeal swab.5,6 We gathered the following data for each patient: age and sex, Cumulative Illness Rating Scale,7 antithrombotic therapy and perioperative mortality (within 30 d). All the patients had signed a consent form approved by the Local Ethic Committee regarding the use of their clinical data. We compared the mortality rate of this new cohort with the data extracted from our pre-COVID-19 CSDH case series. All the patients underwent pre-operative chest X-Ray (CXR) or computed tomography (CT) and routine blood test. An anesthesiological evaluation was performed for each patient. We cohortized our neurosurgical ward to separate COVID-19 positive from negative cases. COVID-19 positive patients started anti-retroviral therapy with lopinavir/ritonavir and hydroxychloroquine. Daily arterial blood gas analysis and CXR were performed. Surgery or endovascular treatment were indicated when the CSDH was symptomatic (presence of focal neurological deficits or mental status changes) and the maximum thickness was greater than 1 cm (Figures 1-4). One case did not present severe neurological impairment and was treated conservatively (Figure 5). No preoperative respiratory impairment was observed.FIGURE 1.: A and B, Head CT scan, axial view: pre- and postoperative CDSH. C, CXR showing bilateral and diffuse IP.FIGURE 2.: A, Head CT scan, axial view: bilateral CSDH. B, CXR showing bilateral and diffuse IP. C and D, MMA embolization procedure.FIGURE 3.: A, Head CT scan, axial view: preoperative CSDH. B, Head CT scan, axial view: postoperative CSDH with signs of recent rebleeding. C, CXR showing bilateral and IP. D, Chest CT scan, axial view, showing ground glass opacity.FIGURE 4.: A, Head CT scan, axial view: preoperative left CSDH. B, Head CT scan, axial view: postoperative CSDH with signs of recent rebleeding. C, CXR showing bilateral IP.FIGURE 5.: A, Head CT scan, axial view: left CSDH. B, normal CXR.We treated all the CSDH patients under general anesthesia. Three patients were operated (2 through craniotomy and 1 with a burr hole). Endovascular occlusion of the middle meningeal artery (MMA) was performed in 1 case (Figure 2). In the surgical cases we placed a subdural drainage for 48 h. After its removal patients underwent a CT head scan (Figures 1, 3, and 4). Afterwards the patients started low-molecular-weight heparin at prophylactic dose before mobilization. The cases are reported in the supplementary material section (Supplemental Digital Content). CSDHs mostly occur in elderly, with an average age of 63.1 Head trauma is the major risk factor, identified in less than 50% (often minor head trauma).8 Other risk factors are described like alcohol abuse, seizures, CSF shunts and coagulopathies, including therapeutic anticoagulation. CSDHs are bilateral in 20% to 25% of cases.9 At present there is no scientific experience regarding surgical outcome of COVID-19 patients. We compared the recent cohort of COVID-19 CSDH patients with our historical series. Between May 2018 and September 2019, we operated 142 patients for CSDH and we observe 5 death at our Institution. Our mortality rate was 3.7% according to the literature.10 In our recent experience, we observed 4 death in 5 COVID-19 patients suffering from CSDH. Therefore, we observed a mortality rate of 80% about 21,6 times greater than our control data. HEMATOLOGICAL DISORDERS We observed 2 cases of mild thrombocytopenia (Table). In these cases, we observed a re-bleeding (Figures 3 and 4) that led from a rapid neurological worsening to death within 5 d. This condition could be related to COVID-19 infection, as reported in literature.11 Independently from the hemorrhagic risk, Lippi et al12 showed that low platelet count is associated with increased risk of severe disease and mortality in patients with COVID-19, and this should serve as clinical indicator of worsening illness during hospitalization.12 In our series, we suppose that thrombocytopenia led to re-bleeding and was associated with poor outcome. TABLE. - Clinical Features of our Cohort Sex Age CIRS AT Side COVID-19 IP TP* Surgery Time to death(d) M 82 13 Yes Left + + No Burr-hole 14 M 86 18 No Bilateral + + No MMA embolization 10 M 77 20 No Right + + Yes Craniotomy 5 M 85 22 Yes Left + + Yes Burr-hole 5 M 78 19 Yes Left + – No NO Alive *TP (Thrombocytopenia): 100.000<PLTs<150.000All patients developed IP after surgery. Thrombocytopenia was observed in 2 cases (40%). The patients suffered rebleeding and showed a shorter time to death. CIRS (Cumulative Illness Rating Scale), AT (antithrombotic drugs), IP (Interstitial Pneumonia), COVID-19 (Coronavirus Disease 2019). INTERSTITIAL PNEUMONIA Dyspnoea and fatigue occurred in all our patients within 48 hours from surgery. CXR revealed severe IP (Figures 1-4). Ground glass opacity and bilateral patchy shadowing were observed on the chest CT (Figure 3).5,13 All our patients had COVID-19 infection without pre-operative respiratory symptoms. Immune system could be impaired following surgical procedures and this may have unmasked a subclinical infection.14,15 On the other hand no respiratory failure was observed in the conservative-treated case (Figure 5). In our experience IP occurred in all cases after treatment and worsened the outcome. This evenience represents a possible complication, often fatal, of COVID-19 infection.5 HEALTH EMERGENCY Nowadays, Italian health care system is the most afflicted in the pandemic scenario.16 The overload of the intensive care units in Lombardy, despite the efforts made, has necessarily influenced the resuscitation possibilities of elderly patients.17 Therefore, we cannot exclude that our results could be affected by the health emergency status. Seung et al. reported good bleeding control from CSDH membrane with MMA embolization, prevention of further growth of hematoma and even spontaneous resolution without surgery.18 Anyway our endovascular treated patient showed the same poor outcome of the surgical cases. Although statistical analysis of these preliminary data is not possible, COVID-19 patients appear to suffer from a negative surgical outcome. It is our belief that conservative treatment should be preferred whenever surgery could be postponed. Disclosures The authors have no personal, financial, or institutional interest in any of the drugs, materials, or devices described in this article.

The recent medical applications of deep-learning (DL) algorithms have demonstrated their clinical efficacy in improving speed and accuracy of image interpretation. If the DL algorithm achieves a performance equivalent to that achieved by physicians in chest radiography (CR) diagnoses with Coronavirus disease 2019 (COVID-19) pneumonia, the automatic interpretation of the CR with DL algorithms can significantly reduce the burden on clinicians and radiologists in sudden surges of suspected COVID-19 patients. The aim of this study was to evaluate the efficacy of the DL algorithm for detecting COVID-19 pneumonia on CR compared with formal radiology reports. This is a retrospective study of adult patients that were diagnosed as positive COVID-19 cases based on the reverse transcription polymerase chain reaction among all the patients who were admitted to five emergency departments and one community treatment center in Korea from February 18, 2020 to May 1, 2020. The CR images were evaluated with a publicly available DL algorithm. For reference, CR images without chest computed tomography (CT) scans classified as positive for COVID-19 pneumonia were used given that the radiologist identified ground-glass opacity, consolidation, or other infiltration in retrospectively reviewed CR images. Patients with evidence of pneumonia on chest CT scans were also classified as COVID-19 pneumonia positive outcomes. The overall sensitivity and specificity of the DL algorithm for detecting COVID-19 pneumonia on CR were 95.6%, and 88.7%, respectively. The area under the curve value of the DL algorithm for the detection of COVID-19 with pneumonia was 0.921. The DL algorithm demonstrated a satisfactory diagnostic performance comparable with that of formal radiology reports in the CR-based diagnosis of pneumonia in COVID-19 patients. The DL algorithm may offer fast and reliable examinations that can facilitate patient screening and isolation decisions, which can reduce the medical staff workload during COVID-19 pandemic situations.

The global pandemic of coronavirus disease 2019 (COVID-19) has resulted in an increased demand for testing, diagnosis, and treatment. Reverse transcription polymerase chain reaction (RT-PCR) is the definitive test for the diagnosis of COVID-19; however, chest X-ray radiography (CXR) is a fast, effective, and affordable test that identifies the possible COVID-19-related pneumonia. This study investigates the feasibility of using a deep learning-based decision-tree classifier for detecting COVID-19 from CXR images. The proposed classifier comprises three binary decision trees, each trained by a deep learning model with convolution neural network based on the PyTorch frame. The first decision tree classifies the CXR images as normal or abnormal. The second tree identifies the abnormal images that contain signs of tuberculosis, whereas the third does the same for COVID-19. The accuracies of the first and second decision trees are 98 and 80%, respectively, whereas the average accuracy of the third decision tree is 95%. The proposed deep learning-based decision-tree classifier may be used in pre-screening patients to conduct triage and fast-track decision making before RT-PCR results are available.

The global pandemic of coronavirus disease 2019 (COVID-19) is continuing to have a significant effect on the well-being of the global population, thus increasing the demand for rapid testing, diagnosis, and treatment. As COVID-19 can cause severe pneumonia, early diagnosis is essential for correct treatment, as well as to reduce the stress on the healthcare system. Along with COVID-19, other etiologies of pneumonia and Tuberculosis (TB) constitute additional challenges to the medical system. Pneumonia (viral as well as bacterial) kills about 2 million infants every year and is consistently estimated as one of the most important factor of childhood mortality (according to the World Health Organization). Chest X-ray (CXR) and computed tomography (CT) scans are the primary imaging modalities for diagnosing respiratory diseases. Although CT scans are the gold standard, they are more expensive, time consuming, and are associated with a small but significant dose of radiation. Hence, CXR have become more widespread as a first line investigation. In this regard, the objective of this work is to develop a new deep transfer learning pipeline, named DenResCov-19, to diagnose patients with COVID-19, pneumonia, TB or healthy based on CXR images. The pipeline consists of the existing DenseNet-121 and the ResNet-50 networks. Since the DenseNet and ResNet have orthogonal performances in some instances, in the proposed model we have created an extra layer with convolutional neural network (CNN) blocks to join these two models together to establish superior performance as compared to the two individual networks. This strategy can be applied universally in cases where two competing networks are observed. We have tested the performance of our proposed network on two-class (pneumonia and healthy), three-class (COVID-19 positive, healthy, and pneumonia), as well as four-class (COVID-19 positive, healthy, TB, and pneumonia) classification problems. We have validated that our proposed network has been able to successfully classify these lung-diseases on our four datasets and this is one of our novel findings. In particular, the AUC-ROC are 99.60, 96.51, 93.70, 96.40% and the F1 values are 98.21, 87.29, 76.09, 83.17% on our Dataset X-Ray 1, 2, 3, and 4 (DXR1, DXR2, DXR3, DXR4), respectively.

Multisystem inflammatory syndrome in children associated with SARS-CoV-2 in a solid organ transplant recipient.

The coronavirus disease 2019 (COVID-19) pandemic had a major impact on global health and was associated with millions of deaths worldwide. During the pandemic, imaging characteristics of chest X-ray (CXR) and chest computed tomography (CT) played an important role in the screening, diagnosis and monitoring the disease progression. Various studies suggested that quantitative image analysis methods including artificial intelligence and radiomics can greatly boost the value of imaging in the management of COVID-19. However, few studies have explored the use of longitudinal multi-modal medical images with varying visit intervals for outcome prediction in COVID-19 patients. This study aims to explore the potential of longitudinal multimodal radiomics in predicting the outcome of COVID-19 patients by integrating both CXR and CT images with variable visit intervals through deep learning. 2274 patients who underwent CXR and/or CT scans during disease progression were selected for this study. Of these, 946 patients were treated at the University of Pennsylvania Health System (UPHS) and the remaining 1328 patients were acquired at Stony Brook University (SBU) and curated by the Medical Imaging and Data Resource Center (MIDRC). 532 radiomic features were extracted with the Cancer Imaging Phenomics Toolkit (CaPTk) from the lung regions in CXR and CT images at all visits. We employed two commonly used deep learning algorithms to analyze the longitudinal multimodal features, and evaluated the prediction results based on the area under the receiver operating characteristic curve (AUC). Our models achieved testing AUC scores of 0.816 and 0.836, respectively, for the prediction of mortality.