Dual Graph Convolutional Networks for Chest X-ray Report Generation

The coronavirus disease 2019 (COVID-19) led to a dramatic increase in the number of cases of patients with pneumonia worldwide. In this study, we aimed to develop an AI-assisted multistrategy image enhancement technique for chest X-ray (CXR) images to improve the accuracy of COVID-19 classification. Our new classification strategy consisted of 3 parts. First, the improved U-Net model with a variational encoder segmented the lung region in the CXR images processed by histogram equalization. Second, the residual net (ResNet) model with multidilated-rate convolution layers was used to suppress the bone signals in the 217 lung-only CXR images. A total of 80% of the available data were allocated for training and validation. The other 20% of the remaining data were used for testing. The enhanced CXR images containing only soft tissue information were obtained. Third, the neural network model with a residual cascade was used for the super-resolution reconstruction of low-resolution bone-suppressed CXR images. The training and testing data consisted of 1,200 and 100 CXR images, respectively. To evaluate the new strategy, improved visual geometry group (VGG)-16 and ResNet-18 models were used for the COVID-19 classification task of 2,767 CXR images. The accuracy of the multistrategy enhanced CXR images was verified through comparative experiments with various enhancement images. In terms of quantitative verification, 8-fold cross-validation was performed on the bone suppression model. In terms of evaluating the COVID-19 classification, the CXR images obtained by the improved method were used to train 2 classification models. Compared with other methods, the CXR images obtained based on the proposed model had better performance in the metrics of peak signal-to-noise ratio and root mean square error. The super-resolution CXR images of bone suppression obtained based on the neural network model were also anatomically close to the real CXR images. Compared with the initial CXR images, the classification accuracy rates of the internal and external testing data on the VGG-16 model increased by 5.09% and 12.81%, respectively, while the values increased by 3.51% and 18.20%, respectively, for the ResNet-18 model. The numerical results were better than those of the single-enhancement, double-enhancement, and no-enhancement CXR images. The multistrategy enhanced CXR images can help to classify COVID-19 more accurately than the other existing methods.

We investigated the application of multiresolution global and local texture features to reduce false-positive detection in a computerized mass detection program. One hundred and sixty-eight digitized mammograms were randomly and equally divided into training and test groups. From these mammograms, two datasets were formed. The first dataset (manual) contained four regions of interest (ROIs) selected manually from each of the mammograms. One of the four ROIs contained a biopsy-proven mass and the other three contained normal parenchyma, including dense, mixed dense/fatty, and fatty tissues. The second dataset (hybrid) contained the manually extracted mass ROIs, along with normal tissue ROIs extracted by an automated Density-Weighted Contrast Enhancement (DWCE) algorithm as false-positive detections. A wavelet transform was used to decompose an ROI into several scales. Global texture features were derived from the low-pass coefficients in the wavelet transformed images. Local texture features were calculated from the suspicious object and the peripheral subregions. Linear discriminant models using effective features selected from the global, local, or combined feature spaces were established to maximize the separation between masses and normal tissue. Receiver Operating Characteristic (ROC) analysis was conducted to evaluate the classifier performance. The classification accuracy using global features were comparable to that using local features. With both global and local features, the average area, Az, under the test ROC curve, reached 0.92 for the manual dataset and 0.96 for the hybrid dataset, demonstrating statistically significant improvement over those obtained with global or local features alone. The results indicated the effectiveness of the combined global and local features in the classification of masses and normal tissue for false-positive reduction.

In this study, we investigated how shapes are classified based on local and global features by four representative convolutional neural networks (CNNs), i.e., AlexNet, VGG, ResNet and Inception. While the local features are based on simple components, such as orientation of line segment, the global features are based on the whole object, such as whether an object has a hole. For example, solid triangles and solid squares are differentiated by local features, solid circles and rings are differentiated by a global feature. Two sets of experiments were performed in this research. In the first experiment, we examined how the four CNNs pre-trained on ImageNet (with transfer learning) learned to differentiate the regular shapes (equilateral-triangles, squares, circles and rings). Our results showed that the pre-trained CNNs exhibited faster learning rates in the tasks discriminating the local features than in the tasks discriminating the global feature. However, the transfer learning of discriminating the global feature in regular shapes were better generalized to irregular shapes than the transfer learning of discriminating local features. In the second experiment, the CNNs were trained from scratch (with random weights initialization) to discriminate local and global features in regular and irregular shapes. Different from the transfer learning, the CNNs exhibited faster learning rates in discriminating the global feature than the local features. Similar to transfer learning, the CNNs exhibited excellent learning generalization to discriminating the global feature of irregular shapes, but poor learning generalization to discriminating the local features in the irregular shapes. The overarching goal of this research is to create a paradigm and benchmark to directly compare how the CNNs and primate visual systems process geometrical invariants. In contrast to the ImageNet approach which employs natural images to train CNNs, we employed the “ShapeNet” approach which features geometrical shapes with well-defined properties. The ShapeNet approach will not only help elucidate the strengths and limitations of CNN computation, but also provide insights into visual information processing in the primates.

One of the main issues of face recognition is to extract features from face images, which include both local and global features. We present a novel method to perform feature fusion at the feature level. First, global features are extracted by principal component analysis (PCA), while local features are obtained by active appearance model (AAM) and Gabor wavelet transform (GWT). Second, two types of features are fused by weighted concatenation. Finally, Euclidean and feature distances of fused features are applied to carry out a nearest neighbor classifier. The method is evaluated by the recognition rates and computation cost over two face image databases [AR (created by A. Martinez and R. Benavente) and SJTU-IPPR (Shanghai JiaoTong University-Institute of Image Processing and Pattern Recognition)]. Compared with PCA and elastic bunch graph matching (EBGM), the presented method is more effective. Though the recognition rate of the presented method is not as good as nonlinear feature combination (NFC), low computation cost is its superiority. In addition, experimental results show that the novel method is robust to variations over time, expression, illumination, and pose to a certain extent.

In this paper, we proposed an image hashing using both global and local features. Global features are determined using tensor decomposition and local features are takeout from salient regions. SLIC algorithm are used to find out the salient area. The hash are constructed from global and local features. The test results on large dataset specify that the suggested method is vigorous to content-preserving operations (CPOs) and has good distinction. In addition, the method can also localize the tampering reigns. In this method, there are two phase. In the first phase,” the different image pairs” and “similar and tampered pairs” are segregated using threshold T1. In second phase, tampering localization and separation of. “the similar (authentic) image pairs” and “tampered image pairs” are carried out with another threshold T2. The receiver operating characteristics (ROC) indicate that this technique is superior than others.

This study analyzes the effectiveness of the global (the whole face) and local (regions of eyes, nose, and mouth) features for face recognition. Features describing human faces are encoded in local ternary patterns. The two-class support vector machine is used as the supervised learning algorithm for training recognition models. In the recognition process, recognition modes based on the global features and local features are cascaded. For identifying a face image, the local features are used iteratively for filtering out candidates that can not be clearly identified by the global features, until the one with highest possibility is concluded. The experimental results show that cascading the recognition models of global and local features obtains better classification accuracy than the single classification process.

Development of a computer-aided tool for detection of COVID-19 pneumonia from CXR images using machine learning algorithm

Sentence similarity is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. However, a variety of linguistic expressions and ambiguities of words in sentences make it difficult to measure sentence similarity. Many studies show that using local features or global features of a sentence will produce satisfactory sentence representations that can be utilized to measure sentence similarity. Local features reflect the relationships of adjacent words for each sentence and the sequence information of a sentence are usually expressed by global features. However, local features lack abilities to capture sequence information while a small amount of extracted global features is not enough to produce sentence representations with good qualities. In this paper, we propose A Hybrid Model combining Local and Global features into a Siamese Network (HM-LGSN) for sentence similarity calculation. We first propose a new convolution neural network architecture called group convolution neural network to extract the most representative local features (or word semantic features). Then we combine these new features with pre-trained embeddings of words as input to the Bidirectional Gated Recurrent Units to extract global features of sentences. Finally, we select the global features to form sentence representations and calculate sentence similarity through Manhattan distance. The experimental results on SICK, MSRVID, STS-B datasets show that the accuracy of our proposed model is significantly improved by combining local features and global features.

This study proposes a new vehicle type recognition method that combines global and local features via a two‐stage classification. To extract the continuous and complete global feature, an improved Canny edge detection algorithm with smooth filtering and non‐maxima suppression abilities is proposed. To extract the local feature from four partitioned key patches, a set of Gabor wavelet kernels with five scales and eight orientations is introduced. Different from the single‐stage classification, where all features are incorporated into one classifier simultaneously, the proposed two‐stage classification strategy leverages two types of features and classifiers. In the first stage, the preliminary recognition of large vehicle or small vehicle is conducted based on the global feature via a k‐nearest neighbor probability classifier. Based on the preliminary result, the specific recognition of bus, truck, van, or sedan is achieved based on the local feature via a discriminative sparse representation based classifier. We experiment with the proposed method on the public and established datasets involving various challenging cases, such as partial occlusion, poor illumination, and scale variation. Experimental results show that the proposed method outperforms existing state‐of‐the‐art methods.

Radiologists' eye movements during medical image interpretation reflect their perceptual-cognitive processes of diagnostic decisions. The eye movement data can be modeled to represent clinically relevant regions in a medical image and potentially integrated into an artificial intelligence (AI) system for automatic diagnosis in medical imaging. In this article, we first conduct a large-scale eye-tracking study involving 13 radiologists interpreting 191 chest X-ray (CXR) images, establishing a best-of-its-kind CXR visual saliency benchmark. We then perform analysis to quantify the reliability and clinical relevance of saliency maps (SMs) generated for CXR images. We develop CXR image saliency prediction method (CXRSalNet), a novel saliency prediction model that leverages radiologists' gaze information to optimize the use of unlabeled CXR images, enhancing training and mitigating data scarcity. We also demonstrate the application of our CXR saliency model in enhancing the performance of AI-powered diagnostic imaging systems.

Tuberculosis (TB) is one of the most common diseases in developing countries such as Indonesia. TB disease oftenly attacks the human lungs due to its high oxygen pressure. The basic characteristics of TB-infected lungs are in the form of wounds, which change its texture and shape. The common detection method is by comparing the X-ray thoracic image with the tuberculin skin test (TST) and fast acid staining methods. This diagnosis method is often constrained by the availability of radiologists in the health facilities. To overcome such situation, a computer-aided diagnosis (CAD) tool is required to assist the doctor read the X-ray images easily. In this paper, a complementary detection method is proposed by using a chest X-ray (CXR) image to assist the diagnosis process. In current method, the image is segmented to reduce unnecessary parts prior to preprocessing step. We use a texture-based feature approach for detection purpose because most of the common examination methods for abnormality detection on CXRs is by analyzing the change of textural (content) CXR. In our texture approach, we use texture-based statistical properties on the histogram’s intensity. Some features that are used in this approach including Mean, Standard Deviation, Smoothness, Entropy, Root Mean Square (RMS), Variance, Kurtosis, Skewness, and Inverse Difference Moment (IDM). As a result of simulation, it indicates that CXR images can be well segmented to localize the focus area, allowing to the features can be extracted from the CXR image for further detection process. The obtained accuracy is about 93.94% and 92.86%, respectively for Shenzhen-Chest X-ray dataset and Montgomery County-Chest X-ray dataset.

As a sub-direction of image retrieval, person re-identification (Re-ID) is usually used to solve the security problem of cross camera tracking and monitoring. A growing number of shopping centers have recently attempted to apply Re-ID technology. One of the development trends of related algorithms is using an attention mechanism to capture global and local features. We notice that these algorithms have apparent limitations. They only focus on the most salient features without considering certain detailed features. People’s clothes, bags and even shoes are of great help to distinguish pedestrians. We notice that global features usually cover these important local features. Therefore, we propose a dual branch network based on a multi-scale attention mechanism. This network can capture apparent global features and inconspicuous local features of pedestrian images. Specifically, we design a dual branch attention network (DBA-Net) for better performance. These two branches can optimize the extracted features of different depths at the same time. We also design an effective block (called channel, position and spatial-wise attention (CPSA)), which can capture key fine-grained information, such as bags and shoes. Furthermore, based on ID loss, we use complementary triplet loss and adaptive weighted rank list loss (WRLL) on each branch during the training process. DBA-Net can not only learn semantic context information of the channel, position, and spatial dimensions but can integrate detailed semantic information by learning the dependency relationships between features. Extensive experiments on three widely used open-source datasets proved that DBA-Net clearly yielded overall state-of-the-art performance. Particularly on the CUHK03 dataset, the mean average precision (mAP) of DBA-Net achieved 83.2%.

In recent years, significant progress has been made in crowd counting with the development of convolutional neural networks (CNNs). However, while CNNs excel at extracting local features, their limited receptive fields restrict their ability to model global context. In contrast, Transformers can effectively model long-distance dependencies, but are inferior to CNN in capturing local detail features. Local details and global context information are crucial to handle large-scale changes in crowds. To address this problem, we propose a novel dual backbone network (DBNet) that integrates CNN and Transformer architectures, aiming to capture and aggregate both global semantic information and local detail features at multiple levels. Specifically, the dual backbone structure is designed to extract fine-grained local features while modeling long-range contextual relationships. Additionally, we introduce a multi-attention hierarchical fusion module (MAHF) that integrates global and local features from the two backbones while suppressing background noise. To further enhance accuracy in the presence of multi-scale variations, we also employ a Feature Enhancement Module (FEM), which enables the network to more effectively identify edge features and facilitates more effective multi-scale feature modeling. Extensive experiments on ShanghaiTech, UCF-QNRF, and JHU-Crowd++ datasets demonstrate that DBNet achieves competitive performance, validating the effectiveness of our approach.

Novel coronavirus disease (COVID-19) is an extremely contagious and quickly spreading coronavirus infestation. Severe acute respiratory syndrome (SARS) and Middle East respiratory syndrome (MERS), which outbreak in 2002 and 2011, and the current COVID-19 pandemic are all from the same family of coronavirus. This work aims to classify COVID-19, SARS, and MERS chest X-ray (CXR) images using deep convolutional neural networks (CNNs). To the best of our knowledge, this classification scheme has never been investigated in the literature. A unique database was created, so-called QU-COVID-family, consisting of 423 COVID-19, 144 MERS, and 134 SARS CXR images. Besides, a robust COVID-19 recognition system was proposed to identify lung regions using a CNN segmentation model (U-Net), and then classify the segmented lung images as COVID-19, MERS, or SARS using a pre-trained CNN classifier. Furthermore, the Score-CAM visualization method was utilized to visualize classification output and understand the reasoning behind the decision of deep CNNs. Several deep learning classifiers were trained and tested; four outperforming algorithms were reported: SqueezeNet, ResNet18, InceptionV3, and DenseNet201. Original and preprocessed images were used individually and all together as the input(s) to the networks. Two recognition schemes were considered: plain CXR classification and segmented CXR classification. For plain CXRs, it was observed that InceptionV3 outperforms other networks with a 3-channel scheme and achieves sensitivities of 99.5%, 93.1%, and 97% for classifying COVID-19, MERS, and SARS images, respectively. In contrast, for segmented CXRs, InceptionV3 outperformed using the original CXR dataset and achieved sensitivities of 96.94%, 79.68%, and 90.26% for classifying COVID-19, MERS, and SARS images, respectively. The classification performance degrades with segmented CXRs compared to plain CXRs. However, the results are more reliable as the network learns from the main region of interest, avoiding irrelevant non-lung areas (heart, bones, or text), which was confirmed by the Score-CAM visualization. All networks showed high COVID-19 detection sensitivity (> 96%) with the segmented lung images. This indicates the unique radiographic signature of COVID-19 cases in the eyes of AI, which is often a challenging task for medical doctors.

In the last century, we have passed two severe pandemics; the 1957 influenza (Asian flu) pandemic and the 1918 influenza (Spanish flu) pandemic with a high fatality rate. In the last few months, we have been again facing a new epidemic (COVID-19), which is a frighteningly high-risk disease and is globally threatening human lives. Among all attempts and presented solutions to tackle the COVID-19, a publicly available dataset of radiological imaging using chest radiography, also called chest X-ray (CXR) images, could efficiently accelerate the detection process of patients infected with COVID-19 through presented abnormalities in their chest radiography images. In this study, we have proposed a deep neural network (DNN), namely RAM-Net, a new combination of MobileNet with Dilated Depthwise Separable Convolution (DDSC), Residual blocks, and Attention augmented convolution. The network has been learned and validated using the COVIDx dataset, one of the most popular public datasets comprising the chest X-ray (CXR) images. Using this model, we could accurately identify the positive cases of COVID-19 viral infection while a new suspicious chest X-ray image is shown to the network. Our network's overall accuracy on the COVIDx test dataset was 95.33%, with a sensitivity and precision of 92% and 99% for COVID-19 cases, respectively, which are the highest results on the COVIDx dataset to date, to the best of our knowledge. Finally, we performed an audit on RAM-Net based on the Grad-CAM's interpretation to demonstrate that our proposed architecture detects SARS-CoV-2 (COVID-19) viral infection by focusing on vital factors rather than relying on irrelevant information.