Siamese evolutionary masking: Enhancing the generalization of self-supervised medical image segmentation model.

目的 基于计算机的胸腔X线影像疾病检测和分类目前存在误诊率高，准确率低的问题。本文在视觉Transformer（vision Transformer，ViT）预训练模型的基础上，通过迁移学习方法，实现胸腔X线影像辅助诊断，提高诊断准确率和效率。方法 选用带有卷积神经网络（convolutional neural network，CNN）的ViT模型，其在超大规模自然图像数据集中进行了预训练；通过微调模型结构，使用预训练的ViT模型参数初始化主干网络，并迁移至胸腔X线影像数据集中再次训练，实现疾病多标签分类。结果 在IU X-Ray数据集中对ViT迁移学习前、后模型平均AUC（area under ROC curve）得分进行对比分析实验。结果表明，预训练ViT模型平均AUC得分为0.774，与不使用迁移学习相比提升了0.208。并针对模型结构和数据预处理进行了消融实验，对ViT中的注意力机制进行可视化，进一步验证了模型有效性。最后使用Chest X-Ray14和CheXpert数据集训练微调后的ViT模型，平均AUC得分为0.839和0.806，与对比方法相比分别有0.014～0.031的提升。结论 与其他方法相比，ViT模型胸腔X线影像的多标签分类精确度更高，且迁移学习可以在降低训练成本的同时提升ViT模型的分类性能和泛化性。消融实验与模型可视化表明，包含CNN结构的ViT模型能重点关注有意义的区域，高效获取胸腔X线影像的视觉特征。;Objective The chest X-ray-relevant screening and diagnostic method is essential for radiology nowadays. Most of chest X-ray images interpretation is still restricted by clinical experience and challenged for misdiagnose and missed diagnoses. To detect and identify one or more potential diseases in images automatically，it is beneficial for improving diagnostic efficiency and accuracy using computer-based technique. Compared to natural images，multiple lesions are challenged to be detected and distinguished accurately in a single image because abnormal areas have a small proportion and complex representations in chest X-ray images. Current convolutional neural network（CNN）based deep learning models have been widely used in the context of medical imaging. The structure of the CNN convolution kernel has sensitive to local detail information，and it is possible to extract richer image features. However，the convolution kernel cannot be used to get global information，and the features-extracted are restricted of redundant information like its relevance of background， muscles，and bones. The model’s performance in multi-label classification tasks are affected to a certain extent. At present，the vision Transformer（ViT）model has achieved its priorities in computer vision-related tasks. The ViT can be used to capture information simultaneously and effectively for multiple regions of the entire image. However，it is required to use large-scale dataset training to achieve good performance. Due to some factors like patient privacy and manual annotate costs，the size of the chest X-ray image data set has been limited. To reduce the model's dependence on data scale and improve the performance of multi-label classification，we develop the CNN-based ViT pre-training model in terms of the transfer learning method for diagnosis-assisted of chest X-ray image and multi-label classification. Method The CNN-based ViT model is pre-trained on a huge scale ground truth dataset，and it is used to obtain the initial parameters of the model. The model structure is fine-tuned according to the features of chest X-ray dataset. A 1×1 convolution layer is used to convert the chest X-ray images channels between 1 to 3. The number of output nodes of the linear layer in the classifier is balanced from 1 000 to the number of chest X-ray classification labels，and the Sigmoid is used as an activation function. The parameters of the backbone network are initialized in terms of the pre-trained ViT model parameters，and it is trained in the chest X-ray dataset after that to complete multi-label classification. The experiment is configured of Python3. 7 and PyTorch1. 8 to construct the model and RTX3090 GPU for training. Stochastic gradient descent（SGD）optimizer，binary cross-entropy（BCE）loss function，an initial learning rate of 1E-3，the cosine annealing learning rate decay are used. For training，each image is scaled to a size of 512×512 pixels，and a 224×224 pixels area and it is then cropped in random as the model input，and data augmentation is performed randomly by some of the flipping，perspective transformation， shearing，translation，zooming，and changing brightness. For testing，the chest X-ray image is scaled to 256×256 pixels and center crop a 224×224 area to input the trained model. Result The experiment is performed on the IU X-Ray，which is a small-scale chest X-ray dataset. This model is evaluated in quantitative using the average of area under ROC curve （AUC）scores across all classification labels. The results show that the average AUC score of the pre-trained ViT model is 0. 774. The accuracy and training efficiency of the non-pre-trained ViT model is dropped significantly. The average AUC score is reached to 0. 566 only，which is 0. 208 lower. In addition，the attention mechanism heat map is generated based on the ViT model，which can strengthen the interpretability of the model. A series of ablation experiments are carried out for data augmentation，model structure，and batch size design. The fine-tuned ViT model is trained on the Chest-Ray14 and CheXpert dataset as well. The average AUC score is reached to 0. 839 and 0. 806，which is optimized by 0. 014 and 0. 031. Conclusion A pre-trained ViT model is used for the multi-label classification of chest X-ray images via transfer learning. The experimental results illustrate that the ViT has its stronger multi-label classification performance in chest Xray images，and its attention mechanism is beneficial for lesions precision-focused like the interior of the chest cavity and the heart. Transfer learning is potential to improve the classification performance and model generalization of the ViT in small-scale datasets，and the training cost is reduced greatly. Ablation experiments demonstrate that the incorporated model of CNN and Transformer has its priority beyond single-structure model. Data enhancement and the batch size cutting can improve the performance of the model，but smaller scale of batch is still interlinked to longer training span. To improve the model's ability，we predict that future research direction can be focused on the extraction for complex disease and highlevel semantic information，such as their small lesions，disease location，and severity.

Local and global information affect cooperation in networked Prisoner’s dilemma games

Feature extraction methods in the industrial process may affect the performance of process monitoring and fault assessment. In order to reduce the influence of model preference, a novel fault assessment method integrating global and local structure information is proposed by using the strategy of parallel extraction and weighted combination. First, kernel sparse locally linear embedding and kernel principal component analysis models are constructed to handle nonlinear system states and describe the local and global information in parallel. Second, the fault assessment indicators are obtained through support vector data description models. Then, a comprehensive fault assessment index is established by the weighted fusion method, which can keep the global and local neighbor information of the original sample space, and thus more feature information is included. Finally, the feasibility and effectiveness of the proposed assessment method are illustrated by the application in the simulation process of Tennessee Eastman Process, and the results show that the fusion of two basic models can bring a model with better generalization performance than a single model.

Multi-view clustering can mine the underlying structure of multi-view data and has attracted increasing attention. Most existing multi-view clustering methods either construct the similarity matrix from the global level through self-representation learning or construct the similarity matrix from the local level through graph learning. Spectral clustering method can be used to yield the clustering results based on the similarity matrix. However, the similarity matrix that only considers global information or local information is not robust. Moreover, separating the similarity matrix learning and clustering as two steps may lead to sub-optimal clustering results. To address these issues, we propose in this paper, a multi-view subspace clustering with local and global information (MVSCLG) method. Our method combines the self-representation learning and graph learning to learn a similarity matrix with global and local information, and simultaneously utilizes the spectral decomposition and the spectral rotation techniques to yield the clustering results. We also develop an effective optimization algorithm to solve the resulting optimization problem. The effectiveness and superiority of this method are verified on four multi-view benchmark data sets.

Most works on financial forecasting use information directly associated with individual companies (e.g., stock prices, news on the company) to predict stock returns for trading. We refer to such company-specific information as local information. Stock returns may also be influenced by global information (e.g., news on the economy in general), and inter-company relationships. Capturing such diverse information is challenging due to the low signal-to-noise ratios, different time-scales, sparsity and distributions of global and local information from different modalities. In this paper, we propose a model that captures both global and local multimodal information for investment and risk management-related forecasting tasks. Our proposed Guided Attention Multimodal Multitask Network (GAME) model addresses these challenges by using novel attention modules to guide learning with global and local information from different modalities and dynamic inter-company relationship networks. Our extensive experiments show that GAME outperforms other state-of-the-art models in several forecasting tasks and important real-world application case studies.

Vibration-based damage detection methods have inherent uncertainties due to the perturbation of measurement noise, modeling errors, and environmental changes, such as variations in temperature. However, other measured local information, including the static strain and deflection measured by the structural health monitoring system, have not been taken full advantage of. The integration of such global and local information has the potential to lead to more accurate structural damage detection. This paper proposes a method for integrating the global and local information for structural damage detection based on Bayesian theory. First, the Bayesian probability model associated with natural frequencies, displacement mode shapes, and strain modes is developed. In this model, the local strain information is also used as an input. Second, to reduce the model's computational cost in complex structures, the strain energy damage index is employed to determine the potential damage ranges. Finally, the exact damage elements are detected by a proposed sequence elimination method. Numerical simulations on a 14-bay planar rigid structure and the experiment on a 20-bay rigid truss are carried out to demonstrate the effectiveness of the proposed method while considering model uncertainty and measurement noise. The results show that the proposed damage detection method can increase the model's identification accuracy and decrease its misjudgment rate, compared with the method that only uses global information.

High annotation costs are a substantial bottleneck in applying modern deep learning architectures to clinically relevant medical use cases, substantiating the need for novel algorithms to learn from unlabeled data. In this work, we propose ContIG, a self-supervised method that can learn from large datasets of unlabeled medical images and genetic data. Our approach aligns images and several genetic modalities in the feature space using a contrastive loss. We design our method to integrate multiple modalities of each individual person in the same model end-to-end, even when the available modalities vary across individuals. Our procedure outperforms state-of-the-art self-supervised methods on all evaluated downstream benchmark tasks. We also adapt gradient-based explainability algorithms to better understand the learned cross-modal associations between the images and genetic modalities. Finally, we perform genome-wide association studies on the features learned by our models, uncovering interesting relationships between images and genetic data. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Source code at: https://github.com/HealthML/ContIG

Lumen segmentation of the carotid artery is an important preprocessing step with clinical application, because it facilitates subsequent analysis, including stenosis grading, the detection and quantification of plaque components in the vessel wall. But it is a challenging task owing to the low and varying contrast between the surrounding tissues. Plus, intensity inhomogeneity often occurs in medical images from different modalities, which causes undesirable side effects when the traditional Chan-Vese (CV) model is applied. In order to overcome the intensity inhomogeneity problem we improve the CV model by adding a local fitting term, which incorporates both local information and global information. We also introduce a penalizing energy to avoid the time-consuming re-initialization procedure in traditional level set method.We propose and validate a semi-automatic method for lumen segmentation of carotid artery in computed tomography angiography(CTA). First we manually select a seedpoint on the lumen area, then the segmentation is automatically obtained using a level set. By combining both global and local statistical information, we manage to overcome the inhomogeneous intensity distribution in the CTA images and provide more satisfying segmentation result than CV model. Experiments on carotid artery CTA data have demonstrated the efficiency and accuracy of our model, in addition our model is also less sensitive to the location of the initial contour. Therefore our method has the potential to replace the manual procedure of lumen segmentation in CTA, which is of great value for doctors with clinical applications.

Deep learning, in recent times, has made remarkable strides when it comes to impressive performance for many tasks, including medical image processing. One of the contributing factors to these advancements is the emergence of large medical image datasets. However, it is exceedingly expensive and time-consuming to construct a large and trustworthy medical dataset; hence, there has been multiple research leveraging medical reports to automatically extract labels for data. The majority of this labor, however, is performed in English. In this work, we propose a data collecting and annotation pipeline that extracts information from Vietnamese radiology reports to provide accurate labels for chest X-ray (CXR) images. This can benefit Vietnamese radiologists and clinicians by annotating data that closely match their endemic diagnosis categories which may vary from country to country. To assess the efficacy of the proposed labeling technique, we built a CXR dataset containing 9,752 studies and evaluated our pipeline using a subset of this dataset. With an F1-score of at least 0.9923, the evaluation demonstrates that our labeling tool performs precisely and consistently across all classes. After building the dataset, we train deep learning models that leverage knowledge transferred from large public CXR datasets. We employ a variety of loss functions to overcome the curse of imbalanced multi-label datasets and conduct experiments with various model architectures to select the one that delivers the best performance. Our best model (CheXpert-pretrained EfficientNet-B2) yields an F1-score of 0.6989 (95% CI 0.6740, 0.7240), AUC of 0.7912, sensitivity of 0.7064 and specificity of 0.8760 for the abnormal diagnosis in general. Finally, we demonstrate that our coarse classification (based on five specific locations of abnormalities) yields comparable results to fine classification (twelve pathologies) on the benchmark CheXpert dataset for general anomaly detection while delivering better performance in terms of the average performance of all classes.

Objective. Although convolutional neural networks (CNN) and Transformers have performed well in many medical image segmentation tasks, they rely on large amounts of labeled data for training. The annotation of medical image data is expensive and time-consuming, so it is common to use semi-supervised learning methods that use a small amount of labeled data and a large amount of unlabeled data to improve the performance of medical imaging segmentation. Approach. This work aims to enhance the segmentation performance of medical images using a triple-teacher cross-learning semi-supervised medical image segmentation with shape perception and multi-scale consistency regularization. To effectively leverage the information from unlabeled data, we design a multi-scale semi-supervised method for three-teacher cross-learning based on shape perception, called Semi-TMS. The three teacher models engage in cross-learning with each other, where Teacher A and Teacher C utilize a CNN architecture, while Teacher B employs a transformer model. The cross-learning module consisting of Teacher A and Teacher C captures local and global information, generates pseudo-labels, and performs cross-learning using prediction results. Multi-scale consistency regularization is applied separately to the CNN and Transformer to improve accuracy. Furthermore, the low uncertainty output probabilities from Teacher A or Teacher C are utilized as input to Teacher B, enhancing the utilization of prior knowledge and overall segmentation robustness. Main results. Experimental evaluations on two public datasets demonstrate that the proposed method outperforms some existing semi-segmentation models, implicitly capturing shape information and effectively improving the utilization and accuracy of unlabeled data through multi-scale consistency. Significance. With the widespread utilization of medical imaging in clinical diagnosis, our method is expected to be a potential auxiliary tool, assisting clinicians and medical researchers in their diagnoses.

ABSTRACTSemantic segmentation is a key task in computer vision, with medical image segmentation as a prominent downstream application that has seen significant advancements in recent years. However, the challenge of requiring extensive annotations in medical image segmentation remains exceedingly difficult. In addressing this issue, semi‐supervised semantic segmentation has emerged as a new approach to mitigate annotation burdens. Nonetheless, existing methods in semi‐supervised medical image segmentation still face challenges in fully exploiting unlabeled data and efficiently integrating labeled and unlabeled data. Therefore, this paper proposes a novel network model—feature similarity multilevel information fusion network (FSMIFNet). First, the feature similarity module is introduced to harness deep feature similarity among unlabeled images, predicting true label constraints and guiding segmentation features with deep feature relationships. This approach fully exploits deep feature information from unlabeled data. Second, the multilevel information fusion framework integrates labeled and unlabeled data to enhance segmentation quality in unlabeled images, ensuring consistency between original and feature maps for comprehensive optimization of detail and global information. In the ACDC dataset, our method achieves an mDice of 0.684 with 5% labeled data, 0.873 with 10%, 0.884 with 20%, and 0.897 with 50%. Experimental results demonstrate the effectiveness of FSMIFNet in semi‐supervised semantic segmentation of medical images, outperforming existing methods on public benchmark datasets. The code and models are available at https://github.com/liujiayin12/FSMIFNet.git.

The generalization challenges faced by neural methods in solving the Travelling Salesman Problem (TSP) have attracted substantial attention. Current neural approaches predominantly rely on Deep Neural Networks with global receptive fields to capture global features, often overlooking the critical role of local topological information. To address this limitation, we propose a Global &amp; Local Encoder (G&amp;L-Encoder) that efficiently integrates local and global information within the feature space. The G-Encoder, with global receptive field, takes charge of capturing global features and facilitating the fusion of global and local information, while the L-Encoder, with its local neighborhood receptive field, extracts finer-grained features specific to each node's neighborhood, serving as an essential complement to the global one. Furthermore, we propose a simple yet efficient supervised learning framework, leveraging the recursive optimality inherent in optimal solutions to dig into their implicit knowledge. Extensive experiments on the TSP demonstrate that our method significantly improves generalization performance and achieves state-of-the-art results on medium-scale TSP tasks. Our source code is available at https://github.com/bxx-seu/GL-POMO.

After continuous development, many researchers are exploring how to better utilize global and local information in single image super-resolution (SISR). Various methods based on convolutional neural network (CNN) and Transformer structures have emerged, but few studies have mentioned how to combine these two parts of information. We study the use of self-attention mechanism to integrate local and global information, aiming to make the model better balance the weights of the two parts of information. At the same time, in order to avoid the huge amount of computation brought by Transformer, we use the selective state space model VMamba to extract global information to achieve the effect of reducing computational complexity and lightweight network. Based on the above situation, we propose a High-frequency Mamba Network (HFMN) for SISR, which includes the local high-frequency extraction module Local High-Frequency Feature Block (LHFB), the global feature extraction module Mamba-Based Attention Block (MAB) based on VMamba, and the dual attention fusion module Dual-information Interactive Attention Block (DIAB). It can better incorporate local and global information and has linear complexity in the global feature extraction branch. Experiments on multiple benchmark datasets demonstrate that the network outforms recent SOTA methods in SISR while using fewer parameters. All codes are available at https://github.com/taoWuuu/HFMN .

Explainable deep learning diagnostic system for prediction of lung disease from medical images

Recent advances in deep learning have led to a promising performance in many medical image analysis tasks. As the most commonly performed radiological exam, chest radiographs are a particularly important modality for which a variety of applications have been researched. The release of multiple, large, publicly available chest X-ray datasets in recent years has encouraged research interest and boosted the number of publications. In this paper, we review all studies using deep learning on chest radiographs published before March 2021, categorizing works by task: image-level prediction (classification and regression), segmentation, localization, image generation and domain adaptation. Detailed descriptions of all publicly available datasets are included and commercial systems in the field are described. A comprehensive discussion of the current state of the art is provided, including caveats on the use of public datasets, the requirements of clinically useful systems and gaps in the current literature.