Probabilistic smooth attention for deep multiple instance learning in medical imaging

Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis.

Non-negative subspace feature representation for few-shot learning in medical imaging

For a considerable amount of time, medical image processing has been an important topic of research in the field of medicine. The advent of deep learning technology has resulted in the development of revolutionary improvements in this particular sector. Because of its remarkable effectiveness in training enormous amounts of data and performing difficult tasks, deep learning has attracted a lot of attention. It has found extensive application in the analysis, diagnosis, and treatment of medical pictures. This work employs the literature review approach to examine and analyze the existing research conducted by academics on deep learning and medical imaging. It also provides a summary of the utilization of deep learning in medical imaging images and highlights the current development trend in this field.

Medical imaging is becoming indispensable for patients' healthcare. Machine learning plays an essential role in the medical imaging field, including computer-aided diagnosis, image segmentation, image registration, image fusion, image-guided therapy, image annotation, and image database retrieval. With advances in medical imaging, new imaging modalities and methodologies such as cone-beam/multislice CT, 3D ultrasound imaging, tomosynthesis, diffusion-weighted magnetic resonance imaging (MRI), positron-emission tomography (PET)/CT, electrical impedance tomography, and diffuse optical tomography, new machine-learning algorithms/applications are demanded in the medical imaging field. Because of large variations and complexityit is generally difficult to derive analytic solutions or simple equations to represent objects such as lesions and anatomy in medical images. Therefore, tasks in medical imaging require “learning from examples” for accurate representation of data and prior knowledge. Because of its essential needs, machine learning in medical imaging is one of the most promising, growing fields. The main aim of this special issue is to help advance the scientific research within the broad field of machine learning in medical imaging. The special issue was planned in conjunction with the International Workshop on Machine Learning in Medical Imaging (MLMI 2010) [1], which was the first workshop on this topic, held at the 13th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2010) in September, 2010, in Beijing, China. This special issue is one in a series of special issues of journals on this topic [2]; it focuses on major trends and challenges in this area, and it presents work aimed at identifying new cutting-edge techniques and their use in medical imaging. The quality level of the submissions for this special issue was very high. A total of 17 papers were submitted to this issue in response to the call for papers. Based on a rigorous review process, 10 papers (59%) were accepted for publication in the special issue. The special issue starts by a review of studies on a class of machine-learning techniques, called pixel/voxel-based machine learning, in medical imaging by K. Suzuki. A series of medical imaging applications of machine-learning techniques are presented. A large variety of applications are well represented here, including organ modeling by D. Wang et al. and X. Qiao and Y.-W. Chen, brain function estimation by V. Michel et al., image reconstruction by H. Shouno et al., lesion classification by P. Wighton et al., modality classification by X.-H. Han and Y.-W. Chen, lesion segmentation by M. Zortea et al., organ segmentation by S. Alzubi et al., and visualization of molecular signals by F. Mattoli et al. Also, the issue covers various biomedical imaging modalities, including MRI by D. Wang et al., CT by X. Qiao and Y.-W. Chen and H. Shouno et al., functional MRI by V. Michel et al., dermoscopy by P. Wighton et al. and M. Zortea et al., scintigraphy by X.-H. Han and Y.-W. Chen, ultrasound imaging by X.-H. Han and Y.-W. Chen, radiography by X.-H. Han and Y.-W. Chen, MR angiography by S. Alzubi et al., and microscopy by F. Mattoli et al. as well as a variety of organs, including the kidneys by D. Wang et al., liver by X. Qiao and Y.-W. Chen, brain by V. Michel et al. and F. Mattoli et al., chest by S. Alzubi et al., skin by P. Wighton et al. and M. Zortea et al., and heart by F. Mattoli et al. Various machine-learning techniques were developed/used to solve the respective problems, including structured dictionary learning by D. Wang et al., generalized N-dimensional principal component analysis by X. Qiao et al., multiclass sparse Bayesian regression by V. Michel et al., Bayesian hyperparameter inference by H. Shouno et al., supervised learning of probabilistic models based on maximum aposteriori estimation and conditional random fields by P. Wighton et al., joint kernel equal contribution in support vector classification by X.-H. Han and Y.-W. Chen, and iterative hybrid classification by M. Zortea et al. We are grateful to all authors for their excellent contributions to this special issue and to all reviewers for their reviews and constructive suggestions. We hope that this special issue will inspire further ideas for creative research, advance the field of machine learning in medical imaging, and facilitate the translation of the research from bench to bedside. Kenji Suzuki Pingkun Yan Fei Wang Dinggang Shen

Robust multiple-instance learning ensembles using random subspace instance selection

Multiple instance learning (MIL) has emerged as the best solution for whole slide image (WSI) classification. It consists of dividing each slide into patches, which are treated as a bag of instances labeled with a global label. MIL includes two main approaches: instance-based and embedding-based. In the former, each patch is classified independently, and then, the patch scores are aggregated to predict the bag label. In the latter, bag classification is performed after aggregating patch embeddings. Even if instance-based methods are naturally more interpretable, embedding-based MILs have usually been preferred in the past due to their robustness to poor feature extractors. Recently, the quality of feature embeddings has drastically increased using self-supervised learning (SSL). Nevertheless, many authors continue to endorse the superiority of embedding-based MIL. We conduct 710 experiments across 4 datasets, comparing 10 MIL strategies, 6 self-supervised methods with 4 backbones, 4 foundation models, and various pathology-adapted techniques. Furthermore, we introduce 4 instance-based MIL methods, never used before in the pathology domain. We show that with a good SSL feature extractor, simple instance-based MILs, with very few parameters, obtain similar or better performance than complex, state-of-the-art (SOTA) embedding-based MIL methods, setting new SOTA results on the BRACS and Camelyon16 datasets. As simple instance-based MIL methods are naturally more interpretable and explainable to clinicians, our results suggest that more effort should be put into well-adapted SSL methods for WSI rather than into complex embedding-based MIL methods.

Due to the trade-off between spatial resolution of the imagery and satellite revisit times, the research topics using multiresolution remote sensing data have attracted much attention. In recent years, the techniques for improving the visibility of multi-resolution imagery have been proposed, including pan-sharpening and super-resolution. However, there are relatively few studies on the techniques of improving the model performance on low-resolution imagery by referring to detailed information from the high-resolution imagery during training time. To tackle this type of task, domain adaptation has been proposed in the field of computer vision to adapt a model trained on one dataset to another with different properties. Yet, domain adaptation for multi-resolution data is difficult due to the scale variation in addition to differences from the camera sensors. In this study, we propose a new approach for multi-resolution modeling that combines the major techniques, semi-supervised domain adaptation (SSDA) and multiple instance learning (MIL). Under the MIL framework, a large scene can be regarded as a bag of instances (e.g., image patches), and information from different receptive field sizes can be exploited. We conducted experiments on a dataset of Japanese oak wilt, which is known to have severe forest damage with two different optical satellite imagery, SPOT6&7 satellite imageries (1.5m) and Pleiades-1B satellite imagery (0.5m). The proposed method improves the discrimination accuracy of the low-resolution model compared to the standard SSDA technique. This obtained result reveals the potential usefulness of MIL for effective multi-resolution modeling.

Multiple-instance learning (MIL) is a variant of the traditional supervised learning. In MIL training examples are bags of instances and labels are associated with bags rather than individual instances. The standard MIL assumption indicates that a bag is labeled positive if at least one of its instances is labeled positive, and otherwise labeled negative. However, many MIL problems do not satisfy this assumption but the more general one that the class of a bag is jointly determined by multiple instances of the bag. To solve such problems, the authors of MILD proposed an efficient disambiguation method to identify the most discriminative instances in training bags and then converted MIL to the standard supervised learning. Nevertheless, MILD does not consider the generalization ability of its disambiguation method, leading to inferior performance compared to other baselines. In this paper, we try to improve the performance of MILD by considering the discrimination of its disambiguation method on the validation set. We have performed extensive experiments on the drug activity prediction and region-based image categorization tasks. The experimental results demonstrate that MILD outperforms other similar MIL algorithms by taking into account the generalization capability of its disambiguation method.

We address the task of automatic detection of lesions caused by multiple myeloma (MM) in femurs or other long bones from CT data. Such detection is already an important part of the multiple myeloma diagnosis and staging. However, it is so far performed mostly manually, which is very time consuming. We formulate the detection as a multiple instance learning (MIL) problem, where instances are grouped into bags and only bag labels are available. In our case, instances are regions in the image and bags correspond to images. This has the advantage of requiring only subject-level annotation (ground truth), which is much easier to get than voxel-level manual segmentation. We consider a generalization of the standard MIL formulation where we introduce a threshold on the number of required positive instances in positive bags. This corresponds better to the classification procedure used by the radiology experts and is more robust with respect to false positive instances. We extend several existing MIL algorithms to solve the generalized case by estimating the threshold during learning. We compare the proposed methods with the baseline method on a dataset of 220 subjects. We show that the generalized MIL formulation outperforms standard MIL methods for this task. For the task of distinguishing between healthy controls and MM patients with infiltrations, our best method makes almost no mistakes with a mean AUC of 0.982 and F1 = 0.965. We outperform the baseline method significantly in all conducted experiments.

The papers in this special issue focus on machine learning for use in medical image processing applications. The use of machine learning in this area has become indispensable in diagnosis and treatment of many diseases. With advances in new imaging techniques, the need to take full advantage of abundant images draws more and more attention. Machine learning, including deep learning particularly, provides us a new paradigm to learn and to utilize the overwhelming volume of big imaging data smartly. Nowadays, machine learning in medical imaging has become one of the most promising and growing fields of research. The main aim of this special issue is to help advance the scientific research within the broad field of machine learning in medical imaging. The special issue was planned in conjunction with the International Workshop on Machine Learning in Medical Imaging (MLMI) 2017.

Multi-instance learning is a type of weakly supervised learning. It deals with tasks where the data is a set of bags and each bag is a set of instances. Only the bag labels are observed whereas the labels for the instances are unknown. An important advantage of multi-instance learning is that by representing objects as a bag of instances, it is able to preserve the inherent dependencies among parts of the objects. Unfortunately, most existing algorithms assume all instances to be identically and independently distributed, which violates real-world scenarios since the instances within a bag are rarely independent. In this work, we propose the Multi-Instance Variational Autoencoder (MIVAE) algorithm which explicitly models the dependencies among the instances for predicting both bag labels and instance labels. Experimental results on several multi-instance benchmarks and end-to-end medical imaging datasets demonstrate that MIVAE performs better than state-of-the-art algorithms for both instance label and bag label prediction tasks.

Image patch classification is an important task in many different medical imaging applications. In this work, we have designed a customized Convolutional Neural Networks (CNN) with shallow convolution layer to classify lung image patches with interstitial lung disease (ILD). While many feature descriptors have been proposed over the past years, they can be quite complicated and domain-specific. Our customized CNN framework can, on the other hand, automatically and efficiently learn the intrinsic image features from lung image patches that are most suitable for the classification purpose. The same architecture can be generalized to perform other medical image or texture classification tasks.

The use of machine learning (ML) has been increasing rapidly in the medical imaging field, including computer-aided diagnosis (CAD), radiomics, and medical image analysis. Recently, an ML area called deep learning emerged in the computer vision field and became very popular in many fields. It started from an event in late 2012, when a deep-learning approach based on a convolutional neural network (CNN) won an overwhelming victory in the best-known worldwide computer vision competition, ImageNet Classification. Since then, researchers in virtually all fields, including medical imaging, have started actively participating in the explosively growing field of deep learning. In this paper, the area of deep learning in medical imaging is overviewed, including (1) what was changed in machine learning before and after the introduction of deep learning, (2) what is the source of the power of deep learning, (3) two major deep-learning models: a massive-training artificial neural network (MTANN) and a convolutional neural network (CNN), (4) similarities and differences between the two models, and (5) their applications to medical imaging. This review shows that ML with feature input (or feature-based ML) was dominant before the introduction of deep learning, and that the major and essential difference between ML before and after deep learning is the learning of image data directly without object segmentation or feature extraction; thus, it is the source of the power of deep learning, although the depth of the model is an important attribute. The class of ML with image input (or image-based ML) including deep learning has a long history, but recently gained popularity due to the use of the new terminology, deep learning. There are two major models in this class of ML in medical imaging, MTANN and CNN, which have similarities as well as several differences. In our experience, MTANNs were substantially more efficient in their development, had a higher performance, and required a lesser number of training cases than did CNNs. "Deep learning", or ML with image input, in medical imaging is an explosively growing, promising field. It is expected that ML with image input will be the mainstream area in the field of medical imaging in the next few decades.

Histopathological Whole Slide Imaging (WSI) has become a standard in the detection of breast cancer. Automated image analysis methods attempt to reduce the workload from the clinicians and Convolutional Neural Networks (CNNs) are a popular choice for this purpose. However, size of a WSI image typically is approximately <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$40,000\times 40.000$ </tex-math></inline-formula> pixels (can reach up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$100,000\times 100.000$ </tex-math></inline-formula> pixels). CNNs cannot handle such large images. Moreover, downscaling a WSI image causes degradation of small-scale visual information. Hence, a large number of small patches (containing critical visual information) from a WSI image are extracted by a trained pathologist and are used for training. However, it requires massive amounts of time to precisely search and label appropriate class-representative patches. To address this issue, a Deep Multiple Instance Learning (MIL) based CNN framework has been introduced in this paper. In the proposed framework every slide is represented as a bag of extracted patches. Only the bag label is used for training, thus eliminating the requirement to provide patchwise labels. The patches inherit the label of the bag containing them. A WSI image (i.e. a bag) is labeled benign if all its patches are benign and labeled malignant even if a single patch contains malignant cells. Learning can be carried out at the bag level even with noisy patch labels. Performance of this method was evaluated using the BreakHis, IUPHL and UCSB breast cancer datasets where 93.06%, 96.63%, 95.83% accuracy was achieved respectively.

Visual Transformers(ViT) have made remarkable achievements in the field of medical image analysis. However, ViT-based methods have poor classification results on some small-scale medical image classification datasets. Meanwhile, many ViT-based models sacrifice computational cost for superior performance, which is a great challenge in practical clinical applications. In this paper, we propose an efficient medical image classification network based on an alternating mixture of CNN and Transformer tandem, which is called Eff-CTNet. Specifically, the existing ViT-based method still mainly relies on multi-head self-attention (MHSA). Among them, the attention maps of MHSA are highly similar, which leads to computational redundancy. Therefore, we propose a group cascade attention (GCA) module to split the feature maps, which are provided to different attention heads to further improves the diversity of attention and reduce the computational cost. In addition, we propose an efficient CNN (EC) module to enhance the ability of the model and extract the local detail information in medical images. Finally, we connect them and design an efficient hybrid medical image classification network, namely Eff-CTNet. Extensive experimental results show that our Eff-CTNet achieves advanced classification performance with less computational cost on three public medical image classification datasets.