Multimodal biomedical image retrieval and indexing system using handcrafted with deep convolution neural network features

The classification of hyperspectral images (HSIs) is one of the most popular topics in the remote sensing community. Numerous feature extraction methods have been proposed to improve the classification accuracy of HSIs. Recently, deep features extracted by convolution neural network (CNN) have been introduced into the classification process of HSIs. Due to the nonlinear and invariant advantages of the features, CNN methods provide a powerful tool for representing geographic objects and classifying HSIs. However, traditional deep features only extracted at pixel-level and often neglect multiscale characteristics of geographic objects. In this study, a new deep feature extraction method is proposed, which takes advantage of multi-scale object analysis and the CNN model. Firstly, multiscale image objects are obtained by the multiscale segmentation algorithm and multiscale low-level features of objects are extracted. Secondly, the CNN is devoted to obtain deep features from low-level object features at each scale, respectively. Thirdly, the obtained deep features at all scales are stacked and fed to one fully connected layer to extract the multiscale deep learning features for classification. Finally, the logistic regression classifier is applied to hyperspectral image (HSI) classification based on object-oriented multiscale deep features. The proposed method was carried out on three widely used hyperspectral data sets: University of Pavia, Salinas, and Washington DC. The results reveal that the proposed method provides better results than other state-of-the-art methods.

Segmentation ability map: Interpret deep features for medical image segmentation.

Recently, monkeypox virus is slowly evolving and there are fears it will spread as COVID-19. Computer-aided diagnosis (CAD) based on deep learning approaches especially convolutional neural network (CNN) can assist in the rapid determination of reported incidents. The current CADs were mostly based on an individual CNN. Few CADs employed multiple CNNs but did not investigate which combination of CNNs has a greater impact on the performance. Furthermore, they relied on only spatial information of deep features to train their models. This study aims to construct a CAD tool named "Monkey-CAD" that can address the previous limitations and automatically diagnose monkeypox rapidly and accurately. Monkey-CAD extracts features from eight CNNs and then examines the best possible combination of deep features that influence classification. It employs discrete wavelet transform (DWT) to merge features which diminishes fused features' size and provides a time-frequency demonstration. These deep features' sizes are then further reduced via an entropy-based feature selection approach. These reduced fused features are finally used to deliver a better representation of the input features and feed three ensemble classifiers. Two freely accessible datasets called Monkeypox skin image (MSID) and Monkeypox skin lesion (MSLD) are employed in this study. Monkey-CAD could discriminate among cases with and without Monkeypox achieving an accuracy of 97.1% for MSID and 98.7% for MSLD datasets respectively. Such promising results demonstrate that the Monkey-CAD can be employed to assist health practitioners. They also verify that fusing deep features from selected CNNs can boost performance.

Along with the popularity of intelligent services and mobile services, service recommendation has become a key task, especially the task based on quality-of-service (QoS) in edge computing environment. Most existing service recommendation methods have some serious defects, and cannot be directly adopted in edge computing environment. For example, most of existing methods cannot learn deep features of users or services, but in edge computing environment, there are a variety of devices with different configurations and different functions, and it is necessary to learn deep features behind those complex devices. In order to fully utilize hidden features, this paper proposes a new matrix factorization (MF) model with deep features learning, which integrates a convolutional neural network (CNN). The proposed mode is named Joint CNN-MF (JCM). JCM is capable of using the learned deep latent features of neighbors to infer the features of a user or a service. Meanwhile, to improve the accuracy of neighbors selection, the proposed model contains a novel similarity computation method. CNN learns the neighbors features, forms a feature matrix and infers the features of the target user or target service. We conducted experiments on a real-world service dataset under a batch of cases of data densities, to reflect the complex invocation cases in edge computing environment. The experimental results verify that compared to counterpart methods, our method can consistently achieve higher QoS prediction results.

An innovative image retrieval agenda by concatenating deep learning features from GoogleNet and low‐level features from HSI and RGB color space is proposed in this article. Most of the CNN features suffer from loss of information due to image resize as a pre‐processing stage. To reduce this information loss super‐resolution technic is used for resizing images. An improved form of dot‐diffused block truncation coding is used for extracting RGB handcraft features. To discover the interdependencies between color and intensity component of an image, interchannel voting between hue, saturation, and intensity component is calculated as a color feature in HSI space. Histogram of orientated gradient (HOG) feature is used as shape feature. Five standard performance parameters, average precision rate, average recall rate, F‐Measure, Average Normalized Modified Retrieval Rank, and Total Minimum Retrieval Epoch, are applied on nine image datasets: Corel‐1K, Corel‐5K, Corel‐10K, VisTex, STex, ColorBrodatz and three subsets of ImageNet dataset for evaluation process of proposed method. For all dataset the best performance is achieved by the proposed method with respect to all performance parameters.

While traditional PIE (pose, illumination and expression) face variations have been well settled by latest methods, a new kind of variation, cross-age variation, is drawing attention from researchers. Most of the existing methods fail to maintain the effectiveness in real world applications that contain significant gap of age. Cross-age variation is caused by the shape deformation and texture changing of human faces while people getting old. It will result in tremendous intra-personal changes of face feature that deteriorate the performance of algorithms. This paper proposed a deep feature based framework for face retrieval problem. Our framework uses deep CNNs feature descriptor and two well designed post-processing methods to achieve age-invariance. To the best of our knowledge, this is the first deep feature based method in cross-age face retrieval problem. The deep CNNs model we use is firstly trained on traditional PIE datasets and then fine-tuned by cross-age dataset. The feature selection and projection post-processing we propose is also proved to be very effective in eliminating cross-age variation of deep CNNs feature. The experiments conducted on Cross-Age Celebrity Dataset (CACD), which is the largest public dataset containing cross-age variation, show that our framework outperforms previous state-of-the-art methods.

In the daily bidding activities, the bid price is estimated by using qualitative and quantitative predicting methods, which are usually poor in accuracy and efficiency. In this paper, we introduce a method based on the Convolutional Neural Networks (CNNs) for the bid price estimation decision. We collect the historical bidding price data from similar projects and then form a time series according to the order of the project start time to extract deep CNN features, and finally, these deep features are used to make bidding decision. We demonstrate that the learned deep models will help the decision makers to get the trend of project quotation and improve the bid winning probability and competitiveness of the bidder.

A Novel Medical Diagnosis model for COVID-19 infection detection based on Deep Features and Bayesian Optimization

e20611 Background: Predicting immunotherapy response in advanced non-small cell lung cancer (NSCLC) through non-invasive means could be a groundbreaking advancement. Deep features extracted from medical images by deep Convolutional Neural Networks (CNNs) along with radiomics have demonstrated promising results in different oncologic diseases. However, the method for extracting these deep features influences the predictive capabilities of the model. Methods: A retrospective analysis of a multicenter study was conducted in advanced NSCLC patients with high PD-L1 expression with the aim of developing an imaging features-based model for the prediction of best overall response (BOR) to a checkpoint inhibitor immunotherapy. All visible lung lesions, beyond RECIST 1.1, were manually segmented in baseline CT scans. Radiomics features were extracted at lesion-level, and various preprocessing methods and CNN architectures were tested for deep features extraction. Radiomics and deep features were harmonized using ComBat methodology. A distinctive feature signature was created for each patient by calculating the mean across all lesions. Using this data, machine learning models were trained to predict treatment response, specifically distinguishing between complete response (CR) and partial response (PR) from stable disease (SD) and progressive disease (PD). The study compared the performance of different deep features extraction methods and assessed shapley additive explanations (SHAP) values of radiomics for interpretability. Results: In the cohort of 116 patients (mean age: 67, 93% ever-smokers, 73% male), 1024 to 5475 quantitative imaging features per lesion were extracted. Among these patients, 67 patients were CR+PR, and 49 were SD+PD. The top-performing predictive model, trained with radiomics and deep features from a deep convolutional autoencoder, achieved an average AUC of 85%. SHAP values showed that impactful radiomics features were correlated with intensity changes in adjacent pixels describing the tumor heterogeneity, along with shape features potentially associated with malignancy. Ultimately, combining both features demonstrated statistical significance, leading to a performance improvement of ~2-5% compared to models exclusively using deep features. Conclusions: Diverse techniques for extracting deep features were evaluated, revealing autoencoder training as the most effective strategy for predicting immunotherapy response. Consequently, the findings suggest that employing deep learning to extract imaging features from baseline CT scans, combined with radiomics, represents a promising non-invasive approach for the predicting immunotherapy outcomes in advanced NSCLC patients.

In large-scale image retrieval, deep features extracted by Convolutional Neural Network (CNN) can effectively express more image information than those extracted by traditional manual methods. However, the deep feature dimensions obtained by Deep Convolutional Neural Network (DCNN) are too high and redundant, which leads to low retrieval efficiency. We propose a novel image retrieval method, which combines deep features selection with improved DCNN and hash transform based on high-dimension features reduction to gain lowdimension deep features and realizes efficient image retrieval. Firstly, the improved network is based on the existing deep model to build a more profound and broader network by adding multiple groups of different branches. Therefore, it is named DFS-Net (Deep Feature Selection Network). The adaptive learning deep features of the Network can effectively alleviate the influence of over-fitting and improve the feature expression of image content. Secondly, the information gain rate method is used to filter the extracted deep features to reduce the feature dimension and ensure the information loss is small. The last step of the method, hash Transform, sparsifies and binarizes this representation to reduce the computation and storage pressure while maintaining the retrieval accuracy. Finally, the scheme is based on the distinguished ResNet50, InceptionV3, and MobileNetV2 models, and studied and evaluated deeply on the CIFAR10 and Caltech256 datasets. The experimental results show that the novel method can train the deep features with stronger recognition ability on limited training samples, and improve the accuracy and efficiency of image retrieval effectively.

The images found within biomedical articles are sources of essential information useful for a variety of tasks. Due to the rapid growth of biomedical knowledge, image retrieval systems are increasingly becoming necessary tools for quickly accessing the most relevant images from the literature for a given information need. Unfortunately, article text can be a poor substitute for image content, limiting the effectiveness of existing text-based retrieval methods. Additionally, the use of visual similarity by content-based retrieval methods as the sole indicator of image relevance is problematic since the importance of an image can depend on its context rather than its appearance. For biomedical image retrieval, multimodal approaches are often desirable. We describe in this work a practical multimodal solution for indexing and retrieving the images contained in biomedical articles. Recognizing the importance of text in determining image relevance, our method combines a predominately text-based image representation with a limited amount of visual information, in the form of quantized content-based visual features, through a process called global feature mapping. The resulting multimodal image surrogates are easily indexed and searched using existing text-based retrieval systems. Our experimental results demonstrate that our multimodal strategy significantly improves upon the retrieval accuracy of existing approaches. In addition, unlike many retrieval methods that utilize content-based visual features, the response time of our approach is negligible, making it suitable for use with large collections.

Convolutional neural networks have been a funda-mental model in computer vision in the last years. Nevertheless, specifically in the analysis of texture images, the use of that model as a feature extractor rather than trained from scratch or extensively fine tuned has demonstrated to be more effective. In this scenario, such deep features can also benefit from further advanced analysis that can provide more meaningful representation than the direct use of feature maps. A successful example of such procedure is the recent use of visibility graphs to analyze deep features in texture recognition. It has been found that models based on complex networks can quantify properties such as periodicity, randomness and chaoticity. All those features demonstrated usefulness in texture classification. Inspired by this context, here we propose an alternative modeling based on complex networks to leverage the effectiveness of deep texture features. More specifically, we employ recurrence matrices of the neural activation at the penultimate layer. Moreover, the importance of complexity attributes, such as chaoticity and fractality, also instigates us to associate the complex networks with a fractal technique. More precisely, we complement the complex network representation with the application of fractal interpolation over the degree distribution of the recurrence matrix. The final descriptors are employed for texture classification and the results are compared, in terms of accuracy, with classical and state-of-the-art approaches. The achieved results are competitive and pave the way for future analysis on how such complexity measures can be useful in deep learning-based texture recognition.

Contour detection serves as the basis of a variety of computer vision tasks such as image segmentation and object recognition. The mainstream works to address this problem focus on designing engineered gradient features. In this work, we show that contour detection accuracy can be improved by instead making the use of the deep features learned from convolutional neural networks (CNNs). While rather than using the networks as a blackbox feature extractor, we customize the training strategy by partitioning contour (positive) data into subclasses and fitting each subclass by different model parameters. A new loss function, named positive-sharing loss, in which each subclass shares the loss for the whole positive class, is proposed to learn the parameters. Compared to the sofmax loss function, the proposed one, introduces an extra regularizer to emphasizes the losses for the positive and negative classes, which facilitates to explore more discriminative features. Our experimental results demonstrate that learned deep features can achieve top performance on Berkeley Segmentation Dataset and Benchmark (BSDS500) and obtain competitive cross dataset generalization result on the NYUD dataset.

9565 Background: Machine learning algorithms, and especially convolutional neural networks, demonstrated modest accuracy on the prediction of melanoma metastasis, based on histological images and clinicopathological information. Whether dermatoscopy deep features could serve as biomarker for the prediction of melanoma metastasis, remains an underexplored area in medical research. Methods: An international, multicenter, cohort study of cutaneous melanoma patients from 3 different continents was conducted. Patients with cutaneous melanoma, who had available clinical and dermatoscopic images and an adequate follow-up time for the development of metastasis (both locoregional and distant) were included. We utilized a support vector machine (SVM) classifier, to distinguish between melanomas that metastasized and those that did not. We used a pre-trained ResNet 50 network, we separated dataset into training set and testing set, stratified by TNM-stage, and to ensure robustness and guard against biased data selection, the stratified split into training and testing sets was repeated five times, resulting in five different training-test sets. The primary outcome was the comparison of the prognostic performance of deep dermatoscopy features based on SVM (model 1) to the performance of established prognostic factors of melanoma, such as Breslow thickness and ulceration (model 2) and to a combined model using deep features and histopathologic factors (model 3). A secondary aim was to examine the diagnostic performance of model 1 in stage IIB and IIC patients at diagnosis. The prognostic performance was assessed using the Area Under the Curve (AUC) and the True Positive Rate (TPR) at a True Negative Rate at 70%. Results: 712 patients were included, 465 (65.3%) non-metastatic and 247 (34.7%) metastatic, within a median follow-up of 60 months. The SVM model demonstrated mean AUC 0.84 (95% CI 0.80 – 0.87) and TPR 0.81 (95%CI 0.73 – 0.90). Similar results were shown for model 2 and model 3, and no statistically significant differences among models were detected in terms of AUC (De Long’s test, p>0.05 and ANOVA Kruskal-Wallis p>0.05). Regarding IIB/IIC patients and combining data from five test sets, SVM correctly classified as metastatic 21 out of 23 (91.3%) stage IIB and 21 out of 23 (91.3%) stage IIC patients, who eventually developed metastasis during follow-up. Conclusions: Our findings suggest that dermatoscopy deep features could offer an immediate, in vivo prediction of melanoma metastasis prior to excision. This advancement holds significant clinical importance, prioritizing high-risk patients for neoadjuvant treatment or guiding selection of patients who might benefit from adjuvant therapy.

Deep manifold feature fusion for classification of breast histopathology images