Computerized Prediction of Radiological Observations Based on Quantitative Feature Analysis: Initial Experience in Liver Lesions

This chapter introduces an advanced content-based image retrieval (CBIR) system, MMIR, where Markov model mediator (MMM) and multiple instance learning (MIL) techniques are integrated seamlessly and act coherently as a hierarchical learning engine to boost both the retrieval accuracy and efficiency. It is well-understood that the major bottleneck of CBIR systems is the large semantic gap between the low-level image features and the highlevel semantic concepts. In addition, the perception subjectivity problem also challenges a CBIR system. To address these issues and challenges, the proposed MMIR system utilizes the MMM mechanism to direct the focus on the image level analysis together with the MIL technique (with the neural network technique as its core) to real-time capture and learn the object-level semantic concepts with some help of the user feedbacks. In addition, from a long-term learning perspective, the user feedback logs are explored by MMM to speed up the learning process and to increase the retrieval accuracy for a query. The comparative studies on a large set of real-world images demonstrate the promising performance of our proposed MMIR system.

Due to the unique challenges of high intra-class variation and low inter-class variation, Fine-Grained Visual Classification (FGVC) tasks extremely need to extract multilevel semantic features of fine-grained images for classification. Moreover, labor-intensive annotations and the existence of long-tailed distribution of fine-grained images in real life make fine-grained few-shot learning an urgent problem to be solved. In this paper, we propose an Adaptive Weighting Pyramidal Convolutional Neural Network (AW-PCNN) for fine-grained few-shot learning. Our AW-PCNN consists of a PCNN module and a AW module, which are improved in two aspects. First, our PCNN module extracts the features of each layer in CNN to obtain both high-level global and low-level local subtle features of images to overcome the challenge of FGVC tasks. Second, We employ the metric learning approach for few-shot learning, and our AW module improves it by selecting decisive pairs and adaptively weighting the pairs based on their similarity scores to mitigate the challenge of FGVC tasks and learn a better embedding space. Our AW-PCNN achieves state-of-the-art performance on three benchmark fine-grained datasets, which proves the effectiveness and superiority of our model.

Ontologies represent domain concepts and relations in a form of semantic network. Many research works use ontologies in the information matchmaking and retrieval. This trend is further accelerated by the convergence of various information sources supported by ontologies. In this paper, we propose a novel multi-modality ontology model that integrates both the low-level image features and the high-level text information to represent image contents for image retrieval. By embedding this ontology into an image retrieval system, we are able to realize intelligent image retrieval with high precision. Moreover, benefiting from the soft-coded ontology model, this system has good flexibility and can be easily extended to the larger domains. Currently, our experiment is conducted on the animal domain canine. An ontology has been built based on the low-level features and the domain knowledge of canine. A prototype retrieval system is set up to assess the performance. We compare our experiment results with traditional text-based image search engine and prove the advantages of our approach.

In this paper, we propose a text extraction method from camera-captured document style images and propose a text segmentation method based on a color clustering method. The proposed extraction method detects text regions from the images using two low-level image features and verifies the regions through a high-level text stroke feature. The two level features are combined hierarchically. The low-level features are intensity variation and color variance. And, we use text strokes as a high-level feature using multi-resolution wavelet transforms on local image areas. The stroke feature vector is an input to a SVM (support vector machine) for verification, when needed. The proposed text segmentation method uses color clustering to the extracted text regions. We improved K-means clustering method and it selects K and initial seed values automatically. We tested the proposed methods with various document style images captured by three different cameras. We confirmed that the extraction rates are good enough to be used in real-life applications.

In this paper, we propose a clustering method in order to effectively present image search results on the Web. In order to reflect the difference of image semantics among the images, we use the meta-tags added by social tagging. Furthermore, we use low-level image features and photo metadata in order to consider the image looks and photo-taking conditions. We applied constrained agglomerative clustering method with must-link constraints for better clustering results by using multiple similarity metric. We conducted experiments to demonstrate that the proposed method effectively clusters image search results comparing to the traditional weighted similarity aggregation approach, and in some cases the clustering performance of our approach is better than other existing approaches for clustering tasks on an online photo sharing site.

As the global population is increasing and agricultural resources are diminishing, effective management of plant diseases become critical. Convolutional neural network-based models have been widely adopted in agriculture cyber–physical systems due to their high accuracy. However, these models require large, labeled datasets for training, limiting their practicality in plant disease detection where data are heterogeneous, scarce, and subjected to continual evolution. To address these challenges, we propose semantic-search, a knowledge-driven system that interprets plant diseases based on high-level semantic features, such as visual patterns, colors, and crop type, rather than relying solely on low-level image features. The system classifies diseases by querying a structured knowledge base using these semantics. “Semantic search” here refers to classification based on semantic feature extraction and knowledge base querying. It enables greater adaptability, interpretability, and robustness to intraclass variability caused by differing environmental conditions. The system can incorporate new disease classes through simple knowledge base updates, eliminating the need for retraining. This facilitates efficient scaling to evolving set of diseases. We validated the proposed approach on a dataset of 11 000 images encompassing 21 diseases across 11 plant species, achieving an accuracy of 90%, thereby demonstrating its effectiveness, scalability.

Shape information plays an important role in biomedical image analysis because of the strong shape characteristics of biological structures. It is often used as a prior to constrain or refine the intermediate shape information derived from low-level image features. In this paper, we give an overview of the sparse shape composition based prior modeling method and its various applications of biomedical image analysis. Instead of learning a generative shape model, it incorporates shape priors on-the-fly through the sparse shape composition. Particularly, a shape instance derived from low level image features is refined by a sparse linear combination of a sparse set of shapes in the repository. We also design three strategies to improve run-time efficiency. 1) When the shape repository contains a large number of instances, K-SVD can be used to learn a more compact but still informative shape dictionary. 2) If the derived shape instance has a large number of vertices, which often appears in 3D problems, affinity propagation method can be used to partition the surface into small subregions. 3) When there are multiple structures, hierarchical scheme is employed to model them simultaneously. These strategies decrease the scale of the sparse optimization problem and thus speed up the algorithm. Our method is applied on different biomedical image analysis problems, including localization, tracking and segmentation of anatomical structures. In all of these applications, this method achieves promising results.

Content-based techniques enable retrieval of remotely sensed data based on low-level features. However, the deep gap between low-level features and high-level semantics concepts is a major obstacle to more effective image retrieval. Therefore, a semantics-based retrieval approach was implemented. The semantics classifiers are trained using heterogeneous features from a group of satellite images. The proposed approach is mainly composed of two steps. The first step is to form hierarchical semantics classifiers based on the low-level features of the training images. In the second step, unknown satellite images are classified into a certain semantics class if their feature vectors are located in the corresponding feature space. To achieve an effective and at the same time efficient identification of the multiple semantics classes within the satellite scenes, an approximation approach was developed. At the query time, the system retrieves the satellite images based on semantics classes extracted from the query image or provided directly by the users

In the traditional image processing approaches, first low-level image features are extracted and then they are sent to a classifier or a recognizer for further processing. While the traditional image processing techniques employ this step-by-step approach, majority of the recent studies prefer layered architectures which both extract features and do the classification or recognition tasks. These architectures are referred as deep learning techniques and they are applicable if sufficient amount of labeled data is available and the minimum system requirements are met. Nevertheless, most of the time either the data is insufficient or the system sources are not enough. In this study, we experimented how it is still possible to obtain an effective visual representation by combining low-level visual features with features from a simple deep learning model. As a result, combinational features gave rise to 0.80 accuracy on the image data set while the performance of low-level features and deep learning features were 0.70 and 0.74 respectively.

In the traditional image processing approaches, first low-level image features are extracted and then they are sent to a classifier or a recognizer for further processing. While the traditional image processing techniques employ this step-by-step approach, majority of the recent studies prefer layered architectures which both extract features and do the classification or recognition tasks. These architectures are referred as deep learning techniques and they are applicable if sufficient amount of labeled data is available and the minimum system requirements are met. Nevertheless, most of the time either the data is insufficient or the system sources are not enough. In this study, we experimented how it is still possible to obtain an effective visual representation by combining low-level visual features with features from a simple deep learning model. As a result, combinational features gave rise to 0.80 accuracy on the image data set while the performance of low-level features and deep learning features were 0.70 and 0.74 respectively.

Scene categorization to indoor vs outdoor may be approached by using low-level features for inferring high-level information about the image. Low-level features such as color and texture have been used extensively in image understanding research, however, they cannot solve the problem completely. We propose the use of a Bayesian network for integrating knowledge from low-level and semantic features for indoor vs outdoor classification of images. Using ground truth data for sky and grass detection, we demonstrate that the classification performance can be significantly improved when semantic features are employed in the classification process.

We consider the problem of image classification with semantic features that are built from a set of base classifier outputs, each reflecting visual concepts. However, existing approaches consider visual concepts independently from each other whereas they are often linked together. When those relations are considered, existing models strongly rely on image low-level features, yielding in irrelevant relations when the low-level representation fails. On the contrary, the approach we propose, uses existing human knowledge, the application context itself and the human categorization mechanism to reflect complex relations between concepts. By nesting this human knowledge and the application context in the concept detection and selection processes, our final semantic feature captures the most useful information for an effective categorization. Thus, it enables to give good representation, even if some important concepts failed to be recognized. Experimental validation is conducted on three publicly available benchmarks of multi-class object classification and leads to results that outperforms comparable approaches.

Mobile and social technologies are providing new opportunities to document, characterize, and gather impressions of urban environments. In this article, we present a study that examines urban perceptions of three cities in central Mexico; the study integrates a mobile crowdsourcing framework to collect geo-localized images of urban environments by a local youth community, an online crowdsourcing platform to gather impressions of urban environments along 12 physical and psychological dimensions, and a deep learning framework to automatically infer human impressions of outdoor urban scenes. Our study resulted in a collection of 7,000 geo-localized images containing outdoor scenes and views of each city’s built environment, including touristic, historical, and residential neighborhoods, and 144,000 individual judgments from Amazon Mechanical Turk. Statistical analyses show that outdoor environments can be assessed in terms of interrater agreement for most of the urban dimensions by the observers of crowdsourced images. Furthermore, we proposed a methodology to automatically infer human perceptions of outdoor scenes using a variety of low-level image features and generic deep learning (CNN) features. We found that CNN features consistently outperformed all the individual low-level image features for all the studied urban dimensions. We obtained a maximum R 2 of 0.49 using CNN features; for 9 out of 12 labels, the obtained R 2 values exceeded 0.44.

In this chapter we discuss how to take advantage of association rule mining to promote feature selection from low-level image features. Feature selection can significantly improve the precision of content-based queries in image databases by removing noisy and redundant features. A new algorithm named StARMiner is presented. StARMiner aims at finding association rules relating low-level image features to high-level knowledge about the images. Such rules are employed to select the most relevant features. We present a case study in order to highlight how the proposed algorithm performs in different situations, regarding its ability to select the most relevant features that properly distinguish the images. We compare the StARMiner algorithm with other well-known feature selection algorithms, showing that StARMiner reaches higher precision rates. The results obtained corroborate the assumption that association rule mining can effectively support dimensionality reduction in image databases.KeywordsFeature VectorFeature SelectionAssociation RuleRelevance FeedbackAssociation Rule MiningThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Learning diagnostic labels from image content has been the standard in computer-aided diagnosis. Most computer-aided diagnosis systems use low-level image features extracted directly from image content to train and test machine learning classifiers for diagnostic label prediction. When the ground truth for the diagnostic labels is not available, reference truth is generated from the experts diagnostic interpretations of the image/region of interest. More specifically, when the label is uncertain, e.g. when multiple experts label an image and their interpretations are different, techniques to handle the label variability are necessary. In this paper, we compare three consensus techniques that are typically used to encode the variability in the experts labeling of the medical data: mean, median and mode, and their effects on simple classifiers that can handle deterministic labels (decision trees) and probabilistic vectors of labels (belief decision trees). Given that the NIH/NCI Lung Image Database Consortium (LIDC) data provides interpretations for lung nodules by up to four radiologists, we leverage the LIDC data to evaluate and compare these consensus approaches when creating computer-aided diagnosis systems for lung nodules. First, low-level image features of nodules are extracted and paired with their radiologists semantic ratings (1= most likely benign, , 5 = most likely malignant); second, machine learning multi-class classifiers that handle deterministic labels (decision trees) and probabilistic vectors of labels (belief decision trees) are built to predict the lung nodules semantic ratings. We show that the mean-based consensus generates the most robust classi- fier overall when compared to the median- and mode-based consensus. Lastly, the results of this study show that, when building CAD systems with uncertain diagnostic interpretation, it is important to evaluate different strategies for encoding and predicting the diagnostic label.