5 - High-level feature extraction: fixed shape matching

Chapter 4 - Low-level feature extraction (including edge detection)

With the development of theories and technologies in medical imaging, most of the tumors can be detected in the early stage. However, the nature of ovarian cysts lacks accurate judgement, leading to that many patients with benign nodules still need Fine Needle Aspiration (FNA) biopsies or surgeries, increasing the physical pain and mental pressure of patients as well as unnecessary medical health care costs. Therefore, we present an image diagnosis system for classifying the ovarian cysts in color ultrasound images, which novelly applies the image features fused by both high-level features from deep learning network and low-level features from texture descriptor. Firstly, the ultrasound images are enhanced to improve the quality of training data set and the rotation invariant uniform local binary pattern (ULBP) features are extracted from each of the images as the low-level texture features. Then the high-level deep features extracted by the fine-tuned GoogLeNet neural network and the low-level ULBP features are normalized and cascaded as one fusion feature that can represent both the semantic context and the texture patterns distributed in the image. Finally, the fusion features are input to the Cost-sensitive Random Forest classifier to classify the images into "malignant" and "benign". The high-level features extracted by the deep neural network from the medical ultrasound image can reflect the visual features of the lesion region, while the low-level texture features can describe the edges, direction and distribution of intensities. Experimental results indicate that the combination of the two types of features can describe the differences between the lesion regions and other regions, and the differences between lesions regions of malignant and benign ovarian cysts.

Over the last decade, it has become increasingly affordable to digitize 2D and 3D shape information using multiple modalities, such as (video) cameras, image-based reconstruction systems, laser-range scanners, or depth cameras. If these dense models can be processed and described in an efficient and informative way, they can be used in applications, such as ergonomic design, virtual shopping, scientific and medical visualization, realistic simulation, photo-realistic rendering, the design of natural user interfaces, and semantic scene understanding. Traditionally, the notion of shape has been studied either by analyzing projections of shapes in images or by analyzing a sparse set of marker positions on 3D shapes. Typical tasks in 2D shape analysis include segmenting objects in images and tracking objects across a sequence of images, and typical tasks in 3D shape analysis include reconstructing the three-dimensional object depth from input images and identifying corresponding points on different 3D models. The analysis and processing of shape data becomes especially challenging because of the increasing amount of data captured by sensors used to acquire shapes, and because modern applications such as natural user interfaces require real-time processing of the input shapes. Meeting these challenges requires models of shape analysis that are compact and informative, thereby allowing the development of algorithms that can process large datasets efficiently. To achieve these goals, interdisciplinary approaches are needed that use concepts from a variety of research areas, including numerical computing, differential geometry, deformable shape modeling, sparse data representation, and machine learning. On the algorithmic side, many shape analysis tasks are modeled using partial differential equations, which can be solved using tools from the field of numerical computing. The fields of differential geometry and deformable shape modeling have recently begun to influence shape analysis methods. Furthermore, tools from the field of sparse representations, which aim to describe input data using a compressible representation with respect to a set of carefully selected basis elements, have the potential to significantly reduce the amount of data that needs to be processed in shape analysis tasks. The related field of machine learning offers similar potential. This seminar brought together 28 researchers from North America and Europe engaged in recent and upcoming developments in shape analysis who view these challenges from different perspectives and who together discussed the pressing open problems and novel solutions to them.

Accurate lung segmentation from chest X-rays is crucial for early disease diagnosis and monitoring of pulmonary diseases. For lung segmentation, CNN-based architecture plays a crucial role but still lacks some edge boundary detection issues while working with low-feature images. Previous research shows that there is still an issue in proper image segmentation with proper edge detection as in earlier methods like TVAC, Active Spline, Random Walker, U-Net as they focus on only high-level feature extraction from images and not low-level features, resulting in poor boundary detection. This research addresses this gap by proposing an Edge Attention-based U-Net architecture for proper edge boundary detection. This Edge Attention-based U-Net uses edge attention mechanisms to precisely locate lung boundaries while working with noisy and challenging X-ray imaging conditions. This improves segmentation accuracy and works efficiently in noisy situations, making it suitable for real-world clinical applications. Unlike previous methods the EA-U-Net surpasses them significantly. This superiority is evident through comprehensive comparisons conducted on the same Montgomery dataset of X-ray images during the training, validation, and testing phases. The EA-UNet consistently outperforms its predecessors, demonstrating higher accuracy metrics, including Dice mean, Jaccard Mean, and Pixel accuracy, while exhibiting superior boundary detection capabilities. The EA-U-Net's exceptional performance in lung segmentation enhances reliability and paves the way for advancements in computer-aided diagnosis and personalized healthcare. By providing clinicians with more accurate and reliable tools for analyzing chest X-rays, the EA-U-Net contributes significantly to improving patient care and medical decision-making processes.

Accurately and quickly classifying high dimensional data using machine learning and data mining techniques is problematic and challenging. This paper proposes an efficient and effective technique to properly extract high level features from medical images using a deep network and precisely classify them using support vector machine. A wavelet filter is applied at the first step of the proposed method to obtain the informative coefficient matrix of each image and to reduce dimensionality of feature space. A four-layer deep belief network is also utilized to extract high level features. These features are then fed to a support vector machine to perform accurate classification. Comparative empirical results demonstrate the strength, precision, and fast-response of the proposed technique.

Template matching is the process of searching the present and the location of a reference image or an object in a scene image. Template matching is a classical problem in a scene analysis: given a reference image of an object, decide whether that object exists in a scene image under analysis, and find its location if it does. The template matching process involves cross-correlating the template with the scene image and computing a measure of similarity between them to determine the displacement. The conventional matching method used the spatial cross-correlation process which is computationally expensive. Some algorithms are proposed for this speed problem, such as pyramid algorithm, but it still can't reach the real-time for bigger model image. Moreover, the cross-correlation algorithm can't be effective when the object in the image is rotated. Therefore, the conventional algorithms can't be used for practical purpose. In this paper, an algorithm for a rotation invariant template matching method based on different value circular projection target tracking algorithm is proposed. This algorithm projects the model image as circular and gets the radius and the sum of the same radius pixel value. The sum of the same radius pixel value is invariable for the same image and the any rotated angle image. Therefore, this algorithm has the rotation invariant property. In order to improve the matching speed and get the illumination invariance, the different value method is combined with circular projection algorithm. This method computes the different value between model image radius pixel sum and the scene image radius pixel sum so that it gets the matching result. The pyramid algorithm also is been applied in order to improve the matching speed. The high speed hardware system also is been design in order to meet the real time requirement of target tracking system. The results show that this system has the good rotate invariance and real-time property.

Due to the complexity of medical images, traditional medical image classification methods have been unable to meet the actual application needs. In recent years, the rapid development of deep learning theory has provided a technical approach for solving medical image classification. However, deep learning has the following problems in the application of medical image classification. First, it is impossible to construct a deep learning model with excellent performance according to the characteristics of medical images. Second, the current deep learning network structure and training strategies are less adaptable to medical images. Therefore, this paper first introduces the visual attention mechanism into the deep learning model so that the information can be extracted more effectively according to the problem of medical images, and the reasoning is realized at a finer granularity. It can increase the interpretability of the model. Additionally, to solve the problem of matching the deep learning network structure and training strategy to medical images, this paper will construct a novel multiscale convolutional neural network model that can automatically extract high-level discriminative appearance features from the original image, and the loss function uses the Mahalanobis distance optimization model to obtain a better training strategy, which can improve the robust performance of the network model. The medical image classification task is completed by the above method. Based on the above ideas, this paper proposes a medical classification algorithm based on a visual attention mechanism-multiscale convolutional neural network. The lung nodules and breast cancer images were classified by the method in this paper. The experimental results show that the accuracy of medical image classification in this paper is not only higher than that of traditional machine learning methods but also improved compared with other deep learning methods, and the method has good stability and robustness.

Cropland abandonment is crucial in agricultural management and has a profound impact on crop yield and food security. In recent years, many cropland abandonment identification methods based on remote sensing observation data have been proposed, but most of these methods are based on coarse-resolution images and use traditional machine learning methods for simple identification. To this end, we perform abandonment recognition on high-resolution remote sensing images. According to the texture features of the abandoned land, we combine the method of statistical texture learning and propose a new deep learning framework called PSPNet-STL. The model integrates high-level semantic feature extraction and deep mining of low-level texture features to identify cropland abandonment. First, we labeled the abandoned cropland area and built the HRAC dataset, a high-resolution cropland abandonment dataset. Secondly, we improved PSPNet by fusing statistical texture learning modules to learn multiple texture information on low-level feature maps, and combined high-level semantic features for cropland abandonment recognition. Experiments are performed on the HRAC dataset. Compared with other methods, the proposed model has the best performance on this dataset, both in terms of accuracy and visualization, proving that deep mining of low-level statistical texture features is beneficial for crop abandonment recognition.

Visual media geo-localization methods aim at determining where an image/video was taken and involve complex matching processes between the querymedia content (2D/3D features) and the reference data (overhead and geo-tagged ground-level imagery). This area of research encompasses many disciplines, including pre-processing and processing earth-scale overhead and ground-level reference imagery (i.e. satellite, elevation maps, LIDAR, open-street maps, geo-tagged street-view images, etc.) to extract relevant information (i.e. land-types, roads, buildings, etc.), efficient indexing of extracted geo-information, alignment and fusion of (heterogeneous) reference imageries, satellite imagery segmentation, low-level and high-level feature extraction, raytracing and synthesized imagery generation, wide imagery registration, multi-view geometry and 3D reconstruction, 2D/3D matching, geometric verification, machine learning, optimization, probabilistic inference, supervised and automatic image/video calibration, fast search, and humanmachine interaction. Advances in this research field were given great impetus by the increasing availability of computer resources

Systems for retrieving and managing content-based medical images are becoming more important, especially as medical imaging technology advances and the medical image database grows. In addition, these systems can also use medical images to better grasp and gain a deeper understanding of the causes and treatments of different diseases, not just for diagnostic purposes. For achieving all these purposes, there is a critical need for an efficient and accurate content-based medical image retrieval (CBMIR) method. This paper proposes an efficient method (RbQE) for the retrieval of computed tomography (CT) and magnetic resonance (MR) images. RbQE is based on expanding the features of querying and exploiting the pre-trained learning models AlexNet and VGG-19 to extract compact, deep, and high-level features from medical images. There are two searching procedures in RbQE: a rapid search and a final search. In the rapid search, the original query is expanded by retrieving the top-ranked images from each class and is used to reformulate the query by calculating the mean values for deep features of the top-ranked images, resulting in a new query for each class. In the final search, the new query that is most similar to the original query will be used for retrieval from the database. The performance of the proposed method has been compared to state-of-the-art methods on four publicly available standard databases, namely, TCIA-CT, EXACT09-CT, NEMA-CT, and OASIS-MRI. Experimental results show that the proposed method exceeds the compared methods by 0.84%, 4.86%, 1.24%, and 14.34% in average retrieval precision (ARP) for the TCIA-CT, EXACT09-CT, NEMA-CT, and OASIS-MRI databases, respectively.

Multi-level feature representations for video semantic concept detection

Content-based video retrieval is very interesting point where it can be used in our daily life. Video retrieval is regarded as one of the most important in multimedia research. The development of multimedia data types there is demand of video retrieval system. Video retrieval can be used for video search and browsing which are useful in web applications. Selection of extracted features play an important role in content based video retrieval. There are two types of feature extraction, low level feature extraction and high level feature extraction. Low level feature extraction based on color, shape, texture, spatial relationship. The main goal of this paper is that, user can give the two different types of input in the form of image query and the text query. First one is that give the input in the form of image query and retrieved image which is similar to the query image by using the CBIR algorithm. CBIR is still developing science. Retrieval of images based on visual features such as color, texture and shape. In this paper gives a detail description of a system developed for retrieving images similar to a query image from a different large set of image. Second one is that give the input in the form of text query and retrieved image by using the ABIR technique. ABIR is powerful algorithm for the information retrieval it can utilize their powerful natural language. Annotation is never complete.

The backbone networks used in Siamese trackers are relatively shallow, such as AlexNet and VGGNet, resulting in insufficient features for tracking task. Therefore, this paper focuses on extracting more discriminative features to improve the performance of Siamese trackers. By comprehensive experimental validations, this goal is achieved through a simple yet effective framework referred as relation-aware Siamese region proposal network (Ra-SiamRPN). Firstly, the deep backbone network ResNet-50 is adopted to extract both low-level detail features and high-level semantic features of an image. Then we propose the feature fusion module (FFM), which can combine low-level detail features with high-level semantic features effectively. Furthermore, we propose the relation reasoning module (RRM) to perform the global relation reasoning in multiple disjoint regions. RRM can generate discriminative information to enhance the features generated by ResNet-50. Extensive experiments are conducted on the dataset OTB2015, VOT2016, VOT2018, UAV123 and LaSOT. The experiment results indicate that Ra-SiamRPN achieves competitive performance with the current advanced algorithms and shows good real-time performance. To be highlighted, in the experiments conducted on the large-scale dataset LaSOT, the success score and the normalized precision score of Ra-SiamRPN are 0.495 and 0.576, respectively. These performance indexes are better than the second best tracker MDNet 24.7% and 25.2%.

Scene classification is an important tool for remote sensing image interpretation, and it has fundamental applications in research and industry. However, given complex backgrounds and scale variations, remote sensing images have large intraclass diversity and interclass similarity, which bring challenges to accurate classification of remote sensing images. We proposed a scene classification method using joint learning and multiscale attention to alleviate the aforementioned problems. To fully utilize the multiscale information of the image and improve the adaptability of the proposed method to objects with various sizes, different from general methods that fuse different scales of features for classification, joint learning using multiscale features is developed to optimize the whole network. Specifically, we leverage a pretrained deep convolutional neural network as the feature extractor to extract low-level, medium-level, and high-level feature maps from the images. Then, due to the poor semantics of low-level and medium-level feature maps compared with the high-level feature maps, we design a multiscale attention module to enhance the semantic information and suppress the noise information. Finally, the global mean pooling is used to obtain the feature vectors and different classifiers are used for different feature vectors. And the decision-level fusion is adopted to obtain more reliable predictions. The experimental results on the AID and NWPU-RESISC45 datasets show that the proposed method makes a significant improvement in terms of overall accuracies compared with the baselines. And the overall accuracies of our method on the two datasets are 97.49% and 95.20%, respectively, which achieves state-of-the-art performance. The code will be public at a Github repository available at https://github.com/Cbanyungong/JLMSAF.

Remote-sensing images (RSIs) generally have strong spatial characteristics for surface features. Various ground objects, such as residential areas, roads, forests, and rivers, differ substantially. According to this visual attention characteristic, regions with complicated texture features require more realistic details to reflect a better description of the topography, while regions such as farmlands should be smooth and have less noise. However, most existing single-image superresolution (SISR) methods fail to fully utilize these properties and therefore apply a uniform reconstruction strategy to the whole image. In this article, we propose a novel saliency-driven unequal single-image reconstruction network in which the demands of various regions in the superresolution (SR) process are distinguished by saliency maps. First, we design a new gradient-based saliency analysis method to produce more accurate saliency maps with imagewise annotations. The method utilizes the superiority of a multireception field to extract both high-level features and low-level features. Second, we propose a novel saliency-driven gate conditional generative adversarial network, where the saliency map is regarded as a medium during the training procedure of the whole network. The saliency map is regarded as a pixelwise condition in a generator to enhance the training capability of the network. Additionally, we design a new loss function that combines normalized content loss, saliency-driven perceptual loss, and gate-control adversarial loss to further refine details of texture-complex areas for RSIs. We evaluate the performance of our algorithm and compare it with many other state-of-the-art SR methods using a remote-sensing data set. The experimental results show that our approach achieves the optimal outcome in salient areas. Our method attains the best effect on global quality and visual performance.