Scaling Semantic Segmentation Beyond 1K Classes on a Single GPU

Conventional application of convolutional neural networks (CNNs) for image classification and recognition is based on the assumption that all target classes are equal (i.e. no hierarchy) and exclusive of one another (i.e. no overlap). CNN-based image classifiers built on this assumption, therefore, cannot take into account an innate hierarchy among target classes (e.g. cats and dogs in animal image classification) or additional information that can be easily derived from the data (e.g. numbers larger than five in the recognition of handwritten digits), thereby resulting in scalability issues when the number of target classes is large. Combining two related but slightly different ideas of hierarchical classification and logical learning by auxiliary inputs, we propose a new learning framework called hierarchical auxiliary learning, which not only address the scalability issues with a large number of classes but also could further reduce the classification/recognition errors with a reasonable number of classes. In the hierarchical auxiliary learning, target classes are semantically or non-semantically grouped into superclasses, which turns the original problem of mapping between an image and its target class into a new problem of mapping between a pair of an image and its superclass and the target class. To take the advantage of a superclass as a hint during the learning phase, we introduce an auxiliary block into a neural network, which generates auxiliary scores used as additional information for final classification/recognition; in this paper, we add the auxiliary block between the last residual block and the fully-connected output layer of the ResNet. Experimental results show that the proposed hierarchical auxiliary learning reduces classification errors up to 0.56, 1.6 and 3.56 percent with MNIST, SVHN and CIFAR-10 datasets, respectively.

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

The label tree-based classification is one of the most popular approaches for reducing the testing complexity to sublinear with the large number of classes. One of the popular approaches to generate the label tree is to apply recursively a spectral clustering algorithm to an affinity matrix for partition set of class labels into subsets, each subset corresponds to a child node of the tree. To obtain the affinity matrix from confusion matrix, a set of N binary one-versus-all classifiers is trained and applied on validation set, where N is number of classes. However, these approaches are not reliable when there are a large number of classes because it is too costly to train these classifiers. Furthermore, the affinity matrix could not reflect the real similarity among classes due to the classification accuracy can be low. In addition, the resulting label tree may not be balanced due to the objective function of spectral clustering penalizes unbalanced partitions. In this paper, to achieve better similarity measurement between classes and without using one-versus-all classifiers, we adopt the sum match kernel to get similarity matrix. Moreover, we propose a heuristic for learning balanced tree structure by adjusting the number of class labels in clusters after the spectral clustering is done. The experimental results on benchmark datasets SUN-397 and Caltech-256 show that the performance of the proposed approach outperforms significantly the other approaches.

This paper examines the possibilities of using Landsat Thematic Mapper images for updating land-use information on topographic maps. Information obtained from satellite images can be kept up-todate and a large number of classes can be distinguished. However, the quality of the information still leaves much to be desired, with a reliability of no more than 40 - 90 %. By using digitized landuse information from a topographic map as a source of contest information the reliability could be improved considerably. The method permitted the Identification of a large number of additional classes and provided quantitative insight into the dynamics of landscap change. Verification using 1:18,000 scale aerial photographs and field data showed the results to have a reliability of over 95 %. It can be expected that whereever topographic maps permit the extraction of land-use information in a similar way as described in this paper, remote sensing information will meet operational accuracy requirements for many fields of application e.g. physical planning, map udating, land-use planning etc. Paper presented at the IGARSS'88 conference Remote Sensing - moving towards the 21st century in Edinburgh, Scotland, 12-16 September 1988

Classification is one of the widely used techniques to quantify and mine the rich information present in hyperspectral imagery. However, the realization of trustworthy classification still continues to be a challenging task. This is due to the presence of various uncertainties, such as incomplete a priori knowledge on the actual number of classes and noise. Especially, mismatching between the number of spectral classes and the number of information classes leads to substantial omission or commission errors in the classification. In this letter, we present a new multiclass classification framework, named supervised cascaded classifier system (SC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> S), that addresses the abovementioned problem by providing reliable results. The SC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> S method is a two-stage cascaded classification procedure that involves quantifying the uncertainty and then classifying the samples. In the first stage, pixels for which no reliable training samples can be found in the training stage are detected. In the second stage, pixels that are matched with the respective training distribution are classified using a supervised learning algorithm, otherwise labeled as unknown. The proposed SC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> S framework has been implemented on eight different classification scenarios with a varying number of unknown classes (UCs) using hyperspectral and multispectral imageries. The performance of the proposed SC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> S method is compared with other widely used classifiers with and without reject-option. The experimental results indicate that our method offers superior classification results even in the case of data sets with a large number of unknown spectral classes.

Object detection and semantic segmentation are two fundamental techniques of various applications in the fields of Intelligent Vehicles (IV) and Advanced Driving Assistance System (ADAS). Early studies separately handle these two problems. In this paper, inspired by some recent works, we propose a deep neural network model for joint object detection and semantic segmentation. Given an image, an encoder-decoder convolution network extracts a set of feature maps, these feature maps are shared by the detection branch and the segmentation branch to jointly carry out the object detection and semantic segmentation. In the detection branch, we design a PriorBox initialization mechanism to propose more object candidates. In the segmentation branch, we use the multi-scale atrous convolution to explore the global and local semantic information in traffic scenes. Benefiting from the PriorBox Initialization Mechanism (PBIM) and Multi-Scale Atrous Convolution (MSAC), our model presents the competitive performance. In the experiments, we widely compare with several recently-proposed methods on the public Cityscapes dataset, achieving the highest accuracy. In addition, to verify the robustness and generalization of our model, the extension experiments are also conducted on the well-known VOC2012 dataset.

The development of technological devices, such as satellites and drones, has made it easier to collect images and videos from above. With rich and diverse data sources, the problem of detecting objects in aerial images is formed to serve real-life situations: rescue missions, social scenarios, security needs, and more. This issue attracts a lot of attention from researchers in computer vision because of its high applicability in real life. However, object detection in images collected from drones faces many significant challenges due to the complex factors of real-life environments. To overcome these difficulties, many studies focus on developing effective models for detecting and tracking objects from UAV systems. In this paper, our primary emphasis is comprehensive survey of methodologies pertaining to horizontal and oriented object detection. We analyzed and compared existing approaches such as CNN-based vs. Transformer-based, Anchor-based vs. Anchor-free. Moreover, we also learned about ways to represent objects using bounding boxes in oriented object detection. After researching many practical and innovative techniques in different approaches, we have carefully selected 15 methods for horizontal object detection and 18 for oriented object detection that were recently researched and developed to conduct classifications, surveys and assessments. We also provide a taxonomy mind map to assist readers in gaining an overview of the methods we surveyed. Additional, we examine 17 aerial image datasets collected from platforms and in various locations worldwide. These are typical datasets used to serve two primary tasks: Horizontal Object Detection and Oriented Object Detection. Then, we compared the experimental resultsof the previously introduced methods on four typical datasets: VisDrone, UAVDT, DOTAv1.0, and HRSC2016. The above datasets present diversity in the number of classes and the complexity of real-life conditions. Furthermore, they have been tested with various methods and shown remarkable results. For the reasons mentioned, we have enough basis to confirm that the comparison results are objective so that other research groups can make accurate comments and assessments on experimental methods on each dataset. We believe that updating and synthesizing the latest research will provide complete and comprehensive information, helping researchers to have more survey material for the development of the field of object detection in the future.

In order to improve the accuracy of image segmentation without changing the structure of original semantic segmentation models, an approach to train semantic segmentation models by using generative adversarial network (SS-GAN) is proposed. Using adversarial network to distinguish the source of segmented images, the model can learn the high-order relationship between pixels to enhance the spatial continuity of pixels in the segmented image. There are three aspects related to the work: constructing the generative model of fully convolutional network (FCN) structure by using VGG, and segment image preliminarily; constructing the adversarial model and training it by combining the original images, fake segmented images and real segmented images; modifying the loss function, adding the anti-loss to assist segmentation model training, encouraging generative network to learn the inter-pixel relationship independently. Experiments on PASCAL VOC and Cityscapes datasets show that the proposed method achieves better performance than the existing advanced methods, and improves IoU by 1.56% and 1.93%, respectively.

Supervised object detection and semantic segmentation require object or even pixel level annotations. When there exist image level labels only, it is challenging for weakly supervised algorithms to achieve accurate predictions. The accuracy achieved by top weakly supervised algorithms is still significantly lower than their fully supervised counterparts. In this paper, we propose a novel weakly supervised curriculum learning pipeline for multi-label object recognition, detection and semantic segmentation. In this pipeline, we first obtain intermediate object localization and pixel labeling results for the training images, and then use such results to train task-specific deep networks in a fully supervised manner. The entire process consists of four stages, including object localization in the training images, filtering and fusing object instances, pixel labeling for the training images, and task-specific network training. To obtain clean object instances in the training images, we propose a novel algorithm for filtering, fusing and classifying object instances collected from multiple solution mechanisms. In this algorithm, we incorporate both metric learning and density-based clustering to filter detected object instances. Experiments show that our weakly supervised pipeline achieves state-of-the-art results in multi-label image classification as well as weakly supervised object detection and very competitive results in weakly supervised semantic segmentation on MS-COCO, PASCAL VOC 2007 and PASCAL VOC 2012.

Recent advances in convolutional neural networks have led to considerably accurate semantic segmentation tasks. However, previous works use a pre-trained backbone borrowed from image classification tasks with considerable computational complexity, resulting in large latencies. Many methods have been proposed to reduce the latency of the segmentation network without loss of accuracy. These methods have reported varying experimental results with different computing devices, acceleration techniques, and input image sizes. Although most studies claim that their results are state-of-the-art, it is necessary to reconsider whether they are being compared under the same conditions. We propose a performance evaluation method of real-time semantic segmentation models to compare the performance under the same conditions fairly. In addition, we carry out an empirical study to evaluate the performance of recent real-time semantic segmentation networks and make a comparative analysis between them. We train the segmentation models using the same input data and data augmentation method. Then, the performance of the segmentation methods is analyzed regarding accuracy and speed. In contrast to most studies that exclude the time required for the pre-processing and post-processing steps, we measured the actual processing time needed to perform semantic segmentation with a real dataset. Further, we measured the processing speed and power consumption of the segmentation models in embedded devices in which real-time segmentation is applied, unlike previous studies that measured performance on a PC. Experimental results showed that the real-time semantic segmentation methods could not run in real-time on embedded devices when considering the pre-processing and post-processing steps. By comprehensively considering the inference speed, energy consumption, and processing time of semantic segmentation models, the experimental results show that FasterSeg-S is suited for embedded devices.

Scalability of object detectors with respect to the number of classes is a very important issue for applications where many object classes need to be detected. While combining single-class detectors yields a linear complexity for testing, multi-class detectors that localize all objects at once come often at the cost of a reduced detection accuracy. In this work, we present a scalable multi-class detection algorithm which scales sublinearly with the number of classes without compromising accuracy. To this end, a shared discriminative codebook of feature appearances is jointly trained for all classes and detection is also performed for all classes jointly. Based on the learned sharing distributions of features among classes, we build a taxonomy of object classes. The taxonomy is then exploited to further reduce the cost of multi-class object detection. Our method has linear training and sublinear detection complexity in the number of classes. We have evaluated our method on the challenging PASCAL VOC'06 and PASCAL VOC'07 datasets and show that scaling the system does not lead to a loss in accuracy. © 2011 IEEE.

Visual thalamic neurons were stained in a Golgilike fashion following cortical injections of horseradish peroxidase (HRP). Such staining allows neurons to be classified on the basis of their dendritic morphology, and it enables the direct relation of morphologically identified neurons with their respective cortical projection target to be visualized. All classes of neurons categorized on the basis of Golgi material (Guillery, '66) were identified in the dorsal part of the lateral geniculate body (LGBd): class 1 and class 2 neurons in the laminae A, A1, and C; class 3 neurons in laminae A and A1; class 4 cells in the C‐laminae. Three subtypes of class 4 neurons were differentiated: round cells, spindle cells, and fusiform cells. The round subtype is similar in dendritic morphology to the class 3 neurons, but its soma is larger (mean soma diameter 17 μm compared with 13 μm). Other types of neurons were identified in the laminae A, A1, and C, having morphological characteristics intermediate between class 1 and class 2, and between class 2 and class 3.Class 1 neurons, class 2 neurons, and flat neurons were identified in the medial interlaminar nucleus (MIN). The flat type is similar in morphology to the fusiform type in the C‐laminae; however, its soma is larger (mean soma diameter 23 μm compared with 19 μm), and its dendrites are free of appendages.In the extrageniculate nuclei three types of neurons were found: class 2 neurons in the pulvinar, medium‐sized cells—probably class 5—in the pulvinar and in the nucleus lateralis posterior, and neurons with an isodendritic branching pattern in the intralaminar nuclei.The cortical projections of the various thalamic cell classes were analyzed by injection of HRP into acallosal parts, or callosal parts of area 17 or area 18. Area 17 receives input from a large number of class 2 neurons and from a moderate number of class 1 neurons in the LGBd. In contrast, area 18 receives a dominant input from class 1 neurons and only weak input from class 2 neurons (up to 10% of the population of projection cells in the individual cases).Class 3 neurons in the A‐laminae and class 4 neurons in the C‐laminae project to either cortical area. Class 2 neurons in MIN project only to callosal parts of area 18, whereas class 2 neurons in pulvinar project in addition to acallosal parts of area 18. The significant change in the relative number of class 1 and class 2 neurons in the LGBd occurs gradually across the callosal zone from area 17 to area 18; it is not strictly related to the 17/18 border as defined by cytoarchitectonic or physiological criteria. The class 1/class 2 relation in the projection to area 18 is independent of eccentricity in the A‐laminae. An increase in the frequency of occurrence with eccentricity is noted, however, for class 4 neurons in the C‐laminae and for flat neurons in MIN. The correlation between neuronal function and morphology is discussed. It is concluded that, in addition to strong input with y‐characteristics, area 18 receives significant inputs with non‐y properties. latter inputs arise mainly from the laminae of the LGBd and from extra‐geniculate nuclei, but also may be seen to originate from MIN and the A‐laminae of the LGBd.

Question classification is an important phase in question answering systems. In this paper, we propose to apply i) hierarchical classifiers, ii) hierarchical classifiers in combination with semi-supervised learning and iii) hierarchy expansion for question classification for improving the precision. When the number of classes is large, the performance of classification algorithms may be affected. In order to improve the performance by reducing the number of classes for each classifier, we propose to use hierarchical classifiers according to the question taxonomy, in which each internal node is attached a classifier. We try to use semi-supervised learning to consume unlabeled questions with expectation to improve the performance of classifiers in the hierarchy. We explored different applications of learning methods in for each classifier of the hierarchy: a) supervised learning for all classifiers at all levels; b) semi-supervised learning for the first-level classifier and supervised learning for other classifiers; c) semi-supervised learning for all classifiers. The experiments show that the first method (a) has better results than those of flat classification; the second method (b) produces better results than those of the first method while the effort to increase the performance of fine classifiers in the last method (c) is not so successful. As another effort, we propose to automatically group question classes by clustering in order to expand a node which has a large number of classes in the question taxonomy. The experiment also shows that the overall precision is improved.

Evaluating the effects of similar-class combination on class integration test order generation

The yield and quality of fresh lettuce can be determined from the growth rate and color of individual plants. Manual assessment and phenotyping for hundreds of varieties of lettuce is very time consuming and labor intensive. In this study, we utilized a “Sensor-to-Plant” greenhouse phenotyping platform to periodically capture top-view images of lettuce, and datasets of over 2000 plants from 500 lettuce varieties were thus captured at eight time points during vegetative growth. Here, we present a novel object detection–semantic segmentation–phenotyping method based on convolutional neural networks (CNNs) to conduct non-invasive and high-throughput phenotyping of the growth and development status of multiple lettuce varieties. Multistage CNN models for object detection and semantic segmentation were integrated to bridge the gap between image capture and plant phenotyping. An object detection model was used to detect and identify each pot from the sequence of images with 99.82% accuracy, semantic segmentation model was utilized to segment and identify each lettuce plant with a 97.65% F1 score, and a phenotyping pipeline was utilized to extract a total of 15 static traits (related to geometry and color) of each lettuce plant. Furthermore, the dynamic traits (growth and accumulation rates) were calculated based on the changing curves of static traits at eight growth points. The correlation and descriptive ability of these static and dynamic traits were carefully evaluated for the interpretability of traits related to digital biomass and quality of lettuce, and the observed accumulation rates of static straits more accurately reflected the growth status of lettuce plants. Finally, we validated the application of image-based high-throughput phenotyping through geometric measurement and color grading for a wide range of lettuce varieties. The proposed method can be extended to crops such as maize, wheat, and soybean as a non-invasive means of phenotype evaluation and identification.