Large scale image classification and object detection

Drones, or general UAVs, equipped with cameras have been fast deployed with a wide range of applications, including agriculture, aerial photography, and surveillance. Consequently, automatic understanding of visual data collected from drones becomes highly demanding, bringing computer vision and drones more and more closely. To promote and track the developments of object detection and tracking algorithms, we have organized three challenge workshops in conjunction with ECCV 2018, ICCV 2019 and ECCV 2020, attracting more than 100 teams around the world. We provide a large-scale drone captured dataset, VisDrone, which includes four tracks, i.e., (1) image object detection, (2) video object detection, (3) single object tracking, and (4) multi-object tracking. In this paper, we first present a thorough review of object detection and tracking datasets and benchmarks, and discuss the challenges of collecting large-scale drone-based object detection and tracking datasets with fully manual annotations. After that, we describe our VisDrone dataset, which is captured over various urban/suburban areas of 14 different cities across China from North to South. Being the largest such dataset ever published, VisDrone enables extensive evaluation and investigation of visual analysis algorithms for the drone platform. We provide a detailed analysis of the current state of the field of large-scale object detection and tracking on drones, and conclude the challenge as well as propose future directions. We expect the benchmark largely boost the research and development in video analysis on drone platforms. All the datasets and experimental results can be downloaded from https://github.com/VisDrone/VisDrone-Dataset.

Object detection is crucial to extracting and analyzing information autonomously from high-resolution remote sensing images (HRSIs). To address ideal blocking for large-scale HRSI object detection, this study uses a novel adaptive orientation object-detection method for largescale HRSIs based on multi-scale block fusion. An adaptive orientation object-detection framework based on a convolutional neural network is applied to detect diverse objects of large-scale HRSIs through different block scales; average precision (AP) values of diverse object detection results are calculated at different block scales. Then, block scales matching the largest AP values of diverse objects are determined based on statistical results of the AP values of the diverse object at different block scales. Finally, object-detection results at block scales matching the largest AP values of diverse objects are fused by the non-maximum suppression algorithm to achieve large-scale HRSI object-detection results. Experimental findings reveal that the proposed method is better than any single block-scale object-detection method, resulting in satisfactory large-scale HRSI object-detection results.

We propose Real Full Binary Neural Network (RFBNN), a method that can reduce the memory and compute power of Deep Neural Networks. This method has similar performance to other BNNs in image classification and object detection, while reducing computation power and memory size. In RFBNN, the weight filters are approximated as a binary value by applying a sign function; we apply real binary weight to the whole layer. Therefore, RFBNN can be efficiently implemented on CPU, FPGA and GPU. Results of the all experiments show that the proposed method works successfully on various task such as image classification and object detection. All layers in our networks are composed of only {1, −1}, and unlike the other BNN, there is no scaling factor. Compared to recent network binarization methods, BC, BWN and BWRN, we have reduced the memory size and computation costs, but the performance is the same or better.

Generic object detection is one of the most fundamental problems in computer vision, yet it is difficult to provide all the bounding-box-level annotations aiming at large-scale object detection for thousands of categories. In this paper, we present a novel cross-supervised learning pipeline for large-scale object detection, denoted as CS-R-FCN. First, we propose to utilize the data flow of image-level annotated images in the fully-supervised two-stage object detection framework, leading to cross-supervised learning combining bounding-box-level annotated data and image-level annotated data. Second, we introduce a semantic aggregation strategy utilizing the relationships among the cross-supervised categories to reduce the unreasonable mutual inhibition effects during the feature learning. Experimental results show that the proposed CS-R-FCN improves the mAP by a large margin compared to previous related works.

Purpose This study aims to automate the visual inspection of piling sheets in water channel construction using artificial intelligence (AI). By employing image classification and object detection techniques, the research focuses on extracting and analysing geometric features to enhance the accuracy and efficiency of the inspection process. It also addresses key challenges associated with the unique characteristics of construction materials and the limited variability of available inspection datasets. Design/methodology/approach Convolutional neural networks (CNNs) with varying complexities are employed for image classification, across four and six classes, and for object detection of piling sheets in water channel environments. A dataset provided by Witteveen + Bos is preprocessed to generate training sets, and the CNN architectures are optimized for enhanced performance. The accuracy and efficiency of the proposed models are evaluated and compared against traditional manual inspection methods. Findings The AI-driven approach significantly reduces processing time, evaluating 40,000 images in just 11.9 h, compared to approximately one month using manual assessment. The 4-class classification model achieves an accuracy of 96%, while the 6-class model attains 72%. The object detection model produces a mean average precision (mAP) of 79%. These results meet the performance standards set by the Dutch company Witteveen + Bos, which demonstrate the effectiveness of AI in automating the inspection of piling sheets. Originality/value This study introduces a novel AI-based approach for assessing piling sheets, demonstrating substantial improvements over traditional inspection methods. It introduces a systematic evaluation of various CNN architectures and hyperparameters to optimize the models specifically for piling sheet inspection rather than relying on off-the-shelf solutions. The use of CNNs for both image classification and object detection adheres to relevant Dutch engineering standards. Notably, the reduction in processing time, from one month to around 12 h, represents a major advancement in the efficiency of civil engineering inspections.

Traffic violation apprehension is one of the traffic problems here in the Philippines. One example is the No Helmet No Ride Law that is implemented but many motorists still choose to ignore. To alleviate the problem the government has offered many solutions, one of which is the No Contact Traffic Apprehension Policy that uses CCTV Monitoring. To further enhance this solution the government has partnered with the De La Salle University to use artificial intelligence in the system. Computer Vision tasks like image classification and object detection can help automate the traffic apprehension system. Image classification and object detection are technologies which are used in computer vision in defining an image or coordinates of an object in an image. In this work, a novel approach to classifying motorcycle riders between wearing a helmet or not will be developed. It will be demonstrated using deep machine learning, specifically convolutional neural network and by utilizing different pre-trained models to a gathered dataset.

Nowadays Convolutional Neural Networks (CNNs) are being employed in a wide range of industrial technologies for a variety of sectors, such as medical, automotive, aviation, agriculture, space, etc. This paper reviews the state-of-the-art in both the field of CNNs for image classification and object detection and Autonomous Driving Systems (ADSs) in a synergetic way. Layer-based details of CNNs along with parameter and floating-point operation number calculations are outlined. Using an evolutionary approach, the majority of the outstanding image classification CNNs, published in the open literature, is introduced with a focus on their accuracy performance, parameter number, model size, and inference speed, highlighting the progressive developments in convolutional operations. Results of a novel investigation of the convolution types and operations commonly used in CNNs are presented, including a timing analysis aimed at assessing their impact on CNN performance. This extensive experimental study provides new insight into the behaviour of each convolution type in terms of training time, inference time, and layer level decomposition. Building blocks for CNN-based object detection are also discussed, such as backbone networks and baseline types, and then representative state-of-the-art designs are outlined. Experimental results from the literature are summarised for each of the reviewed models. This is followed by an overview of recent ADSs related works and current industry activities, aiming to bridge academic research and industry practice on CNNs and ADSs. Design approaches targeted at solving problems of automakers in achieving real-time implementations are also proposed based on a discussion of design constraints, human vs. machine evaluations and trade-off analysis of accuracy vs. size. Current technologies, promising directions, and expectations from the literature on ADSs are introduced including a comprehensive trade-off analysis from a human-machine perspective.

In computer vision, the bag-of-words (BoW) model has been widely applied to image related tasks, such as large scale image retrieval, image classification, and object categorization. The sparse coding (SC) method which leverages SC as a means of feature coding can guarantee both sparsity of coding vector and lower reconstruction error in the BoW model. Thus it can achieve better performance than the traditional vector quantization method. However, it suffers from the side effect introduced by the non-smooth sparsity regularizer that quite different words may be selected for similar patches to favor sparsity, resulting in the loss of correlation between the corresponding coding vectors. To address this problem, in this paper, we propose a novel soft assignment method based on index combination of top-2 large sparse codes of local descriptors to make the SC-based BoW tolerate the case of different word selection for similar patches. To further ensure similar patches select same words to generate similar coding vectors, we propose a collaborative dictionary learning method through imposing the sparse code similarity regularization factor along with the row sparsity regularization across data instances on top of group sparse coding. Experiments on the well-known public Oxford dataset demonstrate the effectiveness of our proposed methods.

CMOS Image sensors play a vital role in the exponentially growing field of Artificial Intelligence (AI). Applications like image classification, object detection and tracking are just some of the many problems now solved with the help of AI, and specifically deep learning. In this work, we target image classification to discern between six categories of fruits — fresh/ rotten apples, fresh/ rotten oranges, fresh/ rotten bananas. Using images captured from high speed CMOS sensors along with lightweight CNN architectures, we show the results on various edge platforms. Specifically, we show results using ON Semiconductor’s global-shutter based, 12MP, 90 frame per second image sensor (XGS-12), and ON Semiconductor’s 13 MP AR1335 image sensor feeding into MobileNetV2, implemented on NVIDIA Jetson platforms. In addition to using the data captured with these sensors, we utilize an open-source fruits dataset to increase the number of training images. For image classification, we train our model on approximately 30,000 RGB images from the six categories of fruits. The model achieves an accuracy of 97% on edge platforms using ON Semiconductor’s 13 MP camera with AR1335 sensor. In addition to the image classification model, work is currently in progress to improve the accuracy of object detection using SSD and SSDLite with MobileNetV2 as the feature extractor. In this paper, we show preliminary results on the object detection model for the same six categories of fruits.

Review on self-supervised image recognition using deep neural networks

In recent years, content-based image retrieval method has attracted the attention of many researchers. With the emergence of large-scale computer vision data set, large-scale image classification is going to become an important challenge in the field of computer vision. The existing image classification algorithms mainly have the following defects: (1) lack of sufficient theoretical basis; (2) lower classification accuracy and larger error; (3) algorithm running time is too long because of large amount of data. In order to solve these problems, we introduce an algorithm based on visual saliency and feature extraction theory. Visual saliency means that intelligent algorithm can mark the significant areas in the picture through the simulation of human visual characteristics, and we can preliminarily screen the large-scale date set in terms of saliency extraction, in order to constitute an initial categories set. Image classification mainly includes three processes, that are feature extraction, computation of image pattern vector, classification. Feature extraction is the key factor that affects the performance of image classification. Comparing to global features of image with blur background, partial occlusion, and illumination changes, local features are mode adaptive. Traditional image classification algorithms only use single feature, thus these methods bring limitations. In this paper, we have a thorough research of image classification methods based on feature combination, and we also propose an algorithm based on multiple feature. At the same time, in order to solve the problem that traditional methods dealing with the whole image in a non-hierarchical way, we introduce the theory of visual saliency detection. The results obviously shows that efficient feature extraction combined with the theory of visual saliency are significant to image classification.

Image classification is one of the important problems in the field of machine learning. Deep learning architectures are used in many machine learning applications such as image classification and object detection. The ability to manipulate large image clusters and implement them quickly makes deep learning a popular method in classifying images. This study points out the success of the convolutional neural networks which is the architecture of deep learning, in solving image classification problems. In the study, the convolutional neural network model of the winner of ilsvrc12 competition is implemented. The method distinguishes 1.2 million images with 1000 categories in success. The application is performed with the caffe library, and the image classification process is employed. In the application that uses the speed facility provided by GPU, the test operation is performed by using the images in Caltech-101 dataset.

There has been a breakthrough in the field of image classification and object detection, owing to the development of GPU and deep learning. However, because of the huge computation of deep learning, it is hard to use the deep learning algorithms in an embedded platform or a mobile device. Therefore, many compression studies have been conducted, and one of the most popular methods is a parameter quantization. In this paper, we propose an adaptive quantization scheme that reduces the loss of accuracy due to the quantization by properly mixing deterministic and stochastic quantization methods, while retaining the characteristics of the hardware-friendly fixed-point quantization method. By applying the proposed method to the weight parameters of image classification and object detection networks, the proposed method shows better mean average precision (mAP) of up to 0.44% in image classification and 0.91 % in object detection.

Deep neural networks (DNNs) play a crucial role in image classification and object detection, with applications in autonomous driving, security, Unmanned Aerial Vehicle (UAV) navigation, and robotics. Ensuring secure execution on mobile devices is challenging due to the sensitivity of input data and DNN architectures. Hardware-based Trusted Execution Environments (TEEs), such as ARM TrustZone, offer security but face resource and performance limitations when handling full-scale DNNs. This work introduces EffTEE, a security framework that enables efficient DNN execution within mobile TEEs. EffTEE employs dynamic suppression to prune unimportant neurons, foundational neuron restructuring to optimize memory usage, and dynamic slicing for effective model partitioning. Experimental results show that EffTEE reduces inference time by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2 - 60 \times $ </tex-math></inline-formula> while maintaining accuracy comparable to existing secure DNN methods. These findings demonstrate EffTEE’s potential for secure and efficient DNN deployment in resource-constrained environments.

Carbonate rocks are important archives of past ocean conditions as well as hosts of economic resources such as hydrocarbons, water, and minerals. Geologists typically perform compositional analysis of grain, matrix, cement and pore types in order to interpret depositional environments, diagenetic modification, and reservoir quality of carbonate strata. Such information can be obtained primarily from petrographic analysis, a task that is costly, labor-intensive, and requires in-depth knowledge of carbonate petrology and micropaleontology. Recent studies have leveraged machine learning-based image analysis, including Deep Convolutional Neural Networks (DCNN), to automate description, classification and interpretation of thin sections, subsurface core images and seismic facies, which would accelerate data acquisition and reproducibility for these tasks. In carbonate rocks, this approach has been applied primarily to recognize carbonate lithofacies, and no attempt has been made to individually identify and quantify various types of carbonate grains, matrix, and cement. In this study, the applicability and performance of DCNN-based object detection and image classification approaches are assessed with respect to carbonate compositional analysis. The training data comprised of more than 13,000 individually labelled objects from nearly 4000 carbonate petrographic images. The dataset is grouped into six and nine different classes for the image classification and object detection tasks, respectively. Even with a small and relatively imbalanced training set, the DCNN was able to achieve an F1 score of 92% for image classification and 84% mean precision for object detection by combining one-cycle policy, class weight, and label mixup-smoothing methods. This study highlights the inefficiency of image classification as an approach to replicating human description and classification of carbonate petrography. By contrast, DCNN-based object detection appears capable of approaching human speed and accuracy in the area of carbonate petrography because it is able to individually locate and identify different carbonate components with greater cost-efficiency, speed, and reproducibility than conventional (human) petrographic analysis.