Overview of Deep Learning

Deep (machine) learning in recent years has significantly increased the predictive modeling strengths for applications in the areas of computer vision, speech recognition, and natural language processing. Being heavily data-driven, conventional general-purpose deep learning methods mainly rely on training data to learn task-specific data representations. In this thesis, we focus on leveraging novel learning algorithms in deep learning to tackle visual recognition challenges at the three key levels -- namely pixel, object, and image levels. Throughout the thesis, we demonstrate how each and every of our proposed methods can address a specific visual recognition problem. Firstly, we apply deep learning to improve the saliency detection performance of deep networks. A coarse saliency map of the input image's salient object is initially predicted by a convolutional-deconvolutional network (or equivalently a fully convolutional network). Following that, the coarse saliency map is sequentially refined by a separate network called Recurrent Attentional Convolutional-Deconvolutional Network (RACDNN) that shares its network weights across its sequential steps. At every sequential (temporal) step, the same RACDNN is applied to a newly selected/attended image sub-region and it performs saliency map refinement for that particular sub-region. Despite having many learnable network parameters and no access to additional training data, RADCNN can be trained well and it outperforms the "single-step" (coarse saliency detection) baseline and state-of-the-art saliency detection methods by large margins, on several saliency detection benchmark datasets. Secondly, we introduce auto-encoder weight transfer networks (AE-WTN+), a deep model for scaling up object detection. Conventional WTN transfers object class knowledge from the classification weights of a pretrained large-scale image classification network, to the classification weights of an object detection network. However, they suffer from under-fitting and are not able to preserve the rich object class information of novel classes. In this work, we propose a new WTN method -- AE-WTN+ which makes use of normalization techniques to ease network optimization. Inputs and intermediate features are normalized to improve trainability for reduced under-fitting. More importantly, a novel auto-encoding (reconstruction) loss is proposed to encourage information preservation of all classes in the weight predictions of AE-WTN+. This enables the predicted weights of AE-WTN+ to maintain the class discriminability and neighbourhood relationships contained in the pre-trained large-scale classification weights. Experiments on large-scale detection datasets validate the effectiveness of AE-WTN+ on both seen and novel classes. Thirdly, we improve the efficiency of cross-layer network connections for very deep convolutional network in the context of image classification. This gives rise to a new convolutional network architecture called Deluge Networks (DelugeNets) which pass the output (feature activations) of any preceding composite network layer as direct inputs to all succeeding composite layers. For the incoming feature activations of preceding composite layers, the succeeding composite layers first apply rather lightweight cross-layer depthwise convolutional weights on them for feature aggregation. The heavy lifting is carried out by the last convolutional weight layer in each preceding composite layer that performs cross-channel convolutions. The proposed technique enables unimpeded information flow among many layers in very deep DelugeNets with great efficiency. Experiments on small-scale and large-scale datasets show that DelugeNets perform competitively to state-of-the-art deep convolutional networks at lower computational and parameter costs. Fourthly, we propose Stochastic Downsampling (SDPoint), a method to overcome a major limitation suffered by conventional deep convolutional networks -- their computational costs during inference are fixed and cannot be changed. SDPoint downsamples the feature maps at a randomly selected point (layer index) in the network hierarchy during training. The many downsampling configurations known as SDPoint instances each entail a unique downsampling point and downsampling ratio. The same convolutional network weights are shared across the different SDPoint instances that downsample feature maps differently. During inference, the computational cost of a network trained with SDPoint can be instantaneously adjusted to fit a given computational budget, by manually selecting an appropriate SDPoint instance. Additionally, sharing weights between different feature map scales provides significant regularization benefits, making the networks trained with SDPoint less sensitive to scales and more competent. In image classification experiments, SDPoint demonstrates significant cost-accuracy and computational advantages, over independently-trained (non-weight-shared) baselines with various inference costs.

Recently, there has been growing attention to the advances in the areas of electronic and biomedical engineering and the great applications that these technologies can offer mainly for health diagnosis and monitoring. In the past decade, deep learning (DL) has revolutionized traditional machine learning (ML) and brought about improved performance in many fields, including image recognition, object detection, speech recognition, and natural language processing. This chapter discusses detection of heart disease using deep learning techniques. Here the input data has been collected based on wearable device-collected data with IoT module. This data has been preprocessed using adaptive histogram normalization, and the authors segment the image based on threshold method using Ostu thresholding technique. The segmented image feature has been extracted using generative adversarial network and classification of extracted features using deep residual network. The experimental analysis is obtained by the proposed GAN_DRN in terms of accuracy as 96%, precision of 85%, recall of 80%, F-1 score of 71%, and AUC of 75%.

Image classification is one of the important problems in the field of machine learning. Deep learning architectures are used in many machine learning applications such as image classification and object detection. The ability to manipulate large image clusters and implement them quickly makes deep learning a popular method in classifying images. This study points out the success of the convolutional neural networks which is the architecture of deep learning, in solving image classification problems. In the study, the convolutional neural network model of the winner of ilsvrc12 competition is implemented. The method distinguishes 1.2 million images with 1000 categories in success. The application is performed with the caffe library, and the image classification process is employed. In the application that uses the speed facility provided by GPU, the test operation is performed by using the images in Caltech-101 dataset.

Contemporary advances in neural networks (NNs) have demonstrated their potential in different applications such as in image classification, object detection or natural language processing. In particular, reconfigurable accelerators have been widely used for the acceleration of NNs due to their reconfigurability and efficiency in specific application instances. To determine the configuration of the accelerator, it is necessary to conduct design space exploration to optimize the performance. However, the process of design space exploration is time consuming because of the slow performance evaluation for different configurations. Therefore, there is a demand for an accurate and fast performance prediction method to speed up design space exploration. This work introduces a novel method for fast and accurate estimation of different metrics that are of importance when performing design space exploration. The method is based on a Gaussian process regression model parametrised by the features of the accelerator and the target NN to be accelerated. We evaluate the proposed method together with other popular machine learning based methods in estimating the latency and energy consumption of our implemented accelerator on two different hardware platforms targeting convolutional neural networks. We demonstrate improvements in estimation accuracy, without the need for significant implementation effort or tuning.

For the past few years, gender classification has been an active area of study and researchers have been putting a lot of effort to contribute quality research in this area. There is a big potential field of study as it can be used in monitoring, surveillance and human-computer interaction. However, there is still a lack of the performance of existing methods on real live images. The rise of deep learning algorithm has been showing a spectacular increase in performance lately. Many difficult tasks involving computer vision, speech recognition, and natural language processing are easily solved with deep learning. Therefore, the approach to deep learning notably growing and this also happens to be on image classification. Gender classification is an important subject in the face recognition process. This paper shows the results of classifying gender using Convolutional Neural Network based Deep Learning architectures using Tensorflow’s Deep Learning framework. We have used models provided by Keras with weights pre-trained on ImageNet. We have made a comparison of the different type of models which includes VGG16, ResNet-50, and MobileNet. Our own database consists of Asian faces inclusive of Malaysians and some Caucasians. Our trained model on a database consisting of 1000 images shows that VGG-16 delivered the highest recognition accuracy.

Deep neural networks (DNNs) have become increasingly effective in difficult machine learning tasks, such as image classification, speech recognition, and natural language processing. Face recognition (FR) using DNNs shows high performance and is widely used in various domains such as payment systems and immigration inspection. However, DNNs are vulnerable to adversarial examples generated by adding a small amount of noise to an original sample, resulting in misclassification by the DNNs. In this study, we attempt to deceive state-of-the-art FR by attaching noise markers on a face in the real world. To deceive an FR model in the real world, we address challenges in the attack process, including selection of locations of noise markers, the differences between colors of digital noise markers and those of noise markers after printing, the differences between the colors of noise markers that are attached to the face and those of noise markers after a picture is taken, and the differences between the locations of digital noise markers and those of noise markers that are attached to the face. In experiments, we generate noise markers considering these challenges and show that state-of-the-art FR can be deceived by attaching a maximum of 10 noise markers to a face. This can cause a security risk for FR models using DNNs.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

Detecting objects from images is a challenging problem in the domain of computer vision and plays a very crucial role for wide range of real-time applications. The ever-increasing growth of deep learning due to availability of large training data and powerful GPUs helped computer vision community to build commercial products and services which were not possible a decade ago. Deep learning architectures especially convolutional neural networks have achieved state-of-the-art performance on worldwide competitions for visual recognition like ILSVRC, PASCAL VOC. Deep learning techniques alleviate the need of human expertise from designing the handcrafted features and automatically learn the features. This resulted into use of deep architectures in many domains like computer vision (image classification, visual recognition) and natural language processing (language modeling, speech recognition). Object detection is one such promising area immensely needed to be used in automated applications like self-driving cars, robotics, drone image analysis. This paper analytically reviews state-of-the-art deep learning techniques based on convolutional neural networks for object detection.

Traffic violation apprehension is one of the traffic problems here in the Philippines. One example is the No Helmet No Ride Law that is implemented but many motorists still choose to ignore. To alleviate the problem the government has offered many solutions, one of which is the No Contact Traffic Apprehension Policy that uses CCTV Monitoring. To further enhance this solution the government has partnered with the De La Salle University to use artificial intelligence in the system. Computer Vision tasks like image classification and object detection can help automate the traffic apprehension system. Image classification and object detection are technologies which are used in computer vision in defining an image or coordinates of an object in an image. In this work, a novel approach to classifying motorcycle riders between wearing a helmet or not will be developed. It will be demonstrated using deep machine learning, specifically convolutional neural network and by utilizing different pre-trained models to a gathered dataset.

There has been a breakthrough in the field of image classification and object detection, owing to the development of GPU and deep learning. However, because of the huge computation of deep learning, it is hard to use the deep learning algorithms in an embedded platform or a mobile device. Therefore, many compression studies have been conducted, and one of the most popular methods is a parameter quantization. In this paper, we propose an adaptive quantization scheme that reduces the loss of accuracy due to the quantization by properly mixing deterministic and stochastic quantization methods, while retaining the characteristics of the hardware-friendly fixed-point quantization method. By applying the proposed method to the weight parameters of image classification and object detection networks, the proposed method shows better mean average precision (mAP) of up to 0.44% in image classification and 0.91 % in object detection.

In the age of pervasive connectivity, cryptography is a vital defensive measure for information security, and the security of cryptographic protection is of critical importance. Deep learning technology has recently made significant strides in areas like image classification and natural language processing, garnering considerable interest. Compared with classic cryptographic algorithms, modern block ciphers are more intricate, and the mappings between plaintext and ciphertext are less distinct, rendering the extraction of plaintext features from ciphertexts by neural networks as almost infeasible. However, the symbiosis of deep learning and traditional differential cryptanalysis holds promise for enhancing crypto-attack performance. Thus, the integration of deep learning theory and methods into the field of cryptography is becoming a significant trend in technological advancement. In this context, cryptanalysis is progressively developing in the direction of intelligence and automation, with an increasing number of researchers employing deep learning to assist in cryptanalytic tasks. This review aims to delve into the current research trends surrounding deep learning-supported differential cryptanalysis. It commences with a thorough recapitulation of differential analysis in cryptography and introduces common models in deep learning, along with their characteristics. Moreover, it encapsulates the design of differential classifiers powered by deep learning, inclusive of various optimization techniques utilized within these algorithms. The paper also posits directions for future research focus. Despite challenges, deep learning possesses vast potential in reinforcing conventional differential cryptanalysis, providing deeper insights for security analysis and response strategies, and serving as a valuable tool and perspective for the design and appraisal of future cryptographic solutions.

Microfluidic-based organs-on-chips (OoCs) are a rapidly developing technology in biomedical and chemical research and have emerged as one of the most advanced and promising in vitro models. The miniaturization, stimulated tissue mechanical forces, and microenvironment of OoCs offer unique properties for biomedical applications. However, the large amount of data generated by the high parallelization of OoC systems has grown far beyond the scope of manual analysis by researchers with biomedical backgrounds. Deep learning, an emerging area of research in the field of machine learning, can automatically mine the inherent characteristics and laws of “big data” and has achieved remarkable applications in computer vision, speech recognition, and natural language processing. The integration of deep learning in OoCs is an emerging field that holds enormous potential for drug development, disease modeling, and personalized medicine. This review briefly describes the basic concepts and mechanisms of microfluidics and deep learning and summarizes their successful integration. We then analyze the combination of OoCs and deep learning for image digitization, data analysis, and automation. Finally, the problems faced in current applications are discussed, and future perspectives and suggestions are provided to further strengthen this integration.

Deep learning has emerged as a state-of-the-art learning technique across a wide range of applications, including image recognition, object detection and localisation, natural language processing, prediction and forecasting systems. With significant applicability, deep learning could be used in new and broader areas of applications, including remanufacturing. Remanufacturing is a process of taking used products through disassembly, inspection, cleaning, reconditioning, reassembly and testing to ascertain that their condition meets new products conditions with warranty. This process is complex and requires a good understanding of the respective stages for proper analysis. Inspection is a critical process in remanufacturing, which guarantees the quality of the remanufactured products. It is currently an expensive manual operation in the remanufacturing process that depends on operator expertise, in most cases. This research investigates the application of deep learning algorithms to inspection in remanufacturing, towards automating the inspection process. This paper presents a novel vision-based inspection system based on deep convolution neural network (DCNN) for eight types of defects, namely pitting, rust, cracks and other combination faults. The materials used for this feasibility study were 100 cm × 150 cm mild steel plate material, purchased locally, and captured using a USB webcam of 0.3 megapixels. The performance of this preliminary study indicates that the DCNN can classify with up to 100% accuracy on validation data and above 96% accuracy on a live video feed, by using 80% of the sample dataset for training and the remaining 20% for testing. Therefore, in the remanufacturing parts inspection, the DCNN approach has high potential as a method that could surpass the current technologies used in the design of inspection systems. This research is the first to apply deep learning techniques in remanufacturing inspection. The proposed method offers the potential to eliminate expert judgement in inspection, save cost, increase throughput and improve precision. This preliminary study demonstrates that deep learning techniques have the potential to revolutionise inspection in remanufacturing. This research offers valuable insight into these opportunities, serving as a starting point for future applications of deep learning algorithms to remanufacturing.

We have used data on patents and publications, and from an Internet-based survey, to analyse corporate technological activities in Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) technologies. Two distinct clusters of firms exist: large firms mainly in telecommunications, desktop computing and consumer electronics; and small firms specialising in speech technologies. The small specialised firms depend heavily on nearby universities and public research institutes and, to some extent, on nearby large firms; their relations with the large firms are complementary as well as competitive. Similar patterns can be observed in other, recently emerging, "new science"-based technologies. Integration between ASR and NLP has so far been weak with the two research communities functioning more or less independently, with the former progressing more rapidly than the latter. Having built technological capabilities in ASR and NLP with a small proportion of their corporate technological resources, the large firms have two options depending on the rate of progress in these technologies (especially NLP) in the future. If it is high, substantial investments (including those in complementary technologies) could open up massive market opportunities. If it is low, modest investments will allow the exploitation of niche markets.

Algorithms based on deep learning are taking off as solutions to several problems in numerous fields. Numerous deep learning architectures like convolutional neural networks, recurrent neural networks, deep belief networks, artificial neural networks and deep neural networks have been used in fields such as audio recognition, machine vision, processing of natural language, recognition of speech, social network filtering, computer vision, translating languages and bioinformatics. One such application of deep learning architectures is object detection. Object detection in satellite and aerial imagery undertakes an important role in a long list of applications and has allured a lot of attention which resulted in exceptional progress in recent years. It has abundant uses in various fields such as surveillance, military, national security, urban planning, environmental applications, traffic control and forestry. Several issues like varying size, background and orientation of the target object make the automatic detection of target in satellite imagery a challenging issue. Recent research in this field has given better speed and accuracy to the detection of the target object. With the rise in the data being generated by the Earth Observation programs, it is becoming much more imperative to find better and efficient ways to exploit this data by improving upon the existing Deep Learning research. In this paper, we bring forth a wide-ranging analysis of the recent advancements in methods of object detection based on deep learning to recognize the objects in satellite imagery.KeywordsDeep learningObject detectionSatellite imageryRemote sensingYou Only Look Once (YOLO)You Only Look Twice (YOLT)Single Shot Detector (SSD)Convolutional Neural Networks (CNN)Faster Region-based Convolutional Neural Networks (Faster RCNN)Hybrid Deep Convolutional Neural Network (HDNN)Satellite Imagery Multiscale Rapid Detection with Windowed Networks (SIMRDWN)