Multi-Layer Neural Network Auto Encoders Learning Method, using Regularization for Invariant Image Recognition

Abstract. Beach litter destroys marine ecosystems and creates aesthetic discomfort that lowers the value of the beach. In order to solve this beach litter problem, it is necessary to study the generation and distribution pattern of waste and the cause of the inflow. However, the data for the study are only sample data collected in some areas of the beach. Also, most of the data covers only the total amount of beach litter. UAV(Unmanned Aerial Vehicle) and Deep Neural Network can be effectively used to detect and monitor beach litter. Using UAV, it is possible to easily survey the entire beach. The Deep Neural Network can also identify the type of coastal litter. Therefore, using UAV and Deep Neural Network, it is possible to acquire spatial information by type of beach litter. This paper proposes a Beach litter detection algorithm based on UAV and Deep Neural Network and a Beach litter monitoring process using it. It also offers optimal shooting altitude and film duplication to detect small beach litter such as plastic bottles and styrofoam pieces found on the beach. In this study, DJI Mavic 2 Pro was used. The camera on the UAV is a 1-inch CMOS with a resolution of 20MP. The images obtained through UAV are produced as orthoimages and input into a pre-trained neural network algorithm. The Deep Neural Network used for Beach litter detection removed the Fully Connected Layer from the Convolutional Neural Network for semantic segmentation.

Unmanned Aerial Vehicles (UAV) have become an integral part of life, from military operations to entertainment. With this popularity, the interest of researchers on these UAVs has gradually increased. The use of UAVs in many areas and the researches carried out by researchers have revealed the limitations of these devices. At the beginning of these restrictions is the passage through the narrow space. In this study, longitudinal flight control of a hexarotor UAV is discussed with morphing. The limitation of this study is that it is difficult to estimate the moment of inertia and proportional integral derivative (PID) coefficient values according to the arm length, since there is a change in the hexarotor arm lengths with the morphing and the rigid body model changes accordingly. In this study, it is aimed to obtain these parameters with the Deep Neural Network(Artificial Neural Network(ANN) with two or more hidden layers) to overcome this problem. 15 drawings of hexarotor morphing states were drawn in Solidworks program. A training set was created by obtaining the PID coefficients for the longitudinal flight of these drawings from the Matlab/Simulink program. The training set was taught to the Deep Neural Network and moments of inertia and PID coefficients were obtained according to the arbitrarily estimated arm extension or shortening rates. In addition, the hexarotor dynamic model was derived according to the Newton-Euler approach and modeled using the state space model. The longitudinal flight of the hexarotor is simulated with the state space model. The moment of inertia and PID coefficients were estimated by Deep Neural Network according to the values determined by the program randomly together with the initial state. Simulations were made with these parameters and the results were given in graphics.

In this study, the deep multi-layered group method of data handling (GMDH)-type neural network algorithm using revised heuristic self-organization method is proposed and applied to medical image diagnosis of liver cancer. The deep GMDH-type neural network can automatically organize the deep neural network architecture which has many hidden layers. The structural parameters such as the number of hidden layers, the number of neurons in hidden layers and useful input variables are automatically selected to minimize prediction error criterion defined as Akaike’s information criterion (AIC) or prediction sum of squares (PSS). The architecture of the deep neural network is automatically organized using the revised heuristic self-organization method which is a type of the evolutionary computation. This new neural network algorithm is applied to the medical image diagnosis of the liver cancer and the recognition results are compared with the conventional 3-layered sigmoid function neural network.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Deep Learning, developed from the multi-layer ANN concept, is a machine learning method that compiles every detail of the learning process to obtain more abstract data, multi-level data, and more complex data features through the composition of various mathematical functions. Deep MLP, another name for DNN (Deep Neural Network), is MLP, which has more than three layers. DNN's ability will increase in solving problems when more layers are used. Another advantage of DNN is the variety of layer types used, including fully connected layers, convolution layers, softmax layers, recurrent layers, and others. Autoencoder is a type of ANN that trained to reconstruct input patterns in such a way that the output of the deepest hidden layer is a vector resulting from the reduction of dimensions of the input pattern. This study proposes a novel DNN architecture called Deep Auto-Encoder Semi Convolutional Neural Network (DAESCNN) to improve the performance of conventional ANN performance. Annual rainfall data will be used to test forecast performance using DAESCNN. In this study, annual rainfall data were obtained from weather stations of Samarinda city, East Kalimantan, Indonesia in the period 2006–2016. In this study, all 2006–2014 data points were used as training input data, all 2007–2015 data points were used as training target data, while 2016 data points were used as validation of training outcomes. The DAESCNN training process executed through a program built using MATLAB. The training process also carried out with varying training parameters to show its performance. The results shown that DAESCNN generally has excellent performance, is above 99%.

Deep convolutional neural network optimized with hybrid marine predator’s and nomadic people optimization for cardiac arrhythmia classification using ECG signals

Over the past few decades, research has focused heavily on automatic speech recognition (ASR). Although ASR for a few languages is close to reality; ASR for the low resource languages like Malayalam is still in its infancy. Here, in this work, the authors discuss the experiment conducted on accented data of Malayalam speech using two approaches. One approach uses the spectral features for modeling using deep convolutional neural network and the other uses the influential features of speech signals for modeling using LSTM-RNN approach. The proposed methodology comprises three distinct stages; dataset preparation, feature extraction, classification, and hence the construction of deep learning models that recognize the accent-based spoken sentences in the Malayalam language. Mel-frequency cepstral coefficient (MFCC) algorithm, short-term Fourier transform (STFT), and mel spectrogram methodologies are used for feature engineering and hence the features that represent the speech signals are used for constructing the accented ASR system for the Malayalam language using long short-term memory (LSTM) a recurrent neural network (RNN). The spectrogram dataset has been constructed for the speech dataset and used for constructing the ASR model with deep convolutional neural network (DCNN). The result of the experiment shows that LSTM-based RNN outperforms DCNN for the proposed dataset that has been constructed for the experiment in the natural recording environment.

This study presents a fault diagnosis system for vehicle heating, ventilation and air conditioning (HVAC) acoustic signal with various feature extractions in deep learning neural network. Traditionally, sound used for fault diagnosis or signal classification is observed the difference of energy in time or frequency domains. Unfortunately, the frequency smearing effect often arises in some critical conditions. In the present study, discrete wavelet transform (DWT) and wavelet packet transform (WPT) are proposed in fault diagnosis. Meanwhile, when using mechanical learning methods, the data are relatively large, in order to reduce the amount of data, DWT and WPT low-frequency decomposition could be used to improve the performance. Furthermore, the signal characteristics more comprehensive, this study attempts to use the feature extraction method of wavelet packet conversion to improve the signal characteristics. In the experiment process, the operation state of the blade blower in the vehicle air conditioner, four different faults were designed, test database was established through sound to classify, and identify the data using deep neural networks to achieve the purpose of blower fault diagnosis. In data analysis, the original signal is presented through wavelet packet decomposition and discrete packet conversion technology, compared with traditional time and frequency domain signals to explore the identification rate, identification speed and related issues. Experimental results show that using WPT combined with deep neural networks have good fault diagnosis and discrimination capabilities, training, and identification time is shorter than time-frequency domain signals.

The distribution of pulmonary vessels in computed tomography (CT) and computed tomography angiography (CTA) images of lung is important for diagnosing disease, formulating surgical plans and pulmonary research. Based on the pulmonary vascular segmentation task of International Symposium on Image Computing and Digital Medicine 2020 challenge, this paper reviews 12 different pulmonary vascular segmentation algorithms of lung CT and CTA images and then objectively evaluates and compares their performances. First, we present the annotated reference dataset of lung CT and CTA images. A subset of the dataset consisting 7,307 slices for training and 3,888 slices for testing was made available for participants. Second, by analyzing the performance comparison of different convolutional neural networks from 12 different institutions for pulmonary vascular segmentation, the reasons for some defects and improvements are summarized. The models are mainly based on U-Net, Attention, GAN, and multi-scale fusion network. The performance is measured in terms of Dice coefficient, over segmentation rate and under segmentation rate. Finally, we discuss several proposed methods to improve the pulmonary vessel segmentation results using deep neural networks. By comparing with the annotated ground truth from both lung CT and CTA images, most of 12 deep neural network algorithms do an admirable job in pulmonary vascular extraction and segmentation with the dice coefficients ranging from 0.70 to 0.85. The dice coefficients for the top three algorithms are about 0.80. Study results show that integrating methods that consider spatial information, fuse multi-scale feature map, or have an excellent post-processing to deep neural network training and optimization process are significant for further improving the accuracy of pulmonary vascular segmentation.

Video hyperlinking represents a classical example of multimodal problems. Common approaches to such problems are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially deep autoencoders, have proven promising both for crossmodal translation and for early fusion via multimodal embedding. A particular architecture, bidirectional symmetrical deep neural networks, have been proven to yield improved multimodal embeddings over classical autoencoders, while also being able to perform crossmodal translation. In this work, we focus firstly at evaluating good single-modal continuous representations both for textual and for visual information. Word2Vec and paragraph vectors are evaluated for representing collections of words, such as parts of automatic transcripts and multiple visual concepts, while different deep convolutional neural networks are evaluated for directly embedding visual information, avoiding the creation of visual concepts. Secondly, we evaluate methods for multimodal fusion and crossmodal translation, with different single-modal pairs, in the task of video hyperlinking. Bidirectional (symmetrical) deep neural networks were shown to successfully tackle downsides of multimodal autoencoders and yield a superior multimodal representation. In this work, we extensively tests them in different settings, with different single-modal representations, within the context of video-hyperlinking. Our novel bidirectional symmetrical deep neural networks are compared to classical autoencoders and are shown to yield significantly improved multimodal embeddings that significantly (alpha=0.0001) outperform multimodal embeddings obtained by deep autoencoders with an absolute improvement in precision at 10 of 14.1% when embedding visual concepts and automatic transcripts and an absolute improvement of 4.3% when embedding automatic transcripts with features obtained with very deep convolutional neural networks, yielding 80% of precision at 10.

Intense short-wavelength pulses from free-electron lasers and high-harmonic-generation sources enable diffractive imaging of individual nanosized objects with a single x-ray laser shot. The enormous data sets with up to several million diffraction patterns present a severe problem for data analysis because of the high dimensionality of imaging data. Feature recognition and selection is a crucial step to reduce the dimensionality. Usually, custom-made algorithms are developed at a considerable effort to approximate the particular features connected to an individual specimen, but because they face different experimental conditions, these approaches do not generalize well. On the other hand, deep neural networks are the principal instrument for today's revolution in automated image recognition, a development that has not been adapted to its full potential for data analysis in science. We recently published [Langbehn etal., Phys. Rev. Lett. 121, 255301 (2018)PRLTAO0031-900710.1103/PhysRevLett.121.255301] the application of a deep neural network as a feature extractor for wide-angle diffraction images of helium nanodroplets. Here we present the setup, our modifications, and the training process of the deep neural network for diffraction image classification and its systematic bench marking. We find that deep neural networks significantly outperform previous attempts for sorting and classifying complex diffraction patterns and are a significant improvement for the much-needed assistance during postprocessing of large amounts of experimental coherent diffraction imaging data.

Deep learning models are highly parameterized, resulting in difficulty in inference and transfer learning for image recognition tasks. In this work, we propose a layered pathway evolution method to compress a deep convolutional neural network (DCNN) for classification of masses in digital breast tomosynthesis (DBT). The objective is to prune the number of tunable parameters while preserving the classification accuracy. In the first stage transfer learning, 19 632 augmented regions-of-interest (ROIs) from 2454 mass lesions on mammograms were used to train a pre-trained DCNN on ImageNet. In the second stage transfer learning, the DCNN was used as a feature extractor followed by feature selection and random forest classification. The pathway evolution was performed using genetic algorithm in an iterative approach with tournament selection driven by count-preserving crossover and mutation. The second stage was trained with 9120 DBT ROIs from 228 mass lesions using leave-one-case-out cross-validation. The DCNN was reduced by 87% in the number of neurons, 34% in the number of parameters, and 95% in the number of multiply-and-add operations required in the convolutional layers. The test AUC on 89 mass lesions from 94 independent DBT cases before and after pruning were 0.88 and 0.90, respectively, and the difference was not statistically significant (p > 0.05). The proposed DCNN compression approach can reduce the number of required operations by 95% while maintaining the classification performance. The approach can be extended to other deep neural networks and imaging tasks where transfer learning is appropriate.

Brain tumor implies development of strange cells in brain. In cutting edge stages, brain tumor is most risky infection which can't be relieved. Thus, it ought to be identified in the beginning phases with the assistance of MRI (Magnetic Resonance Image). So, there will be more changes to the patient to endure. Quite possibly, the most functional and significant technique is to utilize Deep Neural Network (DNN). In this paper, a Deep Convolutional Neural Network (DCNN) based on VGG16 model has been developed to identify a tumor through brain Magnetic Resonance Imaging (MRI) dataset. In the clinical field, the strategies of machine learning (ML) and data mining hold a critical stand. It is effectively used to achieve the efficiency and exact location of tumor. The proposed technique involves automatic segmentation method based on Deep Convolution neural network (DCNN). It is layer based segmentation and classification technique. Different levels are engaged with the proposed technique, first step is data collection and then pre-processing & average filtering is done, after that segmentation, feature extraction and classification via DCNN is performed. In this paper, a fresh technique for classification of brain MRI images has been proposed using DCNN based softmax Classifier. The proposed system has been compared with the existing ones. The results are really encouraging as the loss is reduced and accuracy is increased. The proposed system achieved the training accuracy of 98.9% and validation accuracy of 100%. The training loss is reduced up to 0.0230 and the validation loss reduced up to 0.0109.

The application of deep learning, specifically deep convolutional neural networks (DCNNs), to the classification of remotely-sensed imagery of natural landscapes has the potential to greatly assist in the analysis and interpretation of geomorphic processes. However, the general usefulness of deep learning applied to conventional photographic imagery at a landscape scale is, at yet, largely unproven. If DCNN-based image classification is to gain wider application and acceptance within the geoscience community, demonstrable successes need to be coupled with accessible tools to retrain deep neural networks to discriminate landforms and land uses in landscape imagery. Here, we present an efficient approach to train/apply DCNNs with/on sets of photographic images, using a powerful graphical method called a conditional random field (CRF), to generate DCNN training and testing data using minimal manual supervision. We apply the method to several sets of images of natural landscapes, acquired from satellites, aircraft, unmanned aerial vehicles, and fixed camera installations. We synthesize our findings to examine the general effectiveness of transfer learning to landscape-scale image classification. Finally, we show how DCNN predictions on small regions of images might be used in conjunction with a CRF for highly accurate pixel-level classification of images.

The training process of a deep neural network commonly consists of three phases: forward propagation, backward propagation, and weight update. In this paper, we propose a hardware architecture to accelerate the backward propagation. Our approach applies to neural networks that use rectified linear unit. Considering that the backward propagation results in a zero activation gradient when the corresponding activation is zero, we can safely skip the gradient calculation. Based on this observation, we design an efficient hardware accelerator for training deep neural networks by selectively computing gradients. We show the effectiveness of our approach through experiments with various network models.