Enhanced video analysis framework for action detection using deep learning

Human action detection and recognition has become a significant topic in computer vision research over the last two decades. Intelligent techniques (such as machine learning and deep learning) have grown in popularity as a result of technological advancements in visual data. Due to the exponential growth of video data generated by surveillance cameras, intelligent systems are in great demand for detecting specific human activities. In this work, we critically examine the state-of-the-art (SOA) methodologies for human activity detection (HAD) approaches. The study illustrates a generic classification for HAD methods anda paradigm shift is observed from traditional to modern deep learning-based techniques. We also provide an analysis of publicly available datasets for the classification of human activities. The paper outlines various open research issues as well as future directions for HAD methods. Due to problems such as a dynamic and complex background, camera motion, occlusion, and bad weather, it is evident that detecting human behaviors in surveillance videos is a challenging task. Convolution neural networks (CNNs) are used in the bulk of HAD techniques, prompting researchers to look at sequence learning models like Recurrent Neural Networks (RNNs) and Long-Short-Term Neural Networks (LSTM). Furthermore, our analysis indicates that only a few research articles on the detection of anomalous behavior have been published, with the majority of the work focusing on human action detection. Furthermore, existing HAD deep learning models could be improved by including the notion of transfer learning, which saves training time and enhances accuracy.

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

Electricity load forecasting is a crucial undertaking within all the deregulated markets globally. Among the research challenges on a global scale, the investigation of deep transfer learning (DTL) in the field of electricity load forecasting represents a fundamental effort that can inform artificial intelligence applications in general. In this paper, a comprehensive study is reported regarding day-ahead electricity load forecasting. For this purpose, three sequence-to-sequence (Seq2seq) deep learning (DL) models are used, namely the multilayer perceptron (MLP), the convolutional neural network (CNN) and the ensemble learning model (ELM), which consists of the weighted combination of the outputs of MLP and CNN models. Also, the study focuses on the development of different forecasting strategies based on DTL, emphasizing the way the datasets are trained and fine-tuned for higher forecasting accuracy. In order to implement the forecasting strategies using deep learning models, load datasets from three Greek islands, Rhodes, Lesvos, and Chios, are used. The main purpose is to apply DTL for day-ahead predictions (1–24 h) for each month of the year for the Chios dataset after training and fine-tuning the models using the datasets of the three islands in various combinations. Four DTL strategies are illustrated. In the first strategy (DTL Case 1), each of the three DL models is trained using only the Lesvos dataset, while fine-tuning is performed on the dataset of Chios island, in order to create day-ahead predictions for the Chios load. In the second strategy (DTL Case 2), data from both Lesvos and Rhodes concurrently are used for the DL model training period, and fine-tuning is performed on the data from Chios. The third DTL strategy (DTL Case 3) involves the training of the DL models using the Lesvos dataset, and the testing period is performed directly on the Chios dataset without fine-tuning. The fourth strategy is a multi-task deep learning (MTDL) approach, which has been extensively studied in recent years. In MTDL, the three DL models are trained simultaneously on all three datasets and the final predictions are made on the unknown part of the dataset of Chios. The results obtained demonstrate that DTL can be applied with high efficiency for day-ahead load forecasting. Specifically, DTL Case 1 and 2 outperformed MTDL in terms of load prediction accuracy. Regarding the DL models, all three exhibit very high prediction accuracy, especially in the two cases with fine-tuning. The ELM excels compared to the single models. More specifically, for conducting day-ahead predictions, it is concluded that the MLP model presents the best monthly forecasts with MAPE values of 6.24% and 6.01% for the first two cases, the CNN model presents the best monthly forecasts with MAPE values of 5.57% and 5.60%, respectively, and the ELM model achieves the best monthly forecasts with MAPE values of 5.29% and 5.31%, respectively, indicating the very high accuracy it can achieve.

In this paper, we review a variety of deep learning algorithms and models for modulation recognition and classification of wireless communication signals. Specifically, deep learning (DL) has shown overwhelming advantages in computer vision, robotics, and voice recognition. Recently, DL has been proposed to apply to wireless communications for signal detection and classification in order to better learn the active users for electromagnetic spectrum sharing purposes. Therefore, we aim to provide a survey on the most recent techniques which use DL for recognizing and classifying a wireless signal. We focus on the most widely used DL models, emphasize the advantages and limitations, and discuss the challenges as well as future directions. In addition, we also apply a DL algorithm, convolutional neural network (CNN), to demonstrate the feasibility of using CNN to recognize and classify the over-the-air wireless signals using Mathworks DL toolbox with PlutoSDR and Universal Software Radio Peripheral (USRP), respectively.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Deep Learning is not just a keyword abuzz in the industry and academics alike, it has thrown open a whole new field of possibilities. Deep Learning models are being employed in all sorts of use cases and domains, some of which we saw in the previous chapters. Artificial neural networks have tremendous potential to learn complex non-linear functions, patterns, and representations and their power is driving research in multiple fields, including computer vision, audio-visual analysis, chatbots and natural language understanding, to name a few. In this chapter, we touch on some of the advanced areas in the field of computer vision, which have recently come into prominence with the advent of Deep Learning. This includes real-world applications like image categorization and classification and they very popular concept of image artistic style transfer. Computer vision is all about the art and science of making machines understand high-level useful patterns and representations from images and video so that it would be able to make intelligent decisions similar to what a human would do upon observing its surroundings. Building on core concepts like convolutional neural networks and transfer learning, this chapter provides you with a glimpse into the forefront of Deep Learning research with several real-world case studies from computer vision.

Convolutional neural networks have demonstrated state-of-the-art performance in image classification and various other computer vision tasks. Plant disease detection is an important area of deep learning which has been addressed by many recent methods. However, there is a dire need to optimize these solutions for resource-constrained portable devices such as smartphones. This is a challenging problem because deep learning models are resource extensive in nature. This paper proposes an efficient method to systematically classify plant disease symptoms using convolutional neural networks. These networks are memory efficient and when coupled with the proposed training configuration it enables rapid development of industrial applications by reducing the training times. Another critical problem arises with the improper distribution of samples among classes known as the class imbalance problem, which is addressed by employing a simple statistical methodology. Transfer learning is a known technique for training small datasets which transfers pre-trained weights learned on a large dataset. However, during transfer learning, negative transfer learning is a common problem. Therefore, a stepwise transfer learning approach is proposed which can help in fast convergence, while reducing overfitting and preventing negative transfer learning during knowledge transfer across domains. The system is trained and evaluated on two plant disease datasets i.e., PlantVillage (a publicly available dataset) and pepper disease dataset provided by the National Institute of Horticultural and Herbal Science, Republic of Korea. The pepper dataset is particularly challenging as it contains images from different parts of the plant such as the leaf, pulp, and stem. The proposed system has dominated the previous works on the PlantVillage dataset and achieved 99% and 99.69% accuracy on the Pepper dataset and PlantVillage datasets, respectively.

Blind spot detection is an important feature of Advanced Driver Assistance Systems (ADAS). In this paper, we provide a camera-based deep learning method that accurately detects other vehicles in the blind spot, replacing the traditional higher cost solution using radars. The recent breakthrough of deep learning algorithms shows extraordinary performance when applied to many computer vision tasks. Many new convolutional neural network (CNN) structures have been proposed and most of the networks are very deep in order to achieve the state-of-art performance when evaluated with benchmarks. However, blind spot detection, as a real-time embedded system application, requires high speed processing and low computational complexity. Hereby, we propose a novel method that transfers blind spot detection to an image classification task. Subsequently, a series of experiments are conducted to design an efficient neural network by comparing some of the latest deep learning models. Furthermore, we create a dataset with more than 10,000 labeled images using the blind spot view camera mounted on a test vehicle. Finally, we train the proposed deep learning model and evaluate its performance on the dataset.

Indoor activity monitoring methods ensure the well-being and security of elderly and visually impaired people living in their houses. These systems utilize numerous technologies and sensors to monitor day-to-day activities, including medication adherence, movement, and sleep patterns, providing a comprehensive view of the user’s overall health and daily life. The accuracy and adaptability of deep learning (DL) models make human activity recognition (HAR) a valuable tool for enhancing effectiveness, security, and personalized experiences in indoor environments. HAR, utilizing DL techniques, innovates indoor monitoring by enabling precise understanding and detection of human actions. Deep neural networks (DNNs) analyze data from multiple sensors, such as cameras or accelerometers, to distinguish between various action patterns. DL models mechanically remove and learn discriminating features, making them suitable for identifying intricate human activities in sensor data. Nevertheless, selecting the appropriate DL architectures and optimizing their parameters is crucial for achieving improved solutions. This study proposes an Improved Pelican Optimisation for Indoor Activity Recognition in Persons with Disabilities using the Recurrent Neural Network (IPOIAR-DPRNN) method. The primary aim of the IPOIAR-DPRNN method is to enhance indoor activity detection systems for individuals with disabilities. Initially, the image pre-processing stage applies adaptive bilateral filtering (ABF) to reduce unwanted distortions or artefacts in the image. Furthermore, the EfficientNetB7 method is employed for the feature extraction process. For the detection and classification of indoor activities, the bidirectional long short-term memory with multi-head self-attention (BiLSTM-MHSA) technique is used. Additionally, the improved pelican optimization algorithm (IPOA)-based hyperparameter tuning is performed to enhance the detection results of the BiLSTM-MHSA technique. The validation of the IPOIAR-DPRNN approach is examined using the Florence 3D Actions dataset, and the outcomes are measured against various metrics. The comparison study of the IPOIAR-DPRNN approach revealed a superior accuracy value of 97.11% compared to existing techniques.

Plant diseases pose significant economic challenges and may lead to ecological consequences. Although plant pathologists have a significant ability to diagnose plant diseases, rapid, accurate, and early diagnosis of plant diseases by intelligent systems could improve disease control and management. This study evaluates six efficient classification models (classifiers) based on deep learning to detect common tomato diseases by analyzing symptomatic patterns on leaves. Additionally, group learning techniques, including simple and weighted majority voting methods, were employed to enhance classification performance further. Six tomato leaf diseases, including Pseudomonas syringae pv. syringae bacterial spot, Phytophthora infestance late blight, Cladosporium fulvum leaf mold, Septoria lycopersici Septoria leaf spot, Corynespora cassiicola target spot, and Alternaria solani early blight, as well as healthy leaves, resulting in a total of seven classes, were utilized for the classification. Deep learning models, such as convolutional neural networks (CNNs), GoogleNet, ResNet-50, AlexNet, Inception v3, and MobileNet, were utilized, achieving classification accuracies of 65.8%, 84.9%, 93.4%, 89.4%, 93.4%, and 96%, respectively. Furthermore, applying the group learning approaches significantly improved the results, with simple majority voting achieving a classification accuracy of 99.5% and weighted majority voting achieving 100%. These findings highlight the effectiveness of the proposed deep ensemble learning models in accurately identifying and classifying tomato diseases, featuring their potential for practical applications in tomato disease diagnosis and management.

As one of the representative algorithms of deep learning, a convolutional neural network (CNN) with the advantage of local perception and parameter sharing has been rapidly developed. CNN-based detection technology has been widely used in computer vision, natural language processing, and other fields. Fresh fruit production is an important socioeconomic activity, where CNN-based deep learning detection technology has been successfully applied to its important links. To the best of our knowledge, this review is the first on the whole production process of fresh fruit. We first introduced the network architecture and implementation principle of CNN and described the training process of a CNN-based deep learning model in detail. A large number of articles were investigated, which have made breakthroughs in response to challenges using CNN-based deep learning detection technology in important links of fresh fruit production including fruit flower detection, fruit detection, fruit harvesting, and fruit grading. Object detection based on CNN deep learning was elaborated from data acquisition to model training, and different detection methods based on CNN deep learning were compared in each link of the fresh fruit production. The investigation results of this review show that improved CNN deep learning models can give full play to detection potential by combining with the characteristics of each link of fruit production. The investigation results also imply that CNN-based detection may penetrate the challenges created by environmental issues, new area exploration, and multiple task execution of fresh fruit production in the future.

The significant role of emotion in human daily interaction cannot be over-emphasized, however, the pressing demand for a cutting-edge and highly efficient model for the classification of speech emotion in effective computing has remained a challenging task. Researchers have proposed several approaches for speech emotion classification (SEC) in recent times, but the lingering challenge of the insufficient dataset, which has been limiting the performances of these approaches, is still of major concern. Therefore, this work proposes a deep transfer learning model, a technique that has been yielding tremendous and state-of-the-art results in computer vision, for SEC. Our approach used a pre-trained and optimized model of Visual Geometry Group (VGGNet) convolutional neural network architecture with appropriate fine-tuning for optimal performance. The speech signal is converted to a mel-Spectrogram image suitable for deep learning model input (224\(\,\times \,\)244 x 3) using filterbanks and Fast Fourier transform (FFT) on the speech samples. Multi-layer perceptron (MLP) algorithm is adopted as a classifier after feature extraction is carried out by the deep learning model. Speech pre-processing was carried out on Toronto English Speech Set (TESS) speech emotional corpus used for the study to prevent the low performance of our proposed model. The result of our experiment after evaluation using the TESS dataset shows an improved result in SEC with an accuracy rate of 96.1% and specificity of 97.4%.KeywordsDeep learningSpeech emotionClassificationDeep convolutional neural network

Deep learning algorithms can be used to design systems that report data on patients and deliver warnings to medical applications or electronic health information if there are changes in the patient's health. These systems could be created using deep Learning. This helps verify that patients get the proper effective care at the proper time for each specific patient. A healthcare decision support system was presented using the Internet of Things and deep learning methods. In the proposed system, we examined the capability of integrating deep learning technology into automatic diagnosis and IoT capabilities for faster message exchange over the Internet. We have selected the suitable Neural Network structure (number of best-hidden layers and activation function classes) to construct the e-health system. In addition, the e-health system relied on data from doctors to understand the Neural Network. In the validation method, the total evaluation of the proposed healthcare system for diagnostics provides dependability under various patient conditions. Based on evaluation and simulation findings, a dual hidden layer of feed-forward NN and its neurons store the tanh function more effectively than other NN. To overcome challenges, this study will integrate artificial intelligence with IoT. This study aims to determine the NN's optimal layer counts and activation function variations.

Recent advancements in deep learning, particularly convolutional networks, have rapidly become the preferred methodology for analyzing medical images, facilitating tasks like disease segmentation, classification, and pattern quantification. Central to these advancements is the capacity to leverage hierarchical feature representations acquired solely from data. This comprehensive review meticulously examines a variety of deep learning techniques applied across diverse healthcare domains, delving into the intricate realm of medical imaging to unveil concealed patterns through strategic deep learning methodologies. Encompassing a range of diseases, including Alzheimer’s, breast cancer, brain tumors, glaucoma, heart murmurs, retinal microaneurysms, colorectal liver metastases, and more, the analysis emphasizes contributions succinctly summarized in a tabular form.The table provides an overview of various deep learning approaches applied to different diseases, incorporating methodologies, datasets, and outcomes for each condition.Notably, performance metrics such as accuracy, specificity, sensitivity, and other crucial measures underscore the achieved results. Specifically, an in-depth discussion is conducted on the Convolutional Neural Network (CNN) owing to its widespread adoption as a paramount tool in computer vision tasks. Moreover, an exhaustive exploration encompasses deep learning classification approaches, procedural aspects of medical image processing, as well as a thorough examination of key features and characteristics. At the end, we delve into a range of research challenges and put forth potential avenues for future improvements in the field.

Convolutional neural networks (CNNs) have performed exceptionally well in various computer vision tasks. Previously, researchers relied on feature extraction and then classification. With CNNs, feature extraction and classification are performed in a single step. Therefore, incorporating a set of convolutional neural networks, known as an ensemble, can help increase the effectiveness of their behavior. This paper presents an architecture that is the result of collaborative efforts, consisting of three types of CNN: Densenet121, Resnet101, and MobileNetV2. We also present experimental results using the Cifar10 and Cifar100 datasets, achieving impressive classification accuracy of 99% for Cifar10 and 86% for Cifar100. This case study contributes to deep learning optimization and benefits researchers and practitioners looking for optimal approaches in various computer vision applications. We also evaluate the scalability and robustness of our proposed method in the context of different CNN structures by using more than one model.