LDA-CNN: Linear Discriminant Analysis Convolution Neural Network for Periocular Recognition in the Wild

Emotions expressed by humans can be identified from facial expressions, speech signals, or physiological signals. Among them, the use of physiological signals for emotion classification is a notable emerging area of research. In emotion recognition, a person’s electrocardiogram (ECG) and galvanic skin response (GSR) signals cannot be manipulated, unlike facial and voice signals. Moreover, wearables such as smartwatches and wristbands enable the detection of emotions in people’s naturalistic environment. During the COVID-19 pandemic, it was necessary to detect people’s emotions in order to ensure that appropriate actions were taken according to the prevailing situation and achieve societal balance. Experimentally, the duration of the emotion stimulus period and the social and non-social contexts of participants influence the emotion classification process. Hence, classification of emotions when participants are exposed to the elicitation process for a longer duration and taking into consideration the social context needs to be explored. This work explores the classification of emotions using five pretrained convolutional neural network (CNN) models: MobileNet, NASNetMobile, DenseNet 201, InceptionResnetV2, and EfficientNetB7. The continuous wavelet transform (CWT) coefficients were detected from ECG and GSR recordings from the AMIGOS database with suitable filtering. Scalograms of the sum of frequency coefficients versus time were obtained and converted into images. Emotions were classified using the pre-trained CNN models. The valence and arousal emotion classification accuracy obtained using ECG and GSR data were, respectively, 91.27% and 91.45% using the InceptionResnetV2 CNN classifier and 99.19% and 98.39% using the MobileNet CNN classifier. Other studies have not explored the use of scalograms to represent ECG and GSR CWT features for emotion classification using deep learning models. Additionally, this study provides a novel classification of emotions built on individual and group settings using ECG data. When the participants watched long-duration emotion elicitation videos individually and in groups, the accuracy was around 99.8%. MobileNet had the highest accuracy and shortest execution time. These subject-independent classification methods enable emotion classification independent of varying human behavior.

In the realm of computer vision, image classification is a critical issue with many applications, including multimedia content analysis, security and surveillance, and medical imaging. The accuracy of image classification algorithms has considerably increased with the development of deep learning. The discipline of image classification has dramatically benefited from the usage of pre-trained Convolutional Neural Network (CNN) models. In this study, we perform image classification on the Wang dataset composed of 1000 images separated into ten categories, using six different pre-trained CNN models. The research made use of pre-trained versions of VGG16, Densenet, Mobilenet, Inception V3, Resnet50, and Xception models. We assessed the model performances in terms of training and testing accuracy. Testing accuracies for ten unique batches of data were calculated and averaged out. Densenet outperformed state-of-the-art models like Xception by a small margin, mainly due to low data availability. The findings of this study show how pre-trained models have advanced the field of image classification and shed light on how well these models perform in diverse image classification tasks. This study can serve as a valuable reference for researchers and practitioners in the field of computer vision and deep learning and help inform their choices when selecting pre-trained models for image classification, depending on their needs, while also considering data availability and computational constraints.

Recently, ocular biometrics in unconstrained environments using images obtained at visible wavelength have gained the researchers’ attention, especially with images captured by mobile devices. Periocular recognition has been demonstrated to be an alternative when the iris trait is not available due to occlusions or low image resolution. However, the periocular trait does not have the high uniqueness presented in the iris trait. Thus, the use of datasets containing many subjects is essential to assess biometric systems’ capacity to extract discriminating information from the periocular region. Also, to address the within-class variability caused by lighting and attributes in the periocular region, it is of paramount importance to use datasets with images of the same subject captured in distinct sessions. As the datasets available in the literature do not present all these factors, in this work, we present a new periocular dataset containing samples from 1122 subjects, acquired in 3 sessions by 196 different mobile devices. The images were captured under unconstrained environments with just a single instruction to the participants: to place their eyes on a region of interest. We also performed an extensive benchmark with several Convolutional Neural Network (CNN) architectures and models that have been employed in state-of-the-art approaches based on Multi-class Classification, Multi-task Learning, Pairwise Filters Network, and Siamese Network. The results achieved in the closed- and open-world protocol, considering the identification and verification tasks, show that this area still needs research and development.

In deep convolutional neural networks (DCNNs), model size and computation complexity are two important factors governing throughput and energy efficiency when deployed to hardware for inference. Recent works on compact DCNNs as well as pruning methods are effective, yet with drawbacks. For instance, more than half the size of all MobileNet models lies in their last two layers, mainly because compact separable convolution (CONV) layers are not applicable to their last fully connected (FC) layers. Also, in pruning methods, the compression is gained at the expense of irregularity in the DCNN architecture, which necessitates additional indexing memory to address nonzero weights, thereby increasing memory footprint, decompression delays, and energy consumption. In this article, we propose cyclic sparsely connected (CSC) architectures, with memory/computation complexity of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(N\log {}N)$ </tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> is the number of nodes/channels given a DCNN layer that, contrary to compact depthwise separable layers, can be used as an overlay for both FC and CONV layers of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(N^{2})$ </tex-math></inline-formula> . Also, contrary to pruning methods, CSC architectures are structurally sparse and require no indexing due to their cyclic nature. We show that both standard convolution and depthwise convolution layers are special cases of the CSC layers, whose mathematical function, along with FC layers, can be unified into one single formulation and whose hardware implementation can be carried out under one arithmetic logic component. We examine the efficacy of the CSC architectures for compression of LeNet, AlexNet, and MobileNet DCNNs with precision ranging from 2 to 32 bits. More specifically, we surge upon the compact 8-bit quantized 0.5 MobileNet V1 and show that by compressing its last two layers with CSC architectures, the model is compressed by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 1.5\times $ </tex-math></inline-formula> with a size of only 873 kB and little accuracy loss. Finally, we design a configurable hardware that implements all types of DCNN layers including FC, CONV, depthwise, CSC-FC, and CSC-CONV indistinguishably within a unified pipeline. We implement the hardware on a tiny Xilinx field-programmable gate array (FPGA) for total on-chip processing of the compressed MobileNet that, compared to the related work, has the highest Inference/J while utilizing the smallest FPGA.

Recently, convolutional neural network (CNN) visual features have demonstrated their powerful ability as a universal representation for various recognition tasks. In this paper, cross-modal retrieval with CNN visual features is implemented with several classic methods. Specifically, off-the-shelf CNN visual features are extracted from the CNN model, which is pretrained on ImageNet with more than one million images from 1000 object categories, as a generic image representation to tackle cross-modal retrieval. To further enhance the representational ability of CNN visual features, based on the pretrained CNN model on ImageNet, a fine-tuning step is performed by using the open source Caffe CNN library for each target data set. Besides, we propose a deep semantic matching method to address the cross-modal retrieval problem with respect to samples which are annotated with one or multiple labels. Extensive experiments on five popular publicly available data sets well demonstrate the superiority of CNN visual features for cross-modal retrieval.

Satellite-Derived Bathymetry (SDB) can be calculated using analytical or empirical approaches. Analytical approaches require several water properties and assumptions, which might not be known. Empirical approaches rely on the linear relationship between reflectances and in-situ depths, but the relationship may not be entirely linear due to bottom type variation, water column effect, and noise. Machine learning approaches have been used to address nonlinearity, but those treat pixels independently, while adjacent pixels are spatially correlated in depth. Convolutional Neural Networks (CNN) can detect this characteristic of the local connectivity. Therefore, this paper conducts a study of SDB using CNN and compares the accuracies between different areas and different amounts of training data, i.e., single and multi-temporal images. Furthermore, this paper discusses the accuracies of SDB when a pre-trained CNN model from one or a combination of multiple locations is applied to a new location. The results show that the accuracy of SDB using the CNN method outperforms existing works with other methods. Multi-temporal images enhance the variety in the training data and improve the CNN accuracy. SDB computation using the pre-trained model shows several limitations at particular depths or when water conditions differ.

As part of an effort to develop intelligent agriculture, new methods for enhancing the quality of vegetables are being continually developed. In recent years, the Convolutional Neural Network (CNN) has shown to be the most successful and extensively used approach for identifying the quality of pre-trained vegetables. However, this method is time-consuming due to the scarcity of truly large, significant datasets. Using a pre-trained CNN model as a feature extractor is a straightforward method for utilizing CNNs' capabilities without investing time in training. While, Support Vector Machine (SVM excels at processing data with tiny dimensions and significantly larger instances. SVM more accurately classifies the flatten/vector feature supplied by the CNN fully connected layer with small dimensions. In addition, implementing Data Augmentation (DA) and Weighted Class (WC) for data variety and class imbalance reduction can improve CNN-SVM performance. The research results show highest accuracy during training always achieves 100% across all experimental options. With an average accuracy of 69.66% in the testing process and 92.51% in the prediction process for all data, the experimental findings demonstrate that CNN-SVM outperforms CNN in terms of accuracy performance in all possible experiments, with or without WC and or DA approach.&#x0D;

A chest x-ray is considered to be a readily available and a reliable and fast method for the diagnosis of the pneumonia. In literature, different object detection algorithm are used to the x-ray images. Since, pneumonia can be highly life threatening, the decisions made by the object detection model can influence the lives of thousands of people. It is imperative that we precisely know the performance and the accuracy of the model so that the stakeholders have a wider view of how trustworthy the results are. There is a need to understand the performance and accuracy of the underlying deep learning model. The architecture of the model such as number of neural layers or the computation cost poses a great impact on the accuracy and efficiency. In this paper, our contribution is twofold. Firstly, we have proposed a new deep learning (DL) strategy for the automated identification of pneumonia employing a convolutional neural network (CNN) with improved accuracy. We have introduced a hybrid model in which we have used a pre-trained CNN model, a vision transformer (ViT) and applied N times and N number of CNN layers to improve the accuracy and computational performance. We have identified a point where the accuracy of the model is highest and computational cost is acceptable. Secondly, we have compared our proposed model with different other classifiers such as You Only Look Once (YOLO) v4, and CNN and its multiple layers. Using a publicly accessible dataset, this study attempted to pre-process the input chest X-ray images in order to detect the presence of pneumonia. The experimental results have revealed that the accuracy level of the proposed hybrid CNN model reached a level of 97.6% on datasets of varying sizes.

Mammography is a gold standard imaging modality and is widely used for breast cancer screening. With recent advances in the field of deep learning, the use of deep convolution neural networks (CNNs) in medical image analysis has become very encouraging. The aim of this study is to exploit CNNs for mass detection in mammograms using pre-trained networks. We use the resnet-50 CNN architecture pre-trained with the ImageNet database to perform mass detection on two publicly available image datasets: CBIS-DDSM and INbreast. We demonstrate that the CNN model pretrained using natural image database (ImageNet) can be effectively finetuned to yield better results, compared to randomly initialized models. Further, the benefit of applying transfer learning on a smaller dataset is demonstrated by using the best model obtained from CBIS-DDSM training to finetune on the INbreast database. We analyzed the adaptability of the CNN’s last fully connected (FC) layer and the all convolutional layers to detect masses. The results showed a testing accuracy of 0.92 and an area under the receiver operating characteristic curve (AUC) of 0.98 for the model finetuned on all convolutional layers, while testing accuracy of 0.86 and AUC=0.93 when the model is trained only on the last FC layer.

Scene classification of high resolution remote sensing images plays an important role for a wide range of applications. While significant efforts have been made in developing various methods for scene classification, most of them are based on handcrafted or shallow learning-based features. In this paper, we investigate the use of deep convolutional neural network (CNN) for scene classification. To this end, we first adopt two simple and effective strategies to extract CNN features: (1) using pre-trained CNN models as universal feature extractors, and (2) domain-specifically fine-tuning pre-trained CNN models on our scene classification dataset. Then, scene classification is carried out by using simple classifiers such as linear support vector machine (SVM). In our work, three off-the-shelf CNN models including AlexNet [1], VGGNet [2], and GoogleNet [3] are investigated. Comprehensive evaluations on a publicly available 21 classes land use dataset and comparisons with several state-of-the-art approaches demonstrate that deep CNN features are effective for scene classification of high resolution remote sensing images.

Recent advancements in technology have made available a variety of image capturing devices, ranging from handheld mobiles to space-grade rovers. This has generated a tremendous visual data, which has made a necessity to organize and understand this visual data. Thus, there is a need to caption thousands of such images. This has resulted in tremendous research in computer vision and deep learning. Inspired by such recent works, we present an image caption generating system that uses convolutional neural network (CNN) for extracting the feature embedding of an image and feed that as an input to long short-term memory cells that generates a caption. We are using two pre-trained CNN models on ImageNet, VGG16 and ResNet-101. Both the models were tested and compared on Flickr8K dataset. Experimental results on this dataset demonstrate that the proposed architecture is as efficient as the state-of-the-art multi-label classification models. Experimental results on public benchmark dataset demonstrate that the proposed architecture performs as efficiently as the state-of-the-art image captioning model.

Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals

This project introduces a real-time sign language detection system powered by Convolutional Neural Networks (CNNs), designed to aid individuals with hearing impairments in communication. Leveraging OpenCV for video capture and hand detection, along with custom modules for hand tracking and image classification, the system seamlessly integrates deep learning methodologies. A pre-trained CNN model, trained on a comprehensive dataset of sign language gestures, forms the core of the system, ensuring accurate classification in real-time. By capturing video frames from a webcam, detecting hands, and processing them through the CNN model, the system provides immediate feedback by overlaying predicted gestures onto the video stream. Through its effective implementation, this project underscores the potential of CNNs in facilitating accessibility and inclusivity, while also paving the way for future enhancements and applications in assistive technology and human-computer interaction. Top of Form Keywords— Real-time, Sign Language Detection, Convolutional Neural Networks (CNNs), Communication, Hearing Impairments, Deep Learning.

The latest advances of deep learning and particularly convolutional neural networks (CNNs) have proven more than once their high accuracy in disease detection. In this chapter, we propose a new deep learning-based approach for COVID-19 detection from chest X-ray images. The proposed approach applies, in an efficient way, the techniques of transfer learning and fine-tuning from pre-trained CNN models (InceptionV3, VGG16, MobileNet, EfficientNet, etc.). The dataset used for our experiments has three classes: normal, COVID-19, and other pathologies. The dataset is split into three sub-sets as follows: 70% for training, 15% for validation, and 15% for the final test. To avoid underfitting or overfitting problems during the training process, we apply regularization techniques (L1 & L2 regularizations, dropout, data augmentation, early stopping, cross-validation, etc.), which help in learning and providing a generalizable solution. As a result, we demonstrate the high efficiency of the proposed CNNs for the detection of COVID-19 from chest X-ray images. A comparison of different architectures shows that VGG16 and MobileNet provide the highest scores: 97.5% and 99.3% of accuracy respectively, 98.7% and 99.3% of sensitivity respectively. In addition, both models provide the scores of 96.3% and 99.2% respectively for specificity. The proposed solution is deployed in the cloud to provide high availability in real time, thanks to a responsive website, and this without the need to download, install, and configure the required libraries.

Action recognition using optimized deep autoencoder and CNN for surveillance data streams of non-stationary environments