Self-supervised Neural Architecture Search for Multimodal Deep Neural Networks

Object recognition from RGB-D images has become a hot topic and gained a significant popularity in recent years due to its numerous applications. In this paper, we propose a novel multimodal deep convolutional neural networks architecture for RGB-D object recognition which composed of three streams with two different types of deep CNNs, where each stream can separately learn from each modality. Finally, we propose a combined architecture of joint network of these three streams to classify the objects. Compared to RGB data, RGB-D images provide additional depth information that can be represented as depth colorization methods or surface normals. Our goal is to exploit both colorization and surface normals information to encode depth images. We show that by utilizing both colorization and surface normals of depth images combined with RGB significantly can improves the classification accuracy. We evaluate our model on one of the most challenging RGB-D object dataset and achieves comparable performance to state-of-the-art methods.

With the rapid progress of artificial intelligence (AI) algorithms, multi-modal deep neural networks (DNNs) have been applied to some challenging tasks, e.g., image and video description to process multi-modal information from vision and language. Resistive-memory-based processing-in-memory (ReRAM-based PIM) has been extensively studied to accelerate either convolutional neural network (CNN) or recurrent neural network (RNN). According to the requirements of their core layers, i.e. convolutional layers and linear layers, the existing ReRAM-based PIMs adopt different optimization schemes for them. Directly deploying multi-modal DNNs on the existing ReRAM-based PIMs, however, is inefficient because multi-modal DNNs have combined CNN and RNN where the primary layers differ depending on the specific tasks. Therefore, a high-efficiency ReRAM-based PIM design for multi-modal DNNs necessitates an adaptive optimization to the given network. In this work, we propose HitM, a high-throughput ReRAM-based PIM for multi-modal DNNs with a two-stage workflow, which consists of a static analysis and an adaptive optimization. The static analysis generates the layer-wise resource and computation information with the input multi-modal DNN description and the adaptive optimization produces a high-throughput ReRAM-based PIM design through the dynamic algorithm based on hardware resources and the information from the static analysis. We evaluated HitM using several popular multi-modal DNNs with different parameters and structures and compared it with a naïve ReRAM-based PIM design and an optimal-throughput ReRAM-based PIM design that assumes no hardware resource limitations. The experimental results show that HitM averagely achieves 78.01% of the optimal throughput while consumes 64.52% of the total hardware resources.

ETNAS: An energy consumption task-driven neural architecture search

In the future, vehicles will be equipped with increasingly advanced interactive intelligent electronic devices, which will induce drivers to conduct secondary tasks, thereby leading to distractions. Therefore, the detection and early warning of driver distraction are essential for improving driving safety and pose an important challenge in intelligent transportation systems. Previous studies used traditional machine learning and deep learning transfer models, which have the disadvantages of complicated and time-consuming manual feature engineering, strong subjectivity, and weak generalization performance. In this paper, we propose a fine-grained detection method for driver distraction based on neural architecture search. First, we design an automatic construction algorithm for deep convolutional neural networks based on neural architecture search, which automatically searches for the optimal deep convolutional neural network architecture without human involvement. In addition, we fuse driver-related multisource perception information, use an automatically constructed deep convolutional neural network to extract high-dimensional mapping features, and implement fine-grained detection of various types of driver distraction states. The results on a large-scale multimodal driver distraction dataset demonstrate that our method can efficiently search an optimal deep convolutional neural network, which can quickly converge, and can accurately detect the considered types of driver distraction states, the average detection accuracy reaches 99.7796%; moreover, it has satisfactory robustness.

Universal approximation capability, also referred to as universality, is an important property of deep neural networks, endowing them with the potency to accurately represent the underlying target function in learning tasks. In practice, the architecture of deep neural networks largely influences the performance of the models. However, most existing methodologies for designing neural architectures, such as the heuristic manual design or neural architecture search, ignore the universal approximation property, thus losing a potential safeguard about the performance. In this paper, we propose a unified framework to design the architectures of deep neural networks with a universality guarantee based on first-order optimization algorithms, where the forward pass is interpreted as the updates of an optimization algorithm. The (explicit or implicit) network is designed by replacing each gradient term in the algorithm with a learnable module similar to a two-layer network or its derivatives. Specifically, we explore the realm of width-bounded neural networks, a common practical scenario, showcasing their universality. Moreover, adding operations of normalization, downsampling, and upsampling does not hurt the universality. To the best of our knowledge, this is the first work that width-bounded networks with universal approximation guarantee can be designed in a principled way. Our framework can inspire a variety of neural architectures including some renowned structures such as ResNet and DenseNet, as well as novel innovations. The experimental results on image classification problems demonstrate that the newly inspired networks are competitive and surpass the baselines of ResNet, DenseNet, as well as the advanced ConvNeXt and ViT, testifying to the effectiveness of our framework.

Since mobile food delivery services have become one of the essential issues for the restaurant industry, predicting customer revisits is highlighted as one of the significant academic and research topics. Considering that the use of multimodal datasets has gained notable attention from several scholars to address multiple industrial issues in our society, we introduce CRNet, a multimodal deep convolutional neural network for predicting customer revisits. We evaluated our approach using two datasets [a customer repurchase dataset (CRD) and mobile food delivery revisit dataset (MFDRD)] and two state-of-the-art multimodal deep learning models. The results showed that CRNet obtained accuracies and Fi-Scores of 0.9575 (CRD) and 0.9436 (MFDRD) and 0.9730 (CRD) and 0.9509 (MFDRD), respectively, thus achieving higher performance levels than current state-of-the-art multimodal frameworks (accuracy: 0.7417–0.9012; F1-Score: 0.7461–0.9378). Future research should aim to address other resources that can enhance the proposed framework (e.g., metadata information).

Survey on Deep Neural Networks in Speech and Vision Systems

The estimation of water stress is critical for the reliable production of high-quality fruits cultivated using the tacit knowledge of expert farmers. Multimodal deep neural network has achieved success in the estimation of stem diameter variations as a water stress index, calculated from leaf wilting and environmental data. However, these studies have not addressed the specific role of leaf wilting in the estimation. Revealing the role of leaf wilting not only ensures the reliability of the estimation model but also provides an opportunity for improving the estimation method. In this paper, we investigated the relationships between leaf wilting and stem diameter variations without resorting to black-box approaches such as deep neural network. To clarify the role of leaf wilting, this study uses cross-correlation analysis to analyze the time lag correlation between leaf wilting, quantified by optical flow, and stem diameter variations as a water stress index. The analysis showed that leaf wilting had a significant time lag correlation with short-term stem diameter variations, which were water stress responses in plants. As the results were consistent with known plant water transport mechanisms, it was suggested that leaf wilting quantified by optical flow can explain short-term stem diameter variations.

Recent AI research has witnessed increasing interests in automatically designing the architecture of deep neural networks, which is coined as neural architecture search (NAS). The automatically searched network architectures via NAS methods have outperformed manually designed architectures on some NLP tasks. However, training a large number of model configurations for efficient NAS is computationally expensive, creating a substantial barrier for applying NAS methods in real-life applications. In this paper, we propose to accelerate neural architecture search for natural language processing based on knowledge distillation (called KD-NAS). Specifically, instead of searching the optimal network architecture on the validation set conditioned on the optimal network weights on the training set, we learn the optimal network by minimizing the knowledge loss transferred from a pre-trained teacher network to the searching network based on Earth Mover's Distance (EMD). Experiments on five datasets show that our method achieves promising performance compared to strong competitors on both accuracy and searching speed. For reproducibility, we submit the code at: https://github.com/lxk00/KD-NAS-EMD.

Review for "Identifying chaotic dynamics in noisy time series through multimodal deep neural networks"

Author response for "Identifying chaotic dynamics in noisy time series through multimodal deep neural networks"

Defect recognition is an important task in intelligent manufacturing. Due to the subjectivity of human annotation, the collected defect data usually contains a lot of noise and unpredictable uncertainties, which have a great negative influence on defect recognition. It is a significant challenge to discover an effective defect recognition model with satisfactory uncertainty processing ability. A natural way is to automatically search for an efficient deep model, which can be realized by neural architecture search (NAS). To achieve this, we propose an efficient fuzzy NAS framework for defect recognition, where the searched architecture can effectively handle uncertain information from the given datasets. Specifically, we first design a fuzzy search space and the related encoding strategy for fuzzy NAS. Then, we propose a comparator-based evolutionary search approach, where an online end-to-end comparator is learned to directly determine the selection of candidate architectures from the evolutionary population. The comparator works in an end-to-end way and it transforms the complex ranking problem of evaluating architectures into a simple classification task, which overcomes the rank disorder issue suffered from traditional performance predictors. A series of experimental results demonstrate that the architecture with fewer #Params (1.22 M) search by fuzzy neural architecture search framework for defect recognition method achieves higher accuracy (92.26%) compared to the state-of-the-art results (i.e., DARTS-PV) on the ELPV dataset, as well as competitive results (accuracy = 76.4%, #Params = 1.04 M) on the CODEBRIM dataset. Experimental results show the effectiveness and efficiency of our proposed method in handling uncertain problems.

Decision of the model complexity is a significant challenge in contemporary data-driven modeling applications. Designing neural architecture involves the process of determining the optimal model complexity for deep neural networks (DNNs) models in order to uncover relationships in real-world data patterns. Consequently, optimizing DNN architectures is crucial for enhancing the practical approximation performance of DNN models to real-world data. This study implements a data-driven neuroevolution scheme for the optimal neural architecture search (NAS) and demonstrates a data-driven engineering application for cooling load estimation. The proposed neuroevolution scheme aims at evolving to the best generalizing DNN model that well suits the modeling complexity requirements of the dataset. To this end, the objective function for the neural architecture optimization process is simplified to the mean square error of the test dataset, which enables to reduce the risk of insufficient generalization during NAS. By employing this objective function for evolution field optimization (EFO), the proposed neuroevolution process can automatically achieve the optimal model complexity, preventing overfitting and underfitting cases, and thereby attaining almost the best generalization for the dataset. For this purpose, this approach combines parametric learning with the backpropagation algorithm and structural learning with EFO-based neural architecture search to address data-driven, optimal complexity DNN model generation problems. Effectiveness of the method is demonstrated in the cooling load estimation problem of residential buildings, and performances of the optimal DNN models with four objective functions are analyzed. The design of objective function for the best generalizing model is also elaborated.

A multi-perspective revisit to the optimization methods of Neural Architecture Search and Hyper-parameter optimization for non-federated and federated learning environments

Improving the performance of deep neural networks (DNNs) is important to both the compiler and neural architecture search (NAS) communities. Compilers apply program transformations in order to exploit hardware parallelism and memory hierarchy. However, legality concerns mean they fail to exploit the natural robustness of neural networks. In contrast, NAS techniques mutate networks by operations such as the grouping or bottlenecking of convolutions, exploiting the resilience of DNNs. In this work, we express such neural architecture operations as program transformations whose legality depends on a notion of representational capacity. This allows them to be combined with existing transformations into a unified optimization framework. This unification allows us to express existing NAS operations as combinations of simpler transformations. Crucially, it allows us to generate and explore new tensor convolutions. We prototyped the combined framework in TVM and were able to find optimizations across different DNNs, that significantly reduce inference time - over 3× in the majority of cases. Furthermore, our scheme dramatically reduces NAS search time.