Beyond Performance: Designing a Super-Resolution Architecture Search Space and a Hybrid Multi-Objective Approach for Neural Architecture Optimization

ETNAS: An energy consumption task-driven neural architecture search

Neural Architecture Search (NAS) has emerged as a crucial method for automating the design of deep learning models. Despite its potential, NAS frequently requires substantial computational and hardware resources. To mitigate these challenges, transferable NAS (TNAS) has been introduced, leveraging prior NAS results to enhance performance on new tasks. However, existing methods largely focus on knowledge transfer within identical neural search spaces, overlooking the potential for cross-domain transferability. Motivated by this gap, we explore evolutionary TNAS across heterogeneous search spaces by learning common neural representations. In particular, we introduce a novel approach that encodes both operational and topological information of neural architectures into a unified sequence using a simple tokenizer. This sequence is then processed by a variational auto-encoder, with a Transformer-based encoder to capture rich neural representations and a decoder that reconstructs the original sequence. By utilizing these latent representations, we further establish an inter-domain mapping that acts as a bridge, enabling effective explicit solution transfer among diverse search spaces to enhance the evolutionary NAS process. To harness this capability, we develop an evolutionary sequential transfer optimization approach that transfers knowledge during population initialization, providing both flexibility and adaptability. To the best of our knowledge, this work serves as the first attempt in the literature exploring evolutionary TNAS across diverse spaces. Moreover, we demonstrate the utility of our method through comprehensive empirical studies using different architecture spaces, including NAS-Bench-101, NAS-Bench-201, and the DARTS search space. Our results show that the proposed method significantly enhances the adaptability and performance of NAS across varied domains.

Defect recognition is an important task in intelligent manufacturing. Due to the subjectivity of human annotation, the collected defect data usually contains a lot of noise and unpredictable uncertainties, which have a great negative influence on defect recognition. It is a significant challenge to discover an effective defect recognition model with satisfactory uncertainty processing ability. A natural way is to automatically search for an efficient deep model, which can be realized by neural architecture search (NAS). To achieve this, we propose an efficient fuzzy NAS framework for defect recognition, where the searched architecture can effectively handle uncertain information from the given datasets. Specifically, we first design a fuzzy search space and the related encoding strategy for fuzzy NAS. Then, we propose a comparator-based evolutionary search approach, where an online end-to-end comparator is learned to directly determine the selection of candidate architectures from the evolutionary population. The comparator works in an end-to-end way and it transforms the complex ranking problem of evaluating architectures into a simple classification task, which overcomes the rank disorder issue suffered from traditional performance predictors. A series of experimental results demonstrate that the architecture with fewer #Params (1.22 M) search by fuzzy neural architecture search framework for defect recognition method achieves higher accuracy (92.26%) compared to the state-of-the-art results (i.e., DARTS-PV) on the ELPV dataset, as well as competitive results (accuracy = 76.4%, #Params = 1.04 M) on the CODEBRIM dataset. Experimental results show the effectiveness and efficiency of our proposed method in handling uncertain problems.

Parameter-less Pareto local search for multi-objective neural architecture search with the Interleaved Multi-start Scheme

Deep learning has made substantial breakthroughs in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers’ prior knowledge and experience. And due to the limitations of humans’ inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture.Neural Architecture Search(NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. In addition, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

Neural Architecture Search (NAS) can automatically design model architecture with better performance. Current researchers have searched for local architecture similar to block, then stacked to construct entire models, or searched the entire model based on a manually designed benchmark module. There is no method to directly search the architecture of the global(entire) model at the operation level. The purpose of this article is to search the entire model directly in the operation level search space. We analyzed the search space of past methods which searching for local architectures, then a working mode for global model architecture search named CAM is proposed. Proposed CAM decouples the architectural parameters of the entire model which can complete the entire model architecture search with few architecture parameters. In the experiment, the test error 2.68 % in CIFAR-10 is obtained by the proposed method at the global architecture level, which can compare with the stage-of-art local architecture search methods.

Deep learning has been applied in the field of network intrusion detection and has yielded good results. In malicious network traffic classification tasks, many studies have achieved good performance with respect to the accuracy and recall rate of classification through self-designed models. In deep learning, the design of the model architecture greatly influences the results. However, the design of the network model architecture usually requires substantial professional knowledge. At present, the focus of research in the field of traffic monitoring is often directed elsewhere. Therefore, in the classification task of the network intrusion detection field, there is much room for improvement in the design and optimization of the model architecture. A neural architecture search (NAS) can automatically search the architecture of the model under the premise of a given optimization goal. For this reason, we propose a model that can perform NAS in the field of network traffic classification and search for the optimal architecture suitable for traffic detection based on the network traffic dataset. Each layer of our depth model is constructed according to the principle of maximum coding rate attenuation, which has strong consistency and symmetry in structure. Compared with some manually designed network architectures, classification indicators, such as Top-1 accuracy and F1 score, are also greatly improved while ensuring the lightweight nature of the model. In addition, we introduce a surrogate model in the search task. Compared to using the traditional NAS model to search the network traffic classification model, our NAS model greatly improves the search efficiency under the premise of ensuring that the results are not substantially different. We also manually adjust some operations in the search space of the architecture search to find a set of model operations that are more suitable for traffic classification. Finally, we apply the searched model to other traffic datasets to verify the universality of the model. Compared with several common network models in the traffic field, the searched model (NAS-Net) performs better, and the classification effect is more accurate.

On neural architecture search and hyperparameter optimization: A max-flow based approach.

By labelling high spatial resolution (HSR) images with specific semantic classes according to geographical properties, scene classification has been proven to be an effective method for HSR remote sensing image semantic interpretation. Deep learning is widely applied in HSR remote sensing scene classification. Most of the scene classification methods based on deep learning assume that the training datasets and the test datasets come from the same datasets or obey similar feature distributions. However, in practical application scenarios, it is difficult to guarantee this assumption. For new datasets, it is time-consuming and labor-intensive to repeat data annotation and network design. The neural architecture search (NAS) can automate the process of redesigning the baseline network. However, traditional NAS lacks the generalization ability to different settings and tasks. In this paper, a novel neural network search architecture framework—the spatial generalization neural architecture search (SGNAS) framework—is proposed. This model applies the NAS of spatial generalization to cross-domain scene classification of HSR images to bridge the domain gap. The proposed SGNAS can automatically search the architecture suitable for HSR image scene classification and possesses network design principles similar to the manually designed networks. To obtain a simple and low-dimensional search space, the traditional NAS search space was optimized and the human-the-loop method was used. To extend the optimized search space to different tasks, the search space was generalized. The experimental results demonstrate that the network searched by the SGNAS framework with good generalization ability displays its effectiveness for cross-domain scene classification of HSR images, both in accuracy and time efficiency.

Hyperspectral image classification (HSIC) is a key task in the field of remote sensing, but the complex nature of hyperspectral data poses a serious challenge to traditional methods. Although deep learning significantly improves classification performance through automatic feature extraction, manually designed network architectures suffer from issues such as dependence on expert experience and lack of flexibility. Neural architecture search (NAS) provides new ideas for HSIC through automated network structure optimization. This article systematically reviews the application progress of NAS in HSIC: firstly, the core components of NAS are analyzed, and the characteristics of various methods are compared from three aspects: search space, search strategy, and performance evaluation. Furthermore, the focus is on exploring NAS technology based on convolutional neural networks, covering 1D, 2D, and 3D convolutional architectures and their innovative integration with various technologies, revealing the advantages of NAS in HSIC. However, NAS still faces challenges such as high computing resource requirements and insufficient interpretability. This article systematically reviews the application of NAS in the field of HSIC for the first time, facilitating readers to quickly understand the development process of NAS in HSIC and the advantages and disadvantages of various technologies, proposing possible future research directions.

Differentiable architecture search has demonstrated promising results in automatically designing neural network architectures with desired properties, such as high accuracy and low FLOPs. However, it suffers from a cumbersome training process, and the injection of constraints in the search phase often relies on some hand-crafted heuristic regularizers, the design of which typically requires tremendous human effort. In this paper, to address these critical challenges, we present FGA-NAS, an efficient method for resource-constrained architecture search. First, to reduce the computational cost and improve search flexibility, we propose a novel condensed search space that merges multiple parallel-placed candidates into a single one. Second, to enable the gradient-based optimization for neural architecture search (NAS) under multiple combinatorial constraints, we decompose the constrained NAS into a few simple sub-problems without introducing any heuristics by using the ADMM algorithm [1]. Then, the constrained NAS can be resolved by alternately solving the simple sub-problems. Experimental results on ImageNet show that our method can discover efficient and accurate neural network architectures that achieve the state-of-the-art by only using 0.2 GPU days.

Deep learning methods, especially deep convolutional neural networks (DCNNs), have been widely used in high-resolution remote sensing image (HRSI) semantic segmentation. In literature, most successful DCNNs are artificially designed through a large number of experiments, which often consume lots of time and depend on rich domain knowledge. Recently, neural architecture search (NAS), as a direction for automatically designing network architectures, has achieved great success in different kinds of computer vision tasks. For HRSI semantic segmentation, NAS faces two major challenges: (1) The task’s high complexity degree, which is caused by the pixel-by-pixel prediction demand in semantic segmentation, leads to a rapid expansion of the search space; (2) HRSI semantic segmentation often needs to exploit long-range dependency (i.e., a large spatial context), which means the NAS technique requires a lot of display memory in the optimization process and can be tough to converge. With the aforementioned considerations in mind, we propose a new decoupling NAS (DNAS) framework to automatically design the network architecture for HRSI semantic segmentation. In DNAS, a hierarchical search space with three levels is recommended: path-level, connection-level, and cell-level. To adapt to this hierarchical search space, we devised a new decoupling search optimization strategy to decrease the memory occupation. More specifically, the search optimization strategy consists of three stages: (1) a light super-net (i.e., the specific search space) in the path-level space is trained to get the optimal path coding; (2) we endowed the optimal path with various cross-layer connections and it is trained to obtain the connection coding; (3) the super-net, which is initialized by path coding and connection coding, is populated with kinds of concrete cell operators and the optimal cell operators are finally determined. It is worth noting that the well-designed search space can cover various network candidates and the optimization process can be done efficiently. Extensive experiments on the publicly open GID and FU datasets showed that our DNAS outperformed the state-of-the-art methods, including artificial networks and NAS methods.

A surrogate evolutionary neural architecture search algorithm for graph neural networks

Facing complex and diverse action recognition scenarios, artificially designed neural networks show poor generalization performance. Therefore, an automatic designing method of 3D convolutional neural networks based on neural architecture search is proposed. Firstly, a variety of human behaviors are extracted to construct training sets and validation sets. Additionally, the weights of neuron connections are updated using the loss on the training set, the discrete network architecture search space is continuous through continuous relaxation, and the search space is reduced by using the hierarchical idea. What’s more, the objective function is optimized by combining gradient descent, realizing fast search, and stacking the obtained computing units to form an overall network at the same time. Evaluations based on public data sets show that the designed neural network model achieves comparable performance to the artificially designed network model in the task of human action recognition, solving the difficulty of deep learning method migration between working at different scenarios by automatically customizing the network.

The Neural Architecture Search (NAS) enables Deep Learning engineers to automatically explore the wide variety of all possible neural architectures to design the solution for a given problem. Though the NAS methods automatically generate the best possible architecture, it is time-consuming, and resource constraint since the search space consists of the best architectures(good accuracy) and includes non performing architectures as well. Researchers have proposed various NAS methods especially based on evolutionary algorithms which consists of diverse search space due to the presence of genetic operations leading to expensive search time cost. These NAS methods also deal with multiple objectives like FLOPS, latency and power usage, and model accuracy, for image classification tasks. Though the generated architectures from NAS have multi-objective, they require significant computing resources and are not suitable for running on edge devices with limited computational resources. The proposed method includes a novel model optimization method designed for the neural architecture generated from NAS methods to make it smaller in size, memory efficient, and faster at inference with minimal loss of accuracy. We optimized the models designed for standard datasets and decreased the parameters by 7.82% for CIFAR-10, 2.95% for CIFAR-100, 6.36% for STL-10, and 3.57% for Food-101 datasets. The resulting models have achieved an accuracy of 97.4% for CIFAR-10, 86% for CIFAR-100, 96.7% for STL-10, and 87.4% for Food-101 datasets.