- Conference Article
2
- 10.1109/cipae60493.2023.00014
Adaptive Deep Inference Framework for Cloud-Edge Collaboration
- Aug 26, 2023
- Ze Kou + 3 more +3
The deployment of Deep Neural Networks (DNNs) requires significant computational and storage resources, which is challenging for resource-constrained end devices. To this end, collaborative deep inference is proposed, in which the DNN is divided into two parts and deployed on end devices and cloud respectively. How to find the optimal split point for collaborative deep inference in a changing network environment is a big challenge. It is also important to reduce decision latency for model partition to achieve adaptivity. To address these challenges, in this paper, we propose an Adaptive Partition (AP) framework to achieve adaptive deep inference through cloud-edge collaboration. Considering the inference performance in different network environments, we perform a weighted joint optimization of inference latency and quantization loss so that the focus of the optimization can change adaptively with the network. Then we propose a method to make fast decision on the selection of optimal split point. Extensive experimental evaluations have demonstrated the effectiveness of AP in improving the performance of adaptive collaborative deep inference. Compared with the state-of-the-art methods, AP reduces the decision latency and the total inference latency by up to 65.0x and 2.98x.
Read more