Deep Learning based applications are becoming increasingly prevalent in embedded devices. There are considerable advantages in deploying the Deep Neural Network (DNN) models directly on the device rather than on the cloud. This presents new challenges as popular networks are computationally complex and have huge memory requirements. While exciting research efforts are ongoing to improve the on-device inference performance, other concerns such as reducing the memory usage of DNNs have taken a backseat. This paper focuses specifically on how to efficiently manage the memory requirements of running Convolutional Neural Networks (CNNs) on-device. We propose a novel solution consisting of 2 steps, fine-grained buffer liveness analysis followed by memory pool allocation. We calculate the lifetime of each required buffer (including internal buffers) of a network and assign an offset to each buffer for sharing a common memory pool, resulting in efficient reuse of memory across buffers. The experimental results show that the proposed solution achieves 18x better memory usage than that of opensource Caffe framework and also up-to 22% improvement over existing state-of-the-art for an individual CNN. The scalability of the solution is showcased by a further reduction in memory usage of 31% achieved for inference pipelines that use multiple neural networks. We also show that the proposed solution achieves the theoretically minimum possible memory usage for the networks considered, using a simpler method than existing solutions.