Distributed storage system for small files

Abstract

Translate Article

With the vigorous rise of the Internet and the Internet of Things, as well as the rapid development of computer technology and artificial intelligence technology, the storage of massive data, has gradually become an urgent problem in the development of these technologies [1], and most of these data are various small files. Take Facebook as an example, it has stored more than 65 billion pictures, each picture has 4 different sizes, and the peak period of picture visits is 1 million per second [2]. In the process of image-related deep learning model training, a large number of labeled pictures are required, and the pictures are mostly small files of several hundred KB. Taking ResNet as an example, it only accepts picture input with a pixel of 224⁎224 [3]. In order to cope with the growth of massive small files, traditional stand-alone storage simply cannot meet the demand. The paper designs and implements a micro-service style high-availability distributed storage system, which has the following 4 advantages: 1) The easy-to-use service interface provides external file upload, download, and delete APIs. 2) Support the storage of small files in different data formats, and adopt a unified data description method for each format. 3) Data redundancy backup to make file storage persistently available. 4) Multi-point distributed storage, supporting the elastic expansion of storage capacity.