• Home
  • Search
  • Challenges and Opportunities of User-Level File Systemsfor HPC
  • Cite Icon1
  • https://doi.org/10.2172/1424647Copy DOI Icon

Challenges and Opportunities of User-Level File Systemsfor HPC

  • Aug 23, 2017
  • A Brinkmann +2 more
Show More
  • Abstract
  • Literature Map
  • Citations
  • Similar Papers
Abstract

The performance gap between magnetic disks and data processing on HPC systems has become that huge that an efficient data processing can only be achieved by introducing non-volatile memory (NVRAM) as a new storage tier. Although the benefits of hierarchical storage have been adequately demonstrated to the point that the newest leadership class HPC systems will employ burst buffers, critical questions remain for supporting hierarchical storage systems, including: How should we present hierarchical storage systems to user applications, such that they are easy to use and that application code is portable across systems? How should we manage data movement through a storage hierarchy for best performance and resilience of data? How do the particular I/O use cases mandate the way we manage data? There have been many efforts to explore this space in the form of file systems, with increasingly more implemented at the user level. This is because it is relatively easy to swap in new, specialized user-level file systems for use by applications on a case-by-case basis, as opposed to the current mainstream approach of using general-purpose, system-level file systems which may not be optimized for HPC workloads and must be installed by administrators. In contrast, file systems at the user level can be tailored for specific HPC workloads for high performance and can be used by applications without administrator intervention. Many user-level file system developers have found themselves “having to reinvent the wheel” to implement various optimizations in their file systems. Thus, a main goal of this meeting was to bring together experts in I/O performance, file systems, and storage, and collectively explore the space of current and future problems and solutions for I/O on hierarchical storage systems in order to begin a community effort in enabling user-level file system support for HPC systems. We had a lively week of learning about each other’s approaches as well as unique I/O use cases that can influence the design of a community-driven file and storage system standards. The agenda for this meeting contained talks from participants on the following high level topics: HPC storage and I/O support today; what do HPC users need for I/O; existing userlevel file system efforts; object stores and other alternative storage systems; and components for building user-level file systems. The talks were short and intended to be conversation starters for more in-depth discussions with the whole group. The participants engaged in lengthy discussions on various questions that arose from the talks including: Are we ready to program to a memory hierarchy versus block devices? Are the needs of HPC users reflected in our existing file systems and storage systems? Should we drop or keep POSIX moving forward? What do we mean when we say “user-level file system”? Do we all mean the same thing? How should the IO 500 benchmark be defined so it is fair and useful? and How are stage-in and stage-out actually going to work? The report for this seminar contains a record of the talks from the participants as well as the resulting discussions. Our hope is that the effort initiated during this seminar will result in long-term collaborations that will benefit the HPC community as a whole.

Similar Papers
  • Research Article
  • Citations4

A Fast and Slippery Slope for File Systems

  • Jan 20, 2016
  • ACM SIGOPS Operating Systems Review
  • Ricardo Santana +3
  • Conference Article
  • Citations392

Semantic file systems

  • Sep 01, 1991
  • David K Gifford +3
  • Research Article
  • Citations3

On Space Utilization Enhancement of File Systems for Embedded Storage Systems

  • Apr 11, 2017
  • ACM Transactions on Embedded Computing Systems
  • Tseng-Yi Chen +5
  • Book Chapter
  • Citations1

Feasibility Study of a Future HPC System for Memory Intensive Applications: Conceptual Design of Storage System

  • Nov 03, 2014
  • Ken’Ichi Itakura +5
  • Research Article
  • Citations8

Security constructs for regulatory-compliant storage

  • Jan 01, 2010
  • Communications of the ACM
  • Randal Burns +1
  • Single Report

Composition and Realization of Source-to-Sink High-Performance Flows: File Systems, Storage, Hosts, LAN and WAN

  • Dec 01, 2016
  • Chase Qishi Wu
  • Research Article

Preservation Archiving Storage Technologies

  • Jan 01, 2010
  • Archiving Conference
  • Keith Rajecki
  • Research Article
  • Citations2

APCFS: Autonomous and Parallel Compressed File System

  • Nov 26, 2010
  • International Journal of Parallel Programming
  • Kush K Kella +1
  • Conference Article
  • Citations44

To Zip or not to Zip: effective resource usage for real-time compression

  • Feb 12, 2013
  • Danny Harnik +4
  • Research Article

Technical Insights into the zOS file system and datasets

  • Jan 06, 2021
  • International Journal on Science and Technology
  • Chandra Mouli Yalamanchili -
  • Conference Article
  • Citations64

Adding Secure Deletion to Your Favorite File System

  • Dec 13, 2005
  • N Joukov +1
  • Book Chapter
  • Citations10

Evaluating delayed write in a multilevel caching file system

  • Jan 01, 1996
  • Daniel A Muntz +2
  • Conference Article

NLSS: A near-line storage system design based on the combination of HDFS and ZFS

  • Jan 01, 2016
  • Wei Hu +4
  • Research Article
  • Citations8

A System to Assess Grain Bag Storage Internal Environment

  • Sep 04, 2013
  • Transactions of the ASABE
  • J J Ward +1
  • Conference Article
  • Citations13

ParaTrac

  • Jun 21, 2010
  • Nan Dun +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.