DeltaFS: Creating Trillions of Files to Accelerating Scientific Discovery
Brad Settlemyer, Los Alamos National Laboratory
In this talk we describe how a user-space distributed file system, DeltaFS, provided Los Alamos National Laboratory the scalable metadata approaches needed to enable the creation of trillions of files in just minutes, and we further describe how creating a trillion files made tracking particles one thousand times faster than existing approaches. As part of our description of the DeltaFS metadata architecture we will describe the scaling problems that arose as we scaled the file system to hundreds of thousands of clients and servers and how we overcame those scaling challenges in deploying DeltaFS across a large portion of the Trinity supercomputer.
Brad Settlemyer is a Senior Scientist in Los Alamos National Laboratory's HPC Design group. He received his Ph.D in computer engineering from Clemson University in 2009 with a research focus on the design of parallel file systems. Before coming to Los Alamos he spent 5 years at Oak Ridge National Laboratory performing research into high-performance storage systems. He currently leads the storage systems research efforts within Los Alamos' Ultrascale Research Center and his team is responsible for designing and deploying state of the art storage systems for enabling scientific discovery. He is the Primary Investigator on projects ranging from ephemeral file system design to archival storage systems using molecular information technology and he has published papers on emerging storage systems, long distance data movement, system modeling, and storage system algorithms.