Lustre: The Filesystem That Feeds Supercomputers
Summary
Frontier, one of the fastest supercomputers on Earth, runs on Lustre, an open-source parallel filesystem most GPU teams have never looked inside. Its core move is splitting the two jobs of a filesystem: metadata servers own the map (names, directories, permissions, layout) while object storage servers own the bytes, and once a client has the layout it talks straight to the data servers with the metadata server out of the I/O path. Striping cuts one file across up to 2,000 storage targets so reads pull from every target at once. Orion, Frontier's Lustre filesystem, holds 679 petabytes across 1,350 targets and reads 10 terabytes a second off its flash tier, with a dedicated tier just for metadata. In a training stack, object storage holds canonical bytes, a defined dataset picks what matters, and a Lustre-class tier feeds the GPUs. Go deeper: lustre.org
About this video
Frontier, one of the fastest supercomputers on Earth, runs on Lustre, an open-source parallel filesystem most GPU teams have never looked inside. Its core move is splitting the two jobs of a filesystem: metadata servers own the map (names, directories, permissions, layout) while object storage servers own the bytes, and once a client has the layout it talks straight to the data servers with the metadata server out of the I/O path. Striping cuts one file across up to 2,000 storage targets so reads pull from every target at once. Orion, Frontier's Lustre filesystem, holds 679 petabytes across 1,350 targets and reads 10 terabytes a second off its flash tier, with a dedicated tier just for metadata. In a training stack, object storage holds canonical bytes, a defined dataset picks what matters, and a Lustre-class tier feeds the GPUs. Go deeper: lustre.org