Developments in Parallel File System Techniques and Scientific I/O Libraries

Avery Ching
Seminar

Large-scale computational simulations are an important tool for science. Scientific simulation usually requires parallel storage systems for efficiently handling hundreds of Gigabytes to Petabytes of data. The rich I/O interfaces available in high-level I/O libraries, such as NetCDF or HDF5, and middleware I/O libraries, such as MPI-IO, provide significant opportunities to optimize noncontiguous data access patterns. In this talk, we detail several techniques in both the middleware and file system for improving noncontiguous I/O access.