Rethinking I/O in High-Performance Computing Environments

Nawab Ali
Seminar

As the types of problems we solve in high-performance computing and other areas become more complex, the amount of data generated and used is growing at a rapid rate. Today many terabytes of data are common; tomorrow petabytes of data will be the norm. One of the challenges in high-performance computing is to provide users with reliable data access in a distributed, heterogeneous environment. In this talk, we will review the existing I/O paradigms in high-performance computing environments and explore better alternatives across both local and wide-area networks. We propose three different techniques to accommodate the I/O requirements of scientific applications. We present a new design for a high-performance, scalable parallel file system that obviates the need for dedicated I/O and metadata servers by utilizing Object-based Storage Devices. We also propose a new remote I/O paradigm that takes advantage of the increasing popularity of high-speed wide-area networks and centralized data repositories to perform I/O over wide-area networks. Lastly, we present a scalable I/O forwarding solution that attempts to bridge the increasing performance gap between the processing power and the I/O subsystems of massively-parallel leadership-class machines such as the IBM Blue Gene/P.