Swift/T: Scalable Dataflow Programming for Composite Applications

Justin M. Wozniak
Seminar

Scripts, workflows, and related techniques are widely used in scientific computing. These tools allow developers to compose complex applications out of existing simulators, analysis codes, or library functions. However, this model is difficult to apply on high-performance systems at large scale. Our recent work has produced a new implementation of Swift that allows users to compose applications from library function calls. By translating the Swift program into an MPI program via the ADLB load balancer, we have been able to run applications on up to 128K cores of the BG/P. In this talk, we will present the new distributed dataflow-based runtime that enables highly scalable script processing, as well as the compiler techniques used.