High-Speed Combustion and Detonation Project Scaling Up for Mira

science
Calculations including both viscosity and heat conduction

Researchers at the Argonne Leadership Computing Facility (ALCF) are simulating the high-speed combustion and detonation of hydrogen-oxygen mixtures to enable safer and more widespread use of hydrogen as an alternative fuel.

This is one of 16 projects in the ALCF’s Early Science Program (ESP), which is aimed at preparing key scientific applications for the architecture and scale of Mira, Argonne’s new 10-petaflop IBM Blue Gene/Q supercomputer. Using pre-production time on Mira for real scientific problems, these projects vet the system and gather knowledge that will help future projects take full advantage of Mira’s vastly increased power and capabilities when it goes into production later this year.

Mira, the fourth fastest supercomputer in the world, is enabling detailed simulations that will help scientists better understand how high-speed burning of hydrogen leads to detonations. Led by principal investigator Alexei Khokhlov, a professor at the University of Chicago, results from this ALCF project will ultimately feed into the design of safer hydrogen fuel systems, addressing a major impediment to widespread adoption of hydrogen as a nonpolluting energy source to help replace fossil fuels. 

Safety is an issue with hydrogen fuel because hydrogen-air mixtures ignite over an extremely wide range of concentrations, meaning hydrogen leaks can lead to explosions if ignited by a flame or spark. Furthermore, hydrogen-air combustion can also transition to detonations, which are exponentially more destructive. This can have especially catastrophic consequences in a variety of industrial and energy-producing settings, including the production, transportation, and use of hydrogen fuel, and safety on nuclear reactors where hydrogen can be accumulated in cooling pipe systems due to radiolysis of water.

The high-speed burning of hydrogen-oxygen mixtures and the conditions under which deflagrations transition to detonations have been studied for many years. While these experiments have been valuable, they have yet to yield a fundamental understanding of the deflagration-to-detonation transition (DDT). This is because the events occur in microseconds and details of the explosions usually differ from one experimental trial to the next, making it extremely difficult to make diagnostic measurements while turbulent gases explode.

Numerically, researchers have also long sought to study the DDT in terms of first-principles calculations based on the Navier-Stokes equations, which contain most of the physics of fluid motion. These equations, however, are prohibitively difficult to solve when modeling the turbulent, high-speed burning of hydrogen-oxygen mixtures without the aid of high-performance computing. First-principles simulations based on the Navier-Stokes equations have been conducted but in compromised fashion because of a lack of computer power. This has resulted in models with unrealistically undersized versions of DDT experiments, and turbulence being simulated in an artificially constrained manner. Thanks to Mira and the efforts of the support staff at the ALCF, all that has begun to change.

The goal of the ALCF project is to make first-principles simulations of the DDT for hydrogen-oxygen mixtures that burn as they would in typical experiments, which would permit a degree of comparison between the simulations and actual experiments. The modeling runs will also be direct numerical simulations, in that turbulence scales will be fully resolved rather than modeled, which will require the world-class computing capabilities of Mira.

Khokhlov and fellow researcher Joanna Austin, an assistant professor at the University of Illinois at Urbana-Champaign, initiated the project three years ago. The ALCF work began as a Director’s Discretionary program allocation with 500,000 core hours on Intrepid (the ALCF supercomputer that preceded Mira), but has grown into a project that now operates under two allocations, totaling 280 million core hours. The allocations include a U.S. Department of Energy Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program award for 130 million core hours to support modeling efforts, and an ESP allocation for 150 million core hours to shape the details of this code to take full advantage of Mira’s Blue Gene/Q architecture.

As part of the ESP work, ALCF staff rewrote the rebalance algorithm to reduce inefficiencies that appeared at large scale. They also helped port, tune, and scale the code based on how it was behaving, improving the OpenMP implementation and optimizing library use. Argonne researchers Katherine Riley, Charles Bacon, Ben Clifford, and Barry Smith have contributed to the project thus far.

“In a project at this level, you need practical computer science, and that’s what the people at the ALCF provide,” said Khokhlov. “They created several novel algorithms that sped up the code and always remain available to not only help run it on the machine but also to solve all of the algorithm issues that arise.”

“The previous three years were spent validating the code and solving simpler problems, making sure that all of those simpler problems which are part of this big problem were modeled correctly. This would have been totally impossible without the support of the ALCF,” he added.

Khokhlov and his colleagues are nearing the point of being able to run full-physics, high-resolution simulations of the DDT on Mira for the first time. He estimates that the current ESP and INCITE allocations will allow them to demonstrate the capabilities of the simulations on Mira in two runs. Each run will consume about 100 million core hours.

This work is funded by the U.S. Department of Energy’s Office of Science initially under the direction of Lali Chatterjee (Advanced Scientific Computing Research program). Randall Laviolett is the current program manager for the project.