After more than six years of enabling breakthroughs in scientific computing, the Theta supercomputer at the U.S. Department of Energy’s (DOE) Argonne National Laboratory will be retired at the end of the year. 

Launched in July of 2017, the machine delivered 202 million compute hours to more than 600 projects, propelling advances in areas ranging from battery research to fusion energy science.

“Theta was a pivotal system for science at Argonne and beyond,” said Michael Papka, director of the Argonne Leadership Computing Facility (ALCF), a DOE Office of Science user facility. “Not only did Theta deliver on our mission to enable large-scale computational science campaigns, but it was also the supercomputer that continued our transformation into a user facility that supports machine learning and data science methods alongside more traditional modeling and simulation projects.”

Theta’s run as an Argonne supercomputer coincided with the emergence of artificial intelligence (AI) as a critical tool for science. The system provided researchers with a platform that could handle a mix of simulation, AI and data analysis tasks, catalyzing groundbreaking studies across diverse scientific domains. 

Around the same time that Theta made its debut in 2017, the facility launched the ALCF Data Science Program (ADSP) to support high-performance computing (HPC) projects that were employing machine learning and other AI methods to tackle big data challenges. This initiative gave the facility’s data science and learning capabilities a boost while also building up a new community of users.

“The ADSP helped jump-start our efforts to support more data-intensive and AI-driven research,” said Venkat Vishwanath, ALCF data science team lead. “Our staff gained invaluable experience working with projects, tools and technologies at the forefront of using AI for science. Our work with Theta via the ADSP really paved the way to more science communities coming to the ALCF for their big data challenges.”

Theta will be succeeded by Polaris and the Aurora exascale system as the lab’s primary supercomputers for open scientific research. Theta’s Intel architecture and its expansion to include NVIDIA graphics processing units (GPUs) have played a key role in helping the facility and its user community transition to Polaris’ hybrid architecture and Aurora’s cutting-edge Intel exascale hardware.

“We learned a lot from Theta,” said Kalyan Kumaran, ALCF director of technology. “It was our first Intel-based supercomputer so it kicked off our close collaborations with Intel on the path to exascale. The later addition of GPUs gave us a platform that took our AI for science and data-intensive analysis capabilities to new levels.”

Funded by the Coronavirus Aid, Relief and Economic Security (CARES) Act in 2020, the system’s GPU hardware expansion, known as ThetaGPU, was initially dedicated to COVID-19 research. The GPU-powered component was later made available to all research projects. After Theta’s retirement at the end of the year, the ThetaGPU hardware will be repurposed to create a new machine called Sophia for specialized tasks, with a major focus on supporting AI for science. 

Supercomputer, super science

Beyond its powerful hardware, the system’s legacy will be the research breakthroughs it enabled over the years. From detailed molecular simulations to massive cosmological models, Theta supported hundreds of computationally intensive research projects that are only possible at a supercomputing facility like the ALCF.

Theta allowed researchers to perform some of the world’s largest simulations of engines and supernovae. The system powered efforts to model the spread of COVID-19 and assess the energy use of the nation’s buildings. It enabled AI-driven research to accelerate the search for new catalysts and promising drug candidates. Theta also gave industry R&D a boost, helping TAE Technologies inform the design of its fusion energy devices, advancing 3M’s efforts to improve the energy efficiency of a manufacturing process, and generating data to aid ComEd in preparing for the potential impacts of climate change. The list of impactful science projects goes on and on.

“It’s amazing to look back at the breadth of innovative studies carried out on Theta,” said Katherine Riley, director of science at the ALCF. “We had traditional HPC projects that used Theta for extreme-scale simulations, but we also supported some new use cases that were pushing scientific machine learning in exciting new directions.”

One of the pioneering machine learning projects was led by Jacqueline Cole of the University of Cambridge. With support from the ADSP, her team used Theta to speed up the process of identifying new materials for improved solar cells. It began with an effort to sort through hundreds of thousands of scientific journals to collect data on a wide variety of chemical compounds. The team created an automated workflow that combined simulation, data mining and machine learning techniques to zero in on the most promising candidates from a pool of nearly 10,000 compounds. This allowed the researchers to pinpoint five high-performing materials for laboratory testing.

“The advantage of this process is that it takes away the old manual curation of databases, which involves many years’ worth of work, and reduces it to a matter of a few months and, ultimately, a few days,” Cole said.

Simulating supernova explosions is another area of research that benefitted from Theta’s computational muscle. As part of a multi-year project, Adam Burrows of Princeton University used the supercomputer to advance the state of the art in performing supernova simulations in 3D. The team’s work on Theta has included carrying out one of the largest collections of 3D supernova simulations and the longest duration full-physics 3D supernova calculation ever performed. With their time on Theta now coming to an end, the Princeton team plans to continue their work to carry out longer and more detailed 3D supernova simulations on Polaris and soon Aurora. 

“Big iron HPC resources such as Theta, Polaris and Aurora are central to our ability to actually simulate this complicated problem,” Burrows said.

While Theta will retire from its full-time role at the end of the year, the system will support one last research campaign in 2024 before it’s officially powered down. As part of a collaboration between the DOE-supported LSST Dark Energy Science Collaboration and NASA-supported researchers, a multi-institutional team will use Theta to produce 3 million simulated images for the surveys to be conducted by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory. The team will generate a set of overlapping Roman-Rubin time domain surveys at the individual pixel level. These detailed images will enable the exploration of highly impactful joint science opportunities between the two surveys, especially for dark energy studies.

“This will be a very exciting last hurrah for Theta,” Riley said. “We expect the team’s simulations will spur new findings in cosmology and astrophysics for years to come.”