To support user research, ALCF and ANL have developed workflow tools at the lab and in partnership with Globus Labs that run effectively on our machines. A workflow tool is a piece of software that orchestrates the execution of large numbers of tasks on compute resources, handling dependencies, data flows, and errors/timeouts.
Some workflows are agentic. These leverage AI agents to make decisions, perform complex tasks, and adapt to real-time conditions based on application feedback and results. These workflows can reason, plan, and execute tasks based on new results with minimal human intervention.
In a project, a workflow tool can:
- Run many tasks concurrently and/or one after another asynchronously across one or many batch jobs (bundling work into jobs)
- Manage task dependencies
- Automate error handling and restarts of tasks (with agentic workflows)
- Manage data movement into/out of a file system needed for tasks
Agentic Workflows
While LLMs can be useful in research, they struggle with complex topics. Agentic workflows go beyond the question-and-answer format. They can break down complex tasks into discrete steps, where agents are composed to handle specific subtasks and combine together to tackle more complex tasks. Agents can also conduct autonomous or real-time decision making and adaptive execution.
Users can leverage ALCF agentic workflows within our HPC resources to accelerate and scale up their research. The facility supports both single-system and multi-system workflows.
Applications
In recent years, several teams have leveraged integrated workflows to advance data analysis and AI-based discovery efforts. Some examples include:
- Utilizing Globus to enable rapid data analysis to allow APS users to use near real-time feedback from beam lines
- ChemGraph, an AI framework that helps researchers simplify and automate molecular simulation workflows and other computational chemistry tasks
Systems
Globus
Globus is a data management and compute platform that was developed at Argonne and the University of Chicago. It provides an automated backbone for sharing data, running computations, and linking processes across facilities and research teams
Embedding Globus capabilities into research workflows helps integrate computing as a service for research. This service-based model reduces friction for researchers and enables multi-facility science — a core objective of DOE’s Integrated Research Infrastructure program and a foundational capability for the American Science Cloud (AmSC) and the Genesis Mission.