What you need to do
Data management is an essential facet of your allocation award. It is important to familiarize yourself with our file systems and storage and how to transfer and share your data when starting your project.
File Systems and Storage
Our HPC systems store project data in file systems called Eagle and Flare. Eagle and Flare are Lustre file systems mounted as /eagle and /flare, respectively. For more information on the Lustre file systems, consult this document on Lustre File Striping Basics.
For information on the AI Testbed storage systems, refer to the AI Testbed Data Management page.
Our HPC systems also mount a Lustre home file system, either agile-home or gecko-home. The home file system is mounted as /home and should generally be used for small files and any binaries to be run on Polaris or Aurora. The performance of this file system is reasonable, but using it for intensive I/O from the compute nodes is discouraged because I/O from the compute nodes uses the project data file systems, which are fast parallel systems and have far more storage space and greater I/O performance than the home directory space.
The agile-home file system is regularly backed up to tape. The data file system is not backed up. It is the user’s responsibility to ensure that copies of any critical data on the data file system have either been archived to tape or stored elsewhere.
Learn more about our file systems and storage
Data Transfer and Sharing
The ALCF Community Data Co-Op (ACDC) powers data-driven research by providing a platform for data access and sharing, and value-added services for data discovery and analysis. ACDC is a data fabric that allows programmatic access and straightforward large-scale data sharing with collaborators via Globus. This provides a platform to build out different modalities for data access and use, such as indexing of data for discovery, data portals for interactive search and access, and accessible analysis services.
If you haven’t already, please be sure to review Step 6: Moving Project Data for instructions on transferring your data to Globus.
Once your data is on our systems, you can use our file systems to share your data: