ThetaGPU has AMD processors on the service nodes (thetagpusn1,2) and AMD processors and NVIDIA A100 GPUs on the compute nodes [see overview page]. The service nodes can be used to create containers and launch jobs, and eventually to use as a cross-compiling environment for compute nodes. Until the cross-compiling environment is set up, the compute nodes will have to be used for compiling. This can be done by using an interactive Cobalt job (via qsub -I), or until we have reserved or added a dedicated build node.
The default programming environment on the ThetaGPU compute nodes is the GNU compiler tools coupled with NVIDIA’s CUDA toolkit.
Note: Symlinks to the project directories are not available on the compute nodes. Use the full path (eg: /lus/theta-fs0/projects/<projectname>) to access the project directory.
For non-GPU codes:
- gcc – for C compiler
- g++ – for C++
- gfortran – for Fortran
For CUDA codes:
For MPI, the latest MPI is in /lus/theta-fs0/software/thetagpu/openmpi-4.0.5.
- mpif77/mpif90 not configured yet
mpirun is a wrapper in /usr/local/bin that sets the appropriate options and uses the mpirun in the MPI directory above.
On the service nodes, GNU compilers are available.
Modules are set up on ThetaGPU. See this page for more information.