HPCToolkit is an open-source suite of tools for profile-based performance analysis of applications. Below is a brief description on how to use the HPCToolkit on the ALCF systems. For more detailed information on using HPCToolkit and its capabilities, see the HPCToolkit website.

Installation on the ALCF System

The HPCToolkit is installed in:


Using HPCToolkit

Environment Setup

The HPCToolkit paths may be added or removed from your environment through the use of SoftEnv keys. To add the HPCToolkit paths, add the key:


to your SoftEnv environment file ~/.soft.

Alternatively, if you prefer not to use the Softenv key, the HPCToolkit bin directory should be added to your path as:

   export PATH=$PATH:/soft/perftools/hpctoolkit/bin  (bash)
   setenv PATH $PATH:/soft/perftools/hpctoolkit/bin  (csh)

Compile the Code to Use HPCToolkit

It is not necessary to modify your source code to use the HPCToolkit. However, for statically linked executables; the executable must be built using the HPCToolkit hpclink command as the linker.

The build procedure when using HPCToolkit is:

  • Compile the individual source files as usual, adding the flag for debugging symbols if not already used. For example:
   mpixlc -g -O3 -c routine1.c

This example shows compilation using a medium level of optimization (-O3). Any level of optimization is supported by HPCToolkit. However, to avoid possible problems, programs should be compiled without inter-procedural optimization (IPA) by specifying the "-qnoipa" option.

  • Link using the hpclink command before the name of the compiler normally used to link the program. For example:
   hpclink mpixlc -o myprog routine1.o ...

Note: The link step may take longer than using the usual linker as additional steps are performed during the linking process.

Run the Code

To run the program, submit to the queues as usual (using the executable built with the hpclink command) but set the HPCT environment variables (HPCRUN_TRACE, HPCRUN_EVENT_LIST, HPCRUN_PROCESS_FRACTION) to specify the data to be collected. For example:

   qsub ... --env HPCRUN_EVENT_LIST="WALLCLOCK@10000" --env HPCRUN_PROCESS_FRACTION=0.1 myprog

Possible profiling events include:

  • Time - profile the program using a time interval. Time-based profiling is specified by using the event:
    • WALLCLOCK@<freq> - profile based on wallclock time with a sampling interval <value> in micro-seconds
  • Ha