A thorough discussion of compiling your code for the BG/Q system is too complex to cover in this document. What follows is a brief summary. All compile and link procedures are performed on login nodes with a cross-compilation technique. Because it is cross-compiled, the resulting executable is for the BG/Q only and will not run on a login computer.
The preferred way to compile and link an application is to use the predefined MPI wrapper scripts.
Selecting MPI Wrapper Scripts
The default user environment does not provide a set of MPI compiler wrappers. A wrapper may be selected by using the appropriate SoftEnv key from the following list:
+mpiwrapper-gcc gcc wrappers and toolchain +mpiwrapper-gcc.legacy gcc.legacy wrappers and toolchain +mpiwrapper-xl xl wrappers and toolchain +mpiwrapper-xl.legacy xl.legacy wrappers and toolchain +mpiwrapper-xl.legacy.ndebug xl.legacy.ndebug wrappers and toolchain +mpiwrapper-xl.ndebug xl.ndebug wrappers and toolchain
Note: These are the IBM-supported compilers. For LLVM/Clang, please see see here.
Add the key to your ~/.soft file before the @default line. For example,
# ~/.soft +mpiwrapper-xl @default #end of ~/.soft
Note: After editing your ~/.soft, run the command "resoft" to refresh your environment.
There two basic choice with wrappers, standard and legacy.
- The standard versions uses fine grained locking within MPI.
- The legacy versions uses coarse grained locking within MPI.
Which version is better for performance will depend on the particular application. The other main factor is choosing between GNU gcc and IBM XL compilers.
A version of the libraries that was compiled with the GNU Compiler Collection (GCC) and uses fine-grained locking in MPICH. These libraries also have error checking and assertions enabled.
A version of the libraries that was compiled with the GNU Compiler Collection and uses a coarse-grain lock in MPICH. These libraries also have error checking and assertions enabled and can provide slightly better latency in single-thread codes, such as those that do not call MPI_Init_thread( ... MPI_THREAD_MULTIPLE ...). Use one of the gcc libraries for initial application porting work.
A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the fine-grained MPICH locking and all error checking and asserts enabled. These libraries can provide a performance improvement over the gcc libraries.
A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the coarse-grained MPICH lock and all error checking and assertions are enabled. These libraries can provide a performance improvement over the gcc.legacy libraries for single-threaded applications.
A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the fine-grained MPICH locking. Error checking and assertions are disabled. This setting can provide a substantial performance improvement when an application functions satisfactorily. Do not use this library version for initial porting and application development.
A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the coarse-grained MPICH lock. Error checking and asserts are disabled. This can provide a substantial performance improvement when an application functions satisfactorily. Do not use this library version for porting and application development. This library version can provide a performance improvement over the xl.ndebug library version for single-threaded applications.
Wrapper Command Names
Each wrapper set contains the following set of wrappers:
mpixlc, mpixlcxx, mpixlf77, mpixlf90, mpixlf95, mpixlf2003 (XL non thread-safe versions)
mpixlc_r, mpixlcxx_r, mpixlf77_r, mpixlf90_r, mpixlf95_r, mpixlf2003_r (XL thread-safe versions)
mpicc, mpicxx or mpixc++, mpif77, mpif90 (GNU compilers if 'gcc' wrappers set is selected otherwise these call XL compilers if an XL wrapper is selected)
Examples to compile and link a hello.c C-source program, hello.cc C++ source program, and hello.f Fortran source program into a hello executable with default compiler options are provided below.
mpixlc -O -o hello hello.c mpixlcxx -O -o hello hello.cc mpixlf77 -O -o hello hello.f
The scripts will invoke the compiler, set up required environment variables, and include directories and necessary MPI libraries. The scripts with the *_r suffix do the same except they invoke thread-safe libraries and generate thread-safe object files by adding additional compiler options. The “explicity” command invoked by the MPI wrapper scripts can be shown by passing it the ‘-show’ parameter as shown here.
mpixlc -show /soft/compilers/ibmcmp-nov2012/vac/bg/12.1/bin/bgxlc -I/bgsys/drivers/V1R2M0/ppc64 -I/bgsys/drivers/V1R2M0/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M0/ppc64/spi/include -I/bgsys/drivers/V1R2M0/ppc64/spi/include/kernel/cnk -L/bgsys/drivers/V1R2M0/ppc64/spi/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M0/ppc64/spi/lib -L/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M0/ppc64/spi/lib -I/bgsys/drivers/V1R2M0/ppc64/comm/xl/include -L/bgsys/drivers/V1R2M0/ppc64/comm/xl/lib -lmpich -lopa -lmpl -lpami -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lpthread
For a user who wants to keep full control of the compilation/linking process, the non-wrapped IBM XL compilers may be accessed via:
- add _r suffix for thread-safe versions
Users should be aware that the names of BG/Q compilers always start with the ‘bg’ prefix: bgxlc, bgxlC, bgxlf. The compilers without the bg prefix are intended for a login node and should be used only for syntax verification. Note: If you use the non-wrapped compiler invocation, you will need to explicitly list all required system libraries.
To run a compiled application, submit the job to the resource manager and scheduler.
For additional information on selecting a particular compiler version, see How do I select XL Compiler versions.