Compiling and Linking on BG/Q

A thorough discussion of compiling your code for the BG/Q system is too complex to cover in this document. What follows is a brief summary. All compile and link procedures are performed on login nodes with a cross-compilation technique. Because it is cross-compiled, the resulting executable is for the BG/Q only and will not run on a login computer.

The preferred way to compile and link an application is to use the predefined MPI wrapper scripts.

Selecting MPI Wrapper Scripts

The default user environment does not provide a set of MPI compiler wrappers. A wrapper may be selected by using the appropriate SoftEnv key from the following list:

    +mpiwrapper-gcc                gcc wrappers and toolchain
    +mpiwrapper-gcc.legacy         gcc.legacy wrappers and toolchain
    +mpiwrapper-xl                 xl wrappers and toolchain
    +mpiwrapper-xl.legacy          xl.legacy wrappers and toolchain
    +mpiwrapper-xl.legacy.ndebug   xl.legacy.ndebug wrappers and toolchain
    +mpiwrapper-xl.ndebug          xl.ndebug wrappers and toolchain

Add the key to your ~/.soft file before the @default line.  For example,

    # ~/.soft
    +mpiwrapper-xl
    @default
    #end of ~/.soft

Note: After editting your ~/.soft, run the command "resoft" to refresh your environment.

There two basic choice with wrappers, standard and legacy.

  • The standard versions uses fine grained locking within MPI.
  • The legacy versions uses coarse grained locking within MPI.

Which version is better for performance will depend on the particular application. The other main factor is choosing between GNU gcc and IBM XL compilers.

gcc

A version of the libraries that was compiled with the GNU Compiler Collection (GCC) and uses fine-grained locking in MPICH. These libraries also have error checking and assertions enabled.

gcc.legacy

A version of the libraries that was compiled with the GNU Compiler Collection and uses a coarse-grain lock in MPICH. These libraries also have error checking and assertions enabled and can provide slightly better latency in single-thread codes, such as those that do not call MPI_Init_thread( ... MPI_THREAD_MULTIPLE ...). Use one of the gcc libraries for initial application porting work.

xl

A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the fine-grained MPICH locking and all error checking and asserts enabled. These libraries can provide a performance improvement over the gcc libraries.

xl.legacy

A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the coarse-grained MPICH lock and all error checking and assertions are enabled. These libraries can provide a performance improvement over the gcc.legacy libraries for single-threaded applications.

xl.ndebug

A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the fine-grained MPICH locking. Error checking and assertions are disabled. This setting can provide a substantial performance improvement when an application functions satisfactorily. Do not use this library version for initial porting and application development.

xl.legacy.ndebug

A version of the libraries with MPICH compiled with the XL compilers and PAMI compiled with the GNU compilers. This version has the coarse-grained MPICH lock. Error checking and asserts are disabled. This can provide a substantial performance improvement when an application functions satisfactorily. Do not use this library version for porting and application development. This library version can provide a performance improvement over the xl.ndebug library version for single-threaded applications.

Wrapper Command Names

Each wrapper set contains the following set of wrappers:

mpixlc, mpixlcxx, mpixlf77, mpixlf90, mpixlf95, mpixlf2003 (XL non thread-safe versions)

mpixlc_r, mpixlcxx_r, mpixlf77_r, mpixlf90_r, mpixlf95_r, mpixlf2003_r (XL thread-safe versions)

mpicc, mpicxx or mpixc++, mpif77, mpif90 (GNU compilers if 'gcc' wrappers set is selected otherwise these call XL compilers if an XL wrapper is selected)

Usage

Examples to compile and link a hello.c C-source program, hello.cc C++ source program, and hello.f Fortran source program into a hello executable with default compiler options are provided below.

mpixlc -O -o hello hello.c
mpixlcxx -O -o hello hello.cc
mpixlf77 -O -o hello hello.f

The scripts will invoke the compiler, set up required environment variables, and include directories and necessary MPI libraries. The scripts with the *_r suffix do the same except they invoke thread-safe libraries and generate thread-safe object files by adding additional compiler options. The “explicity” command invoked by the MPI wrapper scripts can be shown by passing it the ‘-show’ parameter as shown here.

mpixlc -show

/soft/compilers/ibmcmp-nov2012/vac/bg/12.1/bin/bgxlc -I/bgsys/drivers/V1R2M0/ppc64 
-I/bgsys/drivers/V1R2M0/ppc64/comm/sys/include -I/bgsys/drivers/V1R2M0/ppc64/spi/include 
-I/bgsys/drivers/V1R2M0/ppc64/spi/include/kernel/cnk -L/bgsys/drivers/V1R2M0/ppc64/spi/lib 
-L/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M0/ppc64/spi/lib 
-L/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib -L/bgsys/drivers/V1R2M0/ppc64/spi/lib 
-I/bgsys/drivers/V1R2M0/ppc64/comm/xl/include -L/bgsys/drivers/V1R2M0/ppc64/comm/xl/lib 
-lmpich -lopa -lmpl -lpami -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lpthread

For a user who wants to keep full control of the compilation/linking process, the non-wrapped IBM XL compilers may be accessed via:

  • bgxlc
  • bgxlC
  • bgxlc++
  • bgxlf
  • bgxlf90
  • bgxlf95
  • bgxlf2003
  • bgf77
  • bgf90
  • bgf95
  • bgf2003
  • bgf2008
  • add _r suffix for thread-safe versions

Users should be aware that the names of BG/Q compilers always start with the ‘bg’ prefix: bgxlc, bgxlC, bgxlf. The compilers without the bg prefix are intended for a login node and should be used only for syntax verification. Note: If you use the non-wrapped compiler invocation, you will need to explicitly list all required system libraries.

To run a compiled application, submit the job to the resource manager and scheduler.

For additional information on selecting a particular compiler version, see How do I select XL Compiler versions.