BG/Q Compiling and Linking FAQs


This FAQ applies to the Blue Gene/Q systems (Mira, Cetus, Vesta)

Where do I find application libraries?

Most application libraries reside in /soft/libraries. Additional information is listed at Software and Libraries.

Most system and IBM-provided libraries are located in /bgsys/drivers/ppcfloor. If you cannot locate a library you need, contact support@alcf.anl.gov.

How do I select XL compiler versions?

You can determine the default XL compiler version by using the option -qversion=verbose. It is important to check the version number ("Level") of all components. Note they are in the form YYMMDD. For example:

[username@miralac1 ~]> mpixlc -qversion=verbose
IBM XL C/C++ for Blue Gene, V12.1
Version: 12.01.0000.0008
Driver Version: 12.01(C/C++) Level: 140417 ID:_gBvIcmjEEeOTbt3so48j-A
C Front End Version: 12.01(C/C++) Level: 140417 ID:_kTr5Qn7SEeOTb93so48j-A
C++ Front End Version: 12.01(C/C++) Level: 140501 ID:_TB1SNdFsEeOC9P7uAVQYOQ
High-Level Optimizer Version: 12.01(C/C++) and 14.01(Fortran) Level: 140417 ID:_kTVT9H7SEeOTb93so48j-A
Low-Level Optimizer Version: 12.01(C/C++) and 14.01(Fortran) Level: 140417 ID:_o-Q4IMZGEeOCnf7uAVQYOQ

[username@miralac1 ~]> mpixlf90 -qversion=verbose
IBM XL Fortran for Blue Gene, V14.1
Version: 14.01.0000.0008
Driver Version: 14.01(Fortran) Level: 140417 ID:_gBvIcmjEEeOTbt3so48j-A
Fortran Front End and Run Time Version: 14.01(Fortran) Level: 140417 ID:_o3fhwsZGEeOCnf7uAVQYOQ
Fortran Transformer Version: 14.01(Fortran) Level: 140417 ID:_o4Glx8ZGEeOCnf7uAVQYOQ
High-Level Optimizer Version: 12.01(C/C++) and 14.01(Fortran) Level: 140417 ID:_kTVT9H7SEeOTb93so48j-A
Low-Level Optimizer Version: 12.01(C/C++) and 14.01(Fortran) Level: 140417 ID:_o-Q4IMZGEeOCnf7uAVQYOQ

To see other versions of the IBM XL compilers that are available, use the command "softenv" with no arguments and look for keys starting with "@ibm-compilers-".

For example:

> softenv | grep @ibm-compilers
...
  P @ibm-compilers-2013-05         IBM BG/Q C/C++ & Fortran Compilers May 2013 PTF
  P @ibm-compilers-2013-08         IBM BG/Q C/C++ & Fortran Compilers August 2013 PTF
  P @ibm-compilers-2013-11         IBM BG/Q C/C++ & Fortran Compilers November 2013 PTF
  P @ibm-compilers-2014-02         IBM BG/Q C/C++ & Fortran Compilers February 2014 PTF
* P @ibm-compilers-2014-05         IBM BG/Q C/C++ & Fortran Compilers May 2014 PTF
  P @ibm-compilers-2014-08         IBM BG/Q C/C++ & Fortran Compilers August 2014 PTF

You can choose a non-default version by modifying your ~/.soft file. Two lines are necessary: one to disable the default compilers, another to select the version you want. These must be in order preceding the @default.

For example:

@ibm-compilers-2014-08
@remove @ibm-compilers-default
#The @default must be the last entry in the file
@default

After modifying your ~/.soft, either use the command "resoft" or log out and back in again to refresh your environment. Confirm your selection by using the -qversion=verbose option as above. (The "resoft" command is a shell function or alias set up by /soft/environment/softenv-1.6.2/etc/softenv-aliases.sh or /soft/environment/softenv-1.6.2/etc/softenv-aliases.csh.) You will generally use the MPI wrappers for the compilers rather than invoke the compilers directly. You will need to choose which flavor of wrappers to use, such as +mpiwrapper-xl.legacy or +mpiwrapper-xl.legacy.ndebug (see "Blue Gene/Q Versus Blue Gene/P" and "Explanation of MPI Variants" for more details).

How do I Specify -D options to XLF?

The XL Fortran compilers require a slightly different syntax to define preprocessor macro symbols. Instead of just "-D", you should use "-WF,-D" like this:

# The correct way for XL Fortran
mpixlf90 -WF,-DMYFLAG myprogram.F90

If you forget and use only -D, you will see errors like these:

1501-214 (W) command option M reserved for future use - ignored
1501-216 (W) command option -YFLAG is not recognized - passed to ld

Multiple options may either be specified in a comma-separated list or by using -WF again:

mpixlf90 -WF,-DMYFLAG,-DFLAG2 myprogram.F90
mpixlf90 -WF,-DMYFLAG -WF,-DFLAG2 myprogram.F90

How do I compile and link a shared library?

  • IBM XL compilers:

Fortran: Create a shared library by using Fortran compiler from func1.f90 and func2.f90:

mpixlf90_r -qpic -c func1.f90 -o func1.o mpixlf90_r -qpic -c func2.f90 -o func2.o mpixlf90_r -qmkshrobj -o libfuncs.so func1.o func2.o

Linking with a shared library:

mpixlf90_r

mpixlc_r -qmkshrobj -o libfoo.so foo.o

# Linking with the shared library:

mpixlf90_r -Wl,-dy -o main main.f90 -L. -lfoo \ -L/bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/lib -lstdc++

Note: The Fortran compiler does not support the -qmkshrobj option, so in the above example the C compiler was used. Alternatively, the linker can be invoked directly:

# Create a shared library from foo.f90:

mpixlf90_r -qpic -c foo.f90 -o foo.o /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-ld -shared -o libfoo.so foo.o

# Linking with the shared library:

mpixlf90_r -Wl,-dy -o main main.f90 -L. -lfoo -L/bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/lib -lstdc++

Also note, the MPI and system libraries introduce the dependence on libstdc++.so

  • Gnu compilers:

/bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/gcc -fPIC -c foo.c -o foo.o /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/gcc -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/gcc -c main.c -o main.o /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/gcc -dynamic -o main main.o -R. -L. -lfoo \\ -L/bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/lib -ldl

How do I build large file support?

To build with large file support (64 bit offsets), use these macros:

 #define _LARGEFILE64_SOURCE
 #define _FILE_OFFSET_BITS 64

The former exposes some 64-bit system calls (lseek64, etc) while the latter makes off_t 8 bytes.

Recent versions of autoconf have an 'AC_SYS_LARGEFILE' macro that does this for you.

I cannot exec cpp or xlfentry. What do I do?

If, when compiling a Fortran program, you cannot exec cpp or xlfentry, it may be because the -B option was specified unintentionally.

For example, if you erroneously compile like this:

 > mpixlf90 -DBG main.f90
/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/bin/.orig/bgxlf90_r: 1501-221 (S) cannot exec program Gxlfentry - No such file or directory
1501-511  Compilation failed for file main.f90. 

The -DBG is interpreted as two options. -D controls processing fixed-form source, and is mostly harmless. But -B is the option to override the path prefix for the XL component executables, and -BG means to use "G" as the path prefix. The same problem may manifest with a cpp error if the source is .F or .F90, which invokes cpp implicitly:

 > mpixlf90 -DBG foo.F90
/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/bin/.orig/bgxlf90_r: 1501-221 (S) cannot exec program Gcpp - No such file or directory
1501-511  Compilation failed for file main.F90.

In the above example, the proper way to define the macro symbol "BG" is with the option "-WF,-DBG".

What is the default version of gcc cross-compiler?

The default version of the gcc cross-compiler on the system is currently 4.4.6.

How do I run time check for array accesses?

Run time checking of array accesses can be performed to verify that the index is in the defined range for the array. Array bound checking is enabled by setting the following compiler options with the XL compilers:

xlf (mpixlf90, ...) - "-C" or "-qcheck"

mpixlf90 -o test -C -g test.f90
•	xlc (mpixlc, ...) - "-qcheck=bounds"
mpixlc -o test -qcheck=bounds -g test.c

In addition to the array bound checking, enable the debugging information for both compilers with the "-g" option.

Programs compiled with array bound checking will check array indexes at run time to verify that they are within the array bounds. When an out of bound index is encountered, a SIGTRAP signal will be generated that terminates the program and causes core files to be written. The resulting core files may be examined to determine where in the program the out of bound reference occurred. To analyze a core file, use the bgq_stack command:

bgq_stack 

(The bgq_stack command should be in your default path; if not, find it at /soft/debuggers/scripts/bin/bgq_stack.) Below is an example of an out of bound reference with the bad reference identified. The following program attempts to reference array indexes beyond the defined 100 entries in the array.

      1 | #include <stdio.h>
        2 | main( int argc, char**argv ){
        3 |   double a[100];
        4 |   printf("Start of Loop\n");
        5 |   for (int i = 0; i < 110; i++ ){
        6 |     a[i] = i;
        7 |   }
        8 |
        9 |   printf("End of Loop\n");
       10 |   return 0;
       11 | }
 

After compiling the code with array bound checking enabled: mpixlc -o boundschk -qcheck=bounds -g boundschk.c Executing the code produces the following output that shows the program exits due to a SIGTRAP signal (signal 5).

...
2013-01-16 21:23:05.621 (INFO ) [0x400013a8ad0] MIR-084C0-3B7F1-512:46539:ibm.runjob.client.options.Parser: set local socket to runjob_mux from properties file
2013-01-16 21:23:07.822 (INFO ) [0x400013a8ad0] MIR-084C0-3B7F1-512:157092:ibm.runjob.client.Job: job 157092 started
2013-01-16 21:23:09.481 (WARN ) [0x400013a8ad0] MIR-084C0-3B7F1-512:157092:ibm.runjob.client.Job: terminated by signal 5
2013-01-16 21:23:09.481 (WARN ) [0x400013a8ad0] MIR-084C0-3B7F1-512:157092:ibm.runjob.client.Job: abnormal termination by signal 5 from rank 0
...

A core file is produced that may be checked to verify where the bad array reference occurred by running

bgq_stack boundschk core.0
------------------------------------------------------------------------
+++ID Rank: 0, TGID: 1, Core: 0, HWTID:0 TID: 1 State: RUN 

0000000001000404
main
/gpfs/mira-home/zippy/smalltests/bounds/boundschk.c:6

00000000010006b8
generic_start_main
/bgsys/drivers/V1R1M2/ppc64/toolchain/gnu/glibc-2.12.2/csu/../csu/libc-start.c:226

00000000010009b4
__libc_start_main
/bgsys/drivers/V1R1M2/ppc64/toolchain/gnu/glibc-2.12.2/csu/../sysdeps/unix/sysv/linux/powerpc/libc-start.c:194

0000000000000000
??
??:0

which shows that the bad reference occurred at line 6 of the file boundschk.c in the subroutine main(), which corresponds to the array index inside the for loop.

What is a fast version of an MPI library?

A "fast" version of the mpi libraries disables error checking and asserts in the library. It is not recommended for development runs, but could improve performance for MPI-correct production code.

Select this library by choosing +mpiwrapper-xl-legacy.ndebug or +mpiwrapper-xl.ndebug your .soft file:

  +mpiwrapper-xl.ndebug

(See "Blue Gene/Q Versus Blue Gene/P" and "Explanation of MPI Variants" for more about the various +mpiwrapper-* choices.)

What do I do if compilation is slow?

The XL compilers will output additional diagnostic information when given the options -v and -qphsinfo. If -qphsinfo shows it spending a lot of time in IPA, a workaround is to compile the file(s) affected using -qnoipa (at the expense of optimization).

How do I link with SCALAPACK and BLACS?

BLACS is part of SCALAPACK, so linking with it separately is not necessary. Link with SCALAPACK this way, for the versions built with the XL compilers:

 mpixlf90_r -o myprogram myprogram.F90 \
            -L/soft/libraries/alcf/current/xl/SCALAPACK/lib –lscalapack \
            -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack \
            -L/soft/libraries/alcf/current/xl/BLAS/lib –lblas

Similar libraries are built with the GNU compilers, under /soft/libraries/alcf/current/gcc/.

My main code is C++. It calls Fortran and ESSL and uses OMP. How do I link?

Try this link line:

 ESSLPATH=/soft/libraries/essl/current
 mpixlcxx_r -o myprogram myprogram.C  \
   -I$(ESSLPATH)/include -L$(ESSLPATH)/lib64 -lesslbg \
   -L$(IBM_MAIN_DIR)/xlf/bg/14.1/lib64 –lxlopt –lxlf90 –lxlfmath -lxl

For multithreaded code, use “-lxlf90_r” in place of “-lxlf90”.

My main code is C/C++ but calls for Fortran-compiled libraries (such as Lapack) or functions. It also uses OpenMP. How do I link?

Usually, you see the following unresolved symbols: `_xldipow', `_xlfBeginIO', `_xlfWriteFmt', `_xlfWriteFmt', `_xlfEndIO', `_xlfStop', etc.

Try adding the Fortran libraries explicitly

-L/soft/compilers/ibmcmp-nov2012/xlf/bg/14.1/lib64 -lxlopt -lxl -lxlf90 -lxlfmath

as well as the libraries for OpenMP

-L /soft/compilers/ibmcmp-nov2012/xlsmp/bg/3.1/lib64 -lxlomp_ser

What are underscore mismatch problems during linking?

The default on BG/Q is to compile without appending any underscores to symbol names. (The -qnoextname option is default.) Because all the system libraries contain symbols without appended underscores, it is not recommended to override this setting. However, if you try to link with other code that was built with -qextname, which appends underscores, there will be a mismatch. The two options for dealing with the problem are to rebuild that library with -qnoextname or, if that is difficult, to explicitly override the underscore behavior just for particular symbols using -qextname=func_name.

For example:

Suppose the mylib.a is built using -qextname and contains a function mylib_initialize.

Compile using -qextname=mylib_initialize.

The -qextname option must be listed for every symbol used from the offending library; so unless the set of symbols is small, this method will not be practical.

My program does not compile or link. There are undefined symbols or definitions that seem to be from system libraries. What do I do?

Make sure you are not compiling or linking login-node include files and/or libraries from, for example, /usr/include or /usr/lib. These are for login node executables only, and are not intended for compute node cross compilation. If you are using the wrapper scripts (such as mpixlc), there is no need to reference compute-node libraries directly. However, for unusual situations you should know that they will be in /bgsys/drivers/ppcfloor. Non-system libraries for compute-node compilation are in /soft/libraries.

What is an R_PPC_REL24 error during linking?

Errors relating to R_PPC_REL24 during linking are due to the generated code being too large for 24-bit relative jumps to reach from some branch sites. When using the XL compilers, IBM's advice is to try one or more of these compiler options to mitigate the problem:

   -qmaxmem=64000 [or some other large value]
   -Wl,--relax
   -qcompact

If these fail to solve the problem, contact support for further assistance.

What are Blue Gene preprocessor macros?

The following macros are predefined on the Blue Gene/Q with the XL C and C++ compilers:

Macro

Description

Value

Info

__bg__

Indicates that this is a Blue Gene® platform

1

Always predefined for all Blue Gene platforms

__bgq__

Indicates that the architecture is PowerPC A2

1

Predefined when the architecture is PowerPC A2.

The following macros are predefined on the Blue Gene/Q with the GNU C and C++ compilers:

Macro

Description

Value

Info

__bg__

Indicates that this is a Blue Gene® platform

1

Always predefined for all Blue Gene platforms

__bgq__

Indicates that the architecture is PowerPC A2

1

Predefined when the architecture is PowerPC A2.

How do I use configure?

If you explicitly specify BlueGene/Q when invoking configure, you will automatically get many of the correct settings.

./configure --host=powerpc64-bgq-linux

Always check the results of configure carefully as not all automatically configured settings will be correct.

What do I do if I have problems with nm?

Note: The default "nm" in your PATH, /usr/bin/nm, works correctly for most cross-compiled objects but is not guaranteed. Depending on the object file, it may fail with an error such as:

 nm: myfile.o: File format not recognized

The correct "nm" to use with cross-compiled objects is:

 /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-nm

and should work for all cross-compiled objects.