Singularity

A Note on Docker

ALCF does not support Docker on our resources as it has security problems that allow privilege escalation. Singularity was developed with these security issues in mind.

Singularity on ALCF Resources

Singularity is a container solution for application science workloads. For details on Singularity and its usage, see the Singularity user guide. This page is dedicated to information pertaining to Theta at the ALCF. Please send questions to support at alcf.anl.gov

Building a Singularity Container on Theta

Singularity containers can be built from docker images or Singularity images already uploaded to Docker Hub or Singularity Hub. In principle, if one already has an image uploaded to these sites, building an image could be achieved with this command:

singularity build new_local_image.simg docker://username/image_name:image_version

or

singularity build new_load_image.simg shub://username/image_name:image_version

Otherwise an image must be built using Singularity Hub via the Github interface (see Section 1) or on a machine where the user has sudo access (see Section 2).

1. Building a Singularity Container for Theta on Singularity Hub

This workflow allows you to build a container on the Singularity Hub, then download it directly on Theta for use. The approach will build MPICH into the container by hand which will allow for building other applications which use MPI, then override the container's MPICH with the libraries supplied by Cray on Theta.

The first you will need a Singularity build recipe as given here:

Bootstrap: docker
From: centos

%setup
   echo ${SINGULARITY_ROOTFS}
   mkdir ${SINGULARITY_ROOTFS}/myapp
   cp pi.c ${SINGULARITY_ROOTFS}/myapp/


%post
   yum update -y
   yum groupinstall -y "Development Tools"
   yum install -y gcc
   yum install -y gcc-c++
   yum install -y wget
   cd /myapp
   # install MPICH
   wget -q http://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz
   tar xf mpich-3.2.1.tar.gz
   cd mpich-3.2.1
   # disable the addition of the RPATH to compiled executables
   # this allows us to override the MPI libraries to use those
   # found via LD_LIBRARY_PATH
   ./configure --prefix=$PWD/install --disable-wrapper-rpath
   make -j 4 install
   # add to local environment to build pi.c
   export PATH=$PATH:$(pwd)/install/bin
   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/install/lib
   env | sort
   cd ..
   mpicc -o pi -fPIC pi.c

%runscript
   /myapp/pi

Link to details on Singularity recipe files.

This recipe uses the docker://centos container as a starting point.

The %setup section occurs prior to the container creation therefore in order to create the folder /myapp in the container I use the environment variable SINGULARITY_ROOTFS to reference what will become the root directory of the container.

The %post section occurs after the container has been created and can be treated as a shell script being executed inside the container. Here we install some basic packages using yum that enable further building. Then we download MPICH and build it. MPICH is configured with --disable-wrapper-rpath which disables a feature that would otherwise build-in the MPICH library location into any MPI-enabled binary compiled using mpicc or mpicxx, etc. This is the key to allow us to override the container's libmpi.so and replace with with Cray's when we run on Theta. Last, we add MPICH to the local environment and build our example application pi.c.

The %runscript section specifies which application to run when the container is called using singularity run myimage.img. In this case, we run the pi binary.

Once you have a recipe, create a github repo that contains the recipe file as Singularity. Our example can be seen here. Notice the input file pi.c is there as well. A submit script is included for reference.

After creating the github repo, go to https://www.singularity-hub.org/login/ and create an account using GitHub to authenticate. Then go to the 'My Container Collections' page. Then click 'Add a Collection'. Select your new Github repo and click 'Submit' at the bottom. You should now see that 'Container Collection' on the 'My Container Collections' page. Click on it. to reach a page specifically for that collection. You will a 'Status' for the container which will tell you the status of the container.

You may need to make one more commit+push to your github repo to trigger the first build in SingularityHub because the process is trigger for every commit to the repo. The Status will report 'Running' while the container is being built, 'Error' if the build failed in which case you can click the 'Error' button to get log information, or 'Complete' if the container completed. If the container completed you can login to Theta and run

singularity build shub://<username>/<container-collection-name> <myimagename.img>

You should then have an image on disk and can run using this batch script:

#!/bin/bash
#COBALT -t 30
#COBALT -q debug-cache-quad
#COBALT -n 2
#COBALT -A datascience

# app build with GNU not Intel
module swap PrgEnv-intel PrgEnv-gnu
# Use Cray's Application Binary Independent MPI build
module swap cray-mpich cray-mpich-abi

# include CRAY_LD_LIBRARY_PATH in to the system library path
export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
# also need this additional library
export LD_LIBRARY_PATH=/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__g388ccd5.ari/lib64/:$LD_LIBRARY_PATH
# in order to pass environment variables to a Singularity container create the variable
# with the SINGULARITYENV_ prefix
export SINGULARITYENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
# print to log file for debug
echo $SINGULARITYENV_LD_LIBRARY_PATH

# this simply runs the command 'ldd /myapp/pi' inside the container and should show that
# the app is running agains the host machines Cray libmpi.so not the one inside the container
# run my contianer like an application, which will run '/myapp/pi'
aprun -n 8 -N 4 singularity run -B /opt:/opt:ro -B /var/opt:/var/opt:ro myimage.simg

An image with MPICH installed already exists at shub://jtchilders/singularity_mpi_test_recipe:latest and therefore some of the steps outlined in the above Singularity recipe can be skipped if this image is used as the base image in your recipe. The following recipe file shows how to do this:

Bootstrap: shub
From: jtchilders/singularity_mpi_test_recipe:latest

%setup
   # make directory for test MPI program
   mkdir ${SINGULARITY_ROOTFS}/uselocalmpi
   cp pi.c ${SINGULARITY_ROOTFS}/uselocalmpi/

%post
   # add to local environment to build pi.c
   export PATH=$PATH:/mpich/install/bin
   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mpich/install/lib
   cd /uselocalmpi
   mpicc -o pi -fPIC pi.c

%runscript
   /uselocalmpi/pi

This recipe file can be seen here: https://github.com/jtchilders/singularity_mpi_test_recipe/blob/master/Si...

Note the name of the file is Singularity.derived. Singularity hub will build containers for each file named Singularity or Singularity.<some-string> and the <some-string> part becomes a 'versioning' string that will appear after the collection name with a colon:

shub://jtchilders/singularity_mpi_test_recipe:latest

2. Building a Singularity Container for Theta on personal machine

This workflow allows you to build a container on a system where you have 'sudo' capabilities and transfer it to Theta for use. The approach is to build MPICH into the container by hand, then once on Theta the Cray version of MPI is used to override the internal MPICH installation.

This example was run on a Mac installed with Singularity following these instructions. There are also instructions for Windows and Linux. If you use a MAC or Windows machine edit your VagrantFile to contain this line:

​config.vm.synced_folder ".", "/vagrant_data"

This will mount your local folder into the Vagrant Virtual Machine. You'll need to stop and restart the VM to pickup the change.

Next, to follow the example create a file named "pi.c" and copy and paste this content to the file:

#include "mpi.h"
#include <math.h>
#include <stdio.h>

int main(int argc, char** argv)
{
  int n, myid, numprocs, i;
  double PI25DT = 3.141592653589793238462643;
  double mypi, pi, h, sum, x;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &myid);

  printf("worker %d of %d\n",myid,numprocs);

  /* Number of intervals */
  n = 25;

  MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
  h = 1.0 / (double) n;
  sum = 0.0;
  for ( i = myid + 1; i <= n; i += numprocs )
  {
       x = h * ((double) i - 0.5);
       sum += ( 4.0 / (1.0 + x*x) );
  }
  mypi = h * sum;
  MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);

  if (myid == 0)
  {
      printf("pi is approximately %.16f, Error is %.16f\n",
               pi, fabs(pi - PI25DT));
  }

  MPI_Finalize();
  return 0;
}

Note that this is just a simple MPI application that calculates pi.

Next create a file with the name "build.sh" and copy/paste this content inside:

#!/bin/bash
wget http://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz
tar xf mpich-3.2.1.tar.gz
cd mpich-3.2.1
./configure --prefix=$PWD/install --disable-wrapper-rpath
make -j 4 install
export PATH=$PATH:$PWD/install/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/install/lib
cd ..
mpicc -o pi -fPIC pi.c

There are some important items in this build script. It downloads and builds MPICH. The configure command requires the 'disable-wrapper-rpath' setting. This ensures that when you use 'mpicc' inside the container to build your application (in this case 'pi.c'), that MPICH does not insert its path into the RPATH of the binary compiled. If you do not disable this feature, any binary built using the MPICH installation will be forced to use this MPICH library. As we intend to run our application on Theta, we must use the Cray MPI installation to get the correct cross-node communication. This also means that you must ensure that the application is dynamically linked against MPICH and not statically linking MPICH into the binary.

Next create a file named "SingularityFile" and copy and paste the following inside:

Bootstrap: docker
From: centos

%setup
   echo ${SINGULARITY_ROOTFS}
   mkdir ${SINGULARITY_ROOTFS}/myapp

%files
   /vagrant_data/pi.c /myapp/
   /vagrant_data/build.sh /myapp/

%post
   yum update -y
   yum groupinstall -y "Development Tools"
   yum install -y gcc
   yum install -y gcc-c++
   yum install -y wget
   cd /myapp
   ./build.sh

%runscript
   /myapp/pi
 
This recipe downloads the docker://centos image as a starting point, unpacks that docker image into the ${SINGULARITY_ROOTFS} on your local system, creates the directory 'myapp' at the ROOT of the new image. Copies the 'pi.c' and 'build.sh' files into the container at '/myapp'. (Notice here that if you are using Vagrant, we mounted our local MAC/Windows directory where our files are located to the '/vagrant_data' path inside the vagrant VM) The recipe inside the container uses the internal 'yum' instance to install some needed libraries for building, then runs the build script we created above to build our software. Lastly we tell the container what binary to execute when running with Singularity.

Now create a ​Singularity image using the command:

sudo singularity build my_image.simg SingularityFile

You should be able to test your application by running:

singularity exec mpitest.img bash -c "LD_LIBRARY_PATH=/myapp/mpich-3.2.1/install/lib:$LD_LIBRARY_PATH /myapp/pi"

As mentioned previously, you must tell the application where to find the correct MPI.

Once completed, this image can be copied to Theta.

Running Singularity on Theta

Singularity has been setup to automatically mount a number of directories, including proc, sys, and $HOME into the running image.

You can 'bind-mount' additional directories into the container with the -B flag, for instance the Cray/Intel programming environment exists inside the /opt directory:

singularity run -B /opt:/opt my_image.img

This will run the application specified by the 'runscript' setting in the SingularityFile recipe. 

You can also run other things with this command:

singularity exec -B /opt:/opt my_image.img ls /myapp

In order to run the example MPI application built in the previous example we need to create a Cobalt submit script. Copy and paste the following into a file named "submit.sh":

#!/bin/bash
#COBALT -t 30
#COBALT -q debug-cache-quad
#COBALT -n 2
#COBALT -A MY_PROJECT_NAME

# app build with GNU not Intel
module swap PrgEnv-intel PrgEnv-gnu
# Use Cray's Application Binary Independent MPI build
module swap cray-mpich cray-mpich-abi

module load darshan

# prints to log file the list of modules loaded (just a check)
module list

# include CRAY_LD_LIBRARY_PATH in to the system library path
export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
# also need this additional library
export LD_LIBRARY_PATH=/opt/cray/wlm_detect/1.2.1-6.0.4.0_22.1__gd26a3dc.ari/lib64/:$LD_LIBRARY_PATH
# in order to pass environment variables to a Singularity container create the variable
# with the SINGULARITYENV_ prefix
export SINGULARITYENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
export SINGULARITYENV_LD_PRELOAD=$DARSHAN_PRELOAD
# print to log file for debug
echo $SINGULARITYENV_LD_LIBRARY_PATH

# this simply runs the command 'ldd /myapp/pi' inside the container and should show that
# the app is running agains the host machines Cray libmpi.so not the one inside the container
#aprun -n 1 -N 1 singularity exec -B /opt:/opt:ro -B /var/opt:/var/opt:ro mpitest.img ldd /myapp/pi
# run my contianer like an application, which will run '/myapp/pi'
aprun -n 8 -N 4 singularity run -B /opt:/opt:ro -B /var/opt:/var/opt:ro mpitest.img

Then you can submit your job with 'qsub -n <Number_of_nodes> -q <queue_name> -t <time> -A <project_name> submit.sh'

Checking Singularity version

Theta usually runs the current stable version of Singularity. To get the version number type

$ singularity --version