Using Cobalt on Cooley

Overview

Like our Blue Gene system, Cooley uses the Cobalt job scheduler. 

Use the "qstat" command to see what jobs are in the queue, and the "nodelist" command to see which of the compute nodes (cc001-cc126) are free.  The “showres” command will display any special reservations in place.

Use the "qsub" command to submit jobs; what you submit should be a script which will be executed on the rank0 node allocated to you by the scheduler. This script will have access to an environment variable named COBALT_NODEFILE, which is the name of a file suitable for use with mpirun's -f option.  One important thing to note is that the --proccount option to qsub has no effect on Cooley – the number of MPI processes run by your job is entirely dependent on the arguments you supply to mpirun in your script.

At a minimum, qsub must be supplied with the number of nodes desired (-n), the walltime of the job (-t), and the path to the job script.  If you are associated with more than one project, you will also need to supply the project name using the -A option (or, alternately, set the PROJECT environment variable to that project name).

Some important notes regarding the job script:

  • The job script runs only on one node of your job (designated as the head node); it is up to your script to distribute processes to the other nodes in a multi-node job.  For a normal MPI job, this is accomplished by including "-f $COBALT_NODEFILE" in the arguments to mpirun/mpiexec. The COBALT_NODEFILE environment variable expands to the location of the nodefile (on the head node), which is a file containing the list of nodes Cobalt has assigned to your job.

  • Your job script must have an interpreter at the top (for example, #!/bin/bash) in order for Cobalt to recognize it as a valid script.

  • The job script will run using your default environment as set up by .bash_profile, .bashrc, and .soft.cooley at the time the job runs.  It does not inherit the environment the qsub command was run in.  If you need environment changes specific to this job, they must be set explicitly within the job script.

For example, you might have a script named test.sh, which runs mpirun with 12 processes per node (one process per core):

#!/bin/sh
NODES=`cat $COBALT_NODEFILE | wc -l`
PROCS=$((NODES * 12))
mpirun -f $COBALT_NODEFILE -n $PROCS /home/lueningh/cpi/cpi-x86_64

Note that the specific mpirun used depends on your software environment; you'll need to use a softenv key to specify which MPICH version you want to use.  By default, we give all new users +mvapich2 to use the most recent version of MVAPICH2, but if desired, a different MPI may be selected from several available.  For more information on softenv, see the softenv-intro man page.

To request 5 nodes from cobalt with 10 minutes of walltime, charging to the MyProject project, you would use the command:

qsub -n 5 -t 10 -A MyProject ./test.sh

Cobalt will produce some files in the same directory where you ran qsub (unless of course you tell it to use a different working directory). By default, they are named <jobid>.error, <jobid>.output, and <jobid>.cobaltlog. The .error output is stderr from your script, .output is stdout from your script, and the .cobaltlog file is some cobalt specific information like what your qsub submission contained, and how cobalt tried to invoke your executable.

There is also an "interactive" mode. "qsub -I" will submit an interactive job to cobalt.  Use it like this:

qsub -I -n 1 -t 30

It will block until your job runs, print out the list of nodes allocated to you, and then ssh to your rank0 node. When you log out (or your requested walltime expires), your job will be removed from the queue and the nodes will be released.  Please note -- the old 'qsubi' method of running interactive jobs is deprecated and no longer available on Cooley -- please use 'qsub -I' instead.

More Information

For Cooley's scheduling policy, please visit the job scheduling page.