Job Scheduling Policy for XC40 Systems

General Policy

We ask that all users follow good etiquette and be excellent to one another.


As with all Argonne Leadership Computing Facility production systems, job priority in the queue is based on several criteria:

  • positive balance of your project
  • size (in nodes) of the job, larger jobs receive higher priority
  • the type of project (e.g. INCITE, ALCC, or discretionary)
  • job duration - shorter duration jobs will accumulate priority more quickly, so it is best to specify the job run time as accurately as possible

Reservations and Scheduling Policy

Some work will require use of Theta that requires deviation from regular policy. On such occasions, normal reservation policy applies. Please send the regular form no fewer than five (5) business days. in advance.


Debugging Queues

  • There are two 16-node debugging queues:
  • debug-cache-quad
  • debug-flat-quad
  • Hardware is dedicated to each queue

  • Nodes are not rebootable to another mode

  • Job wall-clock time is limited to 1:00:00 (1 hour).

  • The maximum running job count is one (1) job per user.

Production Queues

  • There is a single submission queue for the entire system: default

  • Priority is given to jobs using at least 20% of Theta (648 nodes)

  • There is a global limit of ten (10) jobs running per user

  • There is a global limit of twenty (20) jobs in queue per user

  • There is a minimum job time of thirty (00:30:00) minutes for the default queue

  • There is a minimum allocation of 8 nodes

  • While shorter jobs may accumulate priority faster, all requested wall-clock times (job durations) greater than or equal to 12 hours are treated equivalently.

  • Wall-clock limits are a step-wise function designed to encourage scaling:

    • node count >= 8 nodes : maximum 2:00:00 hours
    • node count >= 16 nodes : maximum 4:00:00 hours
    • node count >= 128 nodes : maximum 6:00:00 hours
    • node count >= 384 nodes : maximum 12:00:00 hours
    • node count >= 648 nodes : maximum 24:00:00 hours
  • There is no default mode nodes may be assumed to be booted into. Failure to specify a mode will result in the assumption of cache-quad.

Big Run Mondays

As part of our regular maintenance procedures on Mondays, we will promote to the highest priority any jobs in the queued state requesting 648 nodes or more. Promotion is subject to operational discretion.

We may also, at our discretion, take the opportunity to promote the priority of capability jobs if the system has been drained of jobs for any other reason.

Monday Maintenance

On Mondays where the ALCF is on a regular business schedule the system may be expected to undergo maintenance from 9:00 am until 5:00 pm US Central Time. The showres command may be used to view pending and active maintenance reservations.

KNL Mode Selection and Charging

For the first year of Theta production, beginning July 1, 2017, time spent booting or rebooting nodes to obtain requested modes will not be charged to projects though it will account against requested walltime. This policy may be revisted or revised after the first year.

Please allow up to thirty (30) minutes for rebooting of nodes when submitting jobs.

Failure to specify a mode will result in the selection of cache-quad, the equivalent of listing:

--attrs mcdram=cache:numa=quad

in your qsub or job script.