Key Changes with Mira

announcements

As 2013 INCITE projects begin tapping Mira this week, ALCF users will have a new resource with vastly improved performance and capabilities to advance their research.

Though some of Mira’s architecture is different, the system is grown from the same DNA as Intrepid, its Blue Gene/P predecessor at the ALCF. For programming purposes, it’s very similar to the previous Blue Gene systems—only much faster, which should make for a smooth transition from Intrepid to Mira.

However, users will need to be aware of some operational changes to help them hit the ground running with Mira. From managing quotas to compiling and job submissions, we provide a rundown of the significant changes below.

Access

The login for Mira is mira.alcf.anl.gov. A CRYPTOCard is required for access to the following resources: Mira, Vesta, Tukey, and Cetus. In order to request an ALCF user account, an individual must first be added as a member to an active ALCF Project.

For more information, visit our User Guide on Accounts and our dashboard for Project and Account Management.

SoftEnv

While the SoftEnv system will continue to be used to control user environments, the ‘rc’ file name has changed from ~/.softenvrc  to ~/.soft. The initial file should only contain the ‘@default’ key.

Compiling

There is no longer a default choice for MPI wrappers. Users must choose one of six soft keys and add it to their .soft file. You may also want to check your Makefiles for additional changes.

For more information, visit http://www.alcf.anl.gov/resource-guides/compiling-and-linking

Job Submission

The job submission command, qsub, now has a modified flag ‘--mode’ that takes a new set of values. These values determine the number of MPI ranks per compute node (e.g., c1 is one rank per compute node, c2 is two ranks per compute node, and so on). See example below.

qsub -A <project> -n <nodes> -t <walltime> -q <queue> --mode <c1,c2,c4,c8,c16,c32,c64>

For more information, visit: http://www.alcf.anl.gov/resource-guides/running-jobs

Software

The file system for ALCF-provided software has been reorganized to make access more intuitive. The /soft path now has additional subdirectories, including /soft/libraries for libraries, /soft/perftools for performance tools, and /soft/debuggers for debuggers.

For more information, visit http://www.alcf.anl.gov/resource-guides/software-and-libraries

Quotas and Project Directory

The Mira home directory will be the same as Intreprid’s home directory, with quotas assigned per user. However, unlike Intrepid, Mira will have a project directory with quotas assigned per project. The location of the project directory has changed from /intrepid-fs0/users/${USER}/persistent/ for Intrepid to /projects/<project name> for Mira.

For more information, visit https://www.alcf.anl.gov/user-guides/disk-quota

Ensemble Jobs

The Blue Gene/Q platform now provides users with the ability to allocate and boot blocks with a Cobalt resource allocation within their script. Unlike the Blue Gene/P platform, booting resources with the Blue Gene/Q is separate from running jobs. For ensemble jobs, the --disable_preboot flag must be added to the qsub submission line.

Additionally, the get-bootable-blocks utility provides a list of available blocks. This command takes a parent block as an argument, and also accepts --size and --geometry flags as constraints on the blocks returned.

Documentation

With two systems (Blue Gene/P and Blue Gene/Q) now in the fold, we have reorganized the ALCF’s User Guides (formerly known as Resource Guides) to improve usability. Information that pertains to a specific system is now organized by topic under the appropriate machine, while information that applies to both systems is located outside of the machine-specific information.

We are continuing to develop new documentation for the Blue Gene/Q system. Please let us know if there is anything we can do to improve our User Guides by filling out a Feedback Form (https://www.alcf.anl.gov/documentation-feedback) or contacting us at support@alcf.anl.gov.
 
Architecture

The Blue Gene/Q is a 64-bit PowerPC A2 core that runs at 1.6 GHz, with four hardware threads per core. Sixteen processor cores are used for computing, with a 17th core available for the communication library.

The system also features a quad floating-point unit (FPU) that can be used to execute scalar floating-point instructions, four-wide SIMD instructions, or two-wide complex arithmetic SIMD instructions. This upgrade provides higher single-thread performance for some applications.

Mira’s innovative 5D torus interconnect greatly improves how the Blue Gene/Q processors are organized and networked, resulting in lower latency and more efficient communication between the processors.

For more information, visit http://www.alcf.anl.gov/resource-guides/system-overview