Machine Overview for BG/Q

Blue Gene/Q systems are composed of login nodes, I/O nodes, and compute nodes.

Login Nodes

Login and compile nodes are IBM Power 7-based systems running Red Hat Linux and are the user’s interface to a Blue Gene/Q system. This is where users login, edit files, compile, and submit jobs. These are shared resources with multiple users.

I/O Nodes

The I/O node and compute environments are based around a very simple 1.6 GHz 16 core PowerPC A2 system with 16 GB of RAM. I/O node environments are Linux-based, but do not allow for user access. Their resources may be shared between jobs unless I/O is isolated. Compute node environments run IBM’s CNK (compute node kernel) operating system and lack any interface for direct access by end users.

Compute Nodes

A compute rack is composed of two midplanes with electrical power, cooling, and clock signal provisioned at the rack level. Each midplane within the rack is composed of 16 node boards and a service card. The service card is analogous to a desktop system’s BIOS. It helps manage the booting of the compute nodes and manages activity within a midplane. Each node board hosts 32 compute nodes and the link chips and optics that compose the Blue Gene/Q’s 5D torus interconnect. It is also at the node board level that boards connect to I/O drawers.

ALCF’s I/O configuration aims for maximum isolation. Every I/O drawer has 8 nodes that are identical to a compute node, except for the heatsink being air-cooled rather than water-cooled. Each I/O node drives a PCI Express card that interfaces with a given system’s available file systems. On Cetus and Mira, every rack connects to one drawer with four I/O nodes driving a given midplane, allowing for I/O isolation at the midplane level. The I/O ratio is 128 compute nodes to 1 I/O node. Vesta has four times as many I/O nodes and connections have been arranged to allow for isolation within a 128 node block and provide a 32 compute node to 1 I/O node ratio.

System

Linpack TFlop/s RMax (RPeak)

Compute Racks

I/O Drawers

Total Cores

Minimum block size**

Smallest block with I/O Isolation

I/O Ratio

Login Nodes

Mira

8162.4 (10066.3)

48

48

786,432

512 nodes

512 nodes

1:128

4 Power7

Cetus

*682 (838.88)

4

4

65,536

128 nodes

512 nodes

1:128

1 Power7

Vesta

*341 (419.44)

2

8

32,768

32 nodes

128 nodes

1:32

1 Power7

* extrapolated

** smaller sub-blocks are possible