Computational Systems


Contents

Theta

The Theta supercomputer at Argonne Leadership Computing Facility (ALCF) is a Cray XC40 system with a second-generation Intel Xeon Phi (code-named Knights Landing) processor and the Cray Aries proprietary interconnect. The system is water-cooled and comprises 20 cabinets, 3624 compute nodes, and 960 switches with a peak of 9.65PF. Each compute node is a single Xeon Phi chip with 64 cores, 16 GB Multi-Channel DRAM (MCDRAM), and 192GB DDR4 memory. Four compute nodes and an Aries chip form a compute blade. Sixteen compute blades form a chassis and three of the chassis are in a rack. Each of the 64 cores on the KNL chip supports four hardware threads for a total 256 threads per node. The core is out of order and supports the new AVX-512 vector instructions. 

The interconnect topology is a dual place Dragonfly with ten groups. Each group consists of two cabinets or racks. The total bi-section bandwidth is 7.2 TB/sec.

Theta users will have access to a Lustre file system with 10 PB of capacity and 210 GB/second bandwidth. Users will also have access to the HPSS data archive and Cooley, the ALCF’s analysis and visualization cluster. All of these resources are availalble through high performance networks including ESnet’s 100-gigabit-per-second links.

Theta System Configuration

  • Architecture: Cray XC40
  • Processor: 1.3 GHz Intel Xeon Phi 7230 SKU with 64 cores
  • Cabinets: 20
  • Nodes: 3,624
  • Cores/node: 64
  • Total cores: 231,936
  • Memory/node: 192 GB
  • High bandwidth memory/nodes: 16 GB

Mira

Mira, an IBM Blue Gene/Q supercomputer at the Argonne Leadership Computing Facility, is equipped with 786,432 cores, 768 terabytes of memory and has a peak performance of 10 petaflops. Mira’s 49,152 compute nodes have a PowerPC A2 1600 MHz processor containing 16 cores, each with 4 hardware threads, running at 1.6 GHz, and 16 gigabytes of DDR3 memory. A 17th core is available for the communication library. 

IBM’s 5D torus interconnect configuration, with 2GB/s chip-to-chip links, connects the nodes, enabling highly efficient computation by reducing the average number of hops and latency between compute nodes. The Blue Gene/Q system also features a quad floating point unit (FPU) that can be used to execute scalar floating point instructions, four-wide SIMD instructions, or two-wide complex arithmetic SIMD instructions. This quad FPU provides higher single thread performance for some applications.

Mira users will have access to a GPFS file system with 24 PB of capacity and 240 GB/second bandwidth. Users will also have access to the HPSS data archive and Cooley, the ALCF's analysis and visualization cluster.  All of these resources are available through high performance networks including ESnet’s recently upgraded 100 gigabit per second links.

Mira System Configuration

  • Architecture: IBM BG/Q
  • Processor: 16 1600 MHz PowerPC A2 cores
  • Cabinets: 48
  • Nodes: 49,152
  • Cores/node: 16
  • Total cores: 786,432 cores
  • Memory/node: 16 GB RAM per node
  • Memory/core: 1 GB
  • Interconnect: 5D Torus Proprietary Network
  • Speed: 10 PF
  • Square Footage: 1,632 sq feet

Cetus

Cetus shares the same software environment and file systems as Mira. The primary role of Cetus is to run small jobs in order to debug problems that occurred on Mira.

Cetus System Configuration

  • Architecture: IBM BG/Q
  • Processor: 16 1600 MHz PowerPC A2 cores
  • Cabinets: 4
  • Nodes: 4,096
  • Cores/node: 16
  • Total cores: 65,536 cores
  • Memory/node: 16 GB RAM per node
  • Memory/core: 1 GB
  • Interconnect: 5D Torus Proprietary Network

Vesta

Vesta is the ALCF’s test and development platform, serving as a launching pad for researchers planning to use Mira. Vesta has the same architecture as Mira, but on a much smaller scale (two computer racks compared to Mira’s 48 racks). This system enables researchers to debug and scale up codes for the Blue Gene/Q architecture in preparation for Mira.

Vesta is also a resource for users applying for a Director’s Discretionary allocation and researchers preparing proposals for INCITE and ALCC.  By allocating time on Vesta for testing and development, Mira is kept clear for capability jobs.

Vesta System Configuration

  • Architecture: IBM BG/Q
  • Processor: 16 1600 MHz PowerPC A2 cores
  • Cabinets: 2
  • Nodes: 2,048
  • Cores/node: 16
  • Total cores: 32,768 cores
  • Memory/node: 16 GB RAM per node
  • Memory/core: 1 GB
  • Interconnect: 5D Torus Proprietary Network