LAMMPS on Theta

Help Desk

Theta

Overview

LAMMPS is a general-purpose molecular dynamics software package for massively parallel computers. It is written in an exceptionally clean style that makes it one of the most popular codes for users to extend and it currently has dozens of user-developed extensions.

For details about the code and its usage, see the LAMMPS home page. This page is dedicated to information pertaining to Theta at the ALCF.

Using LAMMPS at ALCF

ALCF provides assistance with build instructions, compiling executables, submitting jobs, and providing prebuilt binaries. For questions, contact us at support@alcf.anl.gov.

How to Obtain the Code

LAMMPS is an open-source code, which can be downloaded at http://lammps.sandia.gov/download.html.

Building on Theta

After [LAMMPS] has been downloaded and unpacked, you should see a directory whose name is of the form lammps-<version>. One should see a lammps-<version>/src/MAKE/MACHINES/Makefile.theta Makefile in recent versions that can be used for compilation on Theta. An example Makefile is provided below with suggested compiler settings, but it is encouraged to edit a Makefile for the specific version of LAMMPS being used. For older versions of LAMMPS, you will need to take an existing Makefile (e.g. Makefile.mpi) and edit the top portion appropriately to create a Makefile.theta for the specific version of LAMMPS used.

# theta = Flags for Knights Landing Xeon Phi Processor, Intel compiler, Cray MPI, MKL FFT
# module unload libsci
# make theta -j 8

SHELL = /bin/sh

# ---------------------------------------------------------------------
# compiler/linker settings
# specify flags and libraries needed for your compiler

KOKKOS_DEVICES = OpenMP
KOKKOS_ARCH = KNL

CC       = CC -mkl
OPTFLAGS = -xMIC-AVX512 -O3 -fp-model fast=2 -no-prec-div -qoverride-limits
CCFLAGS  = -g -qopenmp -qno-offload -ansi-alias -restrict 
CCFLAGS += -DLMP_INTEL_USELRT -DLMP_USE_MKL_RNG $(OPTFLAGS)
CCFLAGS += -std=c++11
CCFLAGS += -DLAMMPS_MEMALIGN=64
SHFLAGS  = -fPIC
DEPFLAGS = -M

LINK      = CC -mkl
LINKFLAGS = -g -qopenmp $(OPTFLAGS) -dynamic
#LIB       = -ltbbmalloc
LIB       = -L$(TBBROOT)/lib/intel64/gcc4.8 -ltbbmalloc -Wl,-rpath=$(TBBROOT)/lib/intel64/gcc4.8
SIZE      = size

ARCHIVE    = ar
ARFLAGS    = -rc
SHLIBFLAGS = -shared

# ---------------------------------------------------------------------
# LAMMPS-specific settings, all OPTIONAL
# specify settings for LAMMPS features you will use
# if you change any -D setting, do full re-compile after "make clean"

# LAMMPS ifdef settings
# see possible settings in Section 2.2 (step 4) of manual

LMP_INC =

# MPI library
# see discussion in Section 2.2 (step 5) of manual
# MPI wrapper compiler/linker can provide this info
# can point to dummy MPI library in src/STUBS as in Makefile.serial
# use -D MPICH and OMPI settings in INC to avoid C++ lib conflicts
# INC = path for mpi.h, MPI compiler settings
# PATH = path for MPI library
# LIB = name of MPI library

MPI_INC  = -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1
MPI_PATH =
MPI_LIB  =

# FFT library
# see discussion in Section 2.2 (step 6) of manaul
# can be left blank to use provided KISS FFT library
# INC = -DFFT setting, e.g. -DFFT_FFTW, FFT compiler settings
# PATH = path for FFT library
# LIB = name of FFT library

FFT_INC  = -DFFT_MKL -DFFT_SINGLE
FFT_PATH =
FFT_LIB  = -L$(MKLROOT)/lib/intel64 -Wl,--start-group -lmkl_intel_lp64 \
           -lmkl_core -lmkl_intel_thread -Wl,--end-group

# JPEG and/or PNG library
# see discussion in Section 2.2 (step 7) of manual
# only needed if -DLAMMPS_JPEG or -DLAMMPS_PNG listed with LMP_INC
# INC = path(s) for jpeglib.h and/or png.h
# PATH = path(s) for JPEG library and/or PNG library
# LIB = name(s) of JPEG library and/or PNG library

JPG_INC  =
JPG_PATH =
JPG_LIB  =

# ---------------------------------------------------------------------
# build rules and dependencies
# do not edit this section

include Makefile.package.settings
include Makefile.package

EXTRA_INC = $(LMP_INC) $(PKG_INC) $(MPI_INC) $(FFT_INC) $(JPG_INC) $(PKG_SYSINC)
EXTRA_PATH = $(PKG_PATH) $(MPI_PATH) $(FFT_PATH) $(JPG_PATH) $(PKG_SYSPATH)
EXTRA_LIB = $(PKG_LIB) $(MPI_LIB) $(FFT_LIB) $(JPG_LIB) $(PKG_SYSLIB)
EXTRA_CPP_DEPENDS = $(PKG_CPP_DEPENDS)
EXTRA_LINK_DEPENDS = $(PKG_LINK_DEPENDS)

# Path to src files

vpath %.cpp ..
vpath %.h ..

# Link target

$(EXE): main.o $(LMPLIB) $(EXTRA_LINK_DEPENDS)
        $(LINK) $(LINKFLAGS) main.o $(EXTRA_PATH) $(LMPLINK) $(EXTRA_LIB) $(LIB) -o $@
        $(SIZE) $@

# Library targets

$(ARLIB): $(OBJ) $(EXTRA_LINK_DEPENDS)
          @rm -f ../$(ARLIB)
          $(ARCHIVE) $(ARFLAGS) ../$(ARLIB) $(OBJ)
          @rm -f $(ARLIB)
          @ln -s ../$(ARLIB) $(ARLIB)

$(SHLIB): $(OBJ) $(EXTRA_LINK_DEPENDS)
        $(CC) $(CCFLAGS) $(SHFLAGS) $(SHLIBFLAGS) $(EXTRA_PATH) -o ../$(SHLIB) \
        $(OBJ) $(EXTRA_LIB) $(LIB)
        @rm -f $(SHLIB)
        @ln -s ../$(SHLIB) $(SHLIB)

# Compilation rules

%.o:%.cpp
        $(CC) $(CCFLAGS) $(SHFLAGS) $(EXTRA_INC) -c $<


# Individual dependencies

depend : fastdep.exe $(SRC)
        @./fastdep.exe $(EXTRA_INC) -- $^ > .depend || exit 1

fastdep.exe: ../DEPEND/fastdep.c
        icc -0 -o $@ $<

sinclude .depend

 

As newer versions of LAMMPS are distributed and changes made to the Makefile, the example Makefile above can be used to generate an updated Makefile using one of the Intel examples packaged with LAMMPS. With the Makefile in place, LAMMPS can be compiled from the lammps-<version>/src directory using the following command.

cd lammps-<version>/src
make theta -j 8

 

Running LAMMPS Jobs on Theta

Following is an example executable script “run_lammps.csh” to run LAMMPS on two nodes of Theta with 64 MPI ranks per node. The job can be submitted with command “qsub run_lammps.csh”, where <project_name> is replaced with an active project allocation.

#!/bin/csh
#COBALT -n 2 -t 10 -q debug-cache-quad -A <project_name> -O LAMMPS

aprun -n 128 -N 64 -d 1 --cc depth -e OMP_NUM_THREADS=1 -j 1 ./lmp_theta -in lmp.in

Performance Notes

When possible, users will want to build LAMMPS executables with the USER-OMP and USER-INTEL packages for best performance on Theta. Following is an example script “run_lammps_intel.csh” to run LAMMPS on two nodes of Theta with 64 MPI ranks per node and two OpenMP threads per rank with the USER-INTEL and USER-OMP packages. The job can be submitted with command “qsub run_lammps_intel.csh.”

#!/bin/csh
#COBALT -n 2 -t 10 -q debug-cache-quad -A <project_name> -O LAMMPS

aprun -n 128 -N 64 -d 2 --cc depth -e OMP_NUM_THREADS=2 -j 2 ./lmp_theta -in lmp.in -sf hybrid intel omp

Not all available forcefields in LAMMPS are supported in one or both of these packages. For the latest information, please check LAMMPS website and documentation.

Systems
Topics