ALIs
kommt nochGROMACS - A Molecular Dynamics Package
Description of the LRZ specific usage of GROMACS on the Linux Cluster HPC Systems.
Introductory Remarks
What is GROMACS?
GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles.
It is primarily designed for biochemical molecules like proteins and lipids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
GROMACS is licensed and redistributed under the GPL.
Please consult the GROMACS web site for further information on this package.
Authors
GROMACS was first developed in Herman Berendsens group, department of Biophysical Chemistry of Groningen University. It is a team effort, with contributions from several current and former developers all over world.
GROMACS installations at LRZ
The following table gives an overview of GROMACS installations at LRZ. Default versions are indicated by a bold red font.
|
Platform |
Releases |
Precision |
Remarks |
|---|---|---|---|
|
x86_64 Linux Cluster |
3.3.3, 4.0.5 |
single |
MPI parallel command mdrun_mpi available; need to start with mpiexec. |
|
x86_64 bit Linux Cluster |
3.3.3d , 4.0.5d |
double |
MPI parallel command mdrun_mpi available; need to start with mpiexec. |
|
Itanium Linux Cluster |
3.3.3, 4.0.5 |
single |
MPI parallel commands available for Parastation MPI (mdrun_mpi) and SGI Altix (mdrun_altix). |
|
Itanium Linux Cluster |
3.3.3d , 4.0.5d |
double |
MPI parallel commands available for Parastation MPI (mdrun_mpi) and SGI Altix (mdrun_altix). |
Please consult the example batch scripts below for how to use the MPI parallel versions. The single precision builds typically show larger numerical instabilities than the double precision builds. Furthermore, the GROMACS executables always have the same name (no additional _d suffix for the double precision version) with exceptionof the MPI parallel mdrun_xxx binaries.
Usage
Before using GROMACS, you need to load an appropriate environment module:
module load gromacs
-
On all systems, the double precision version will be loaded by default. To use the single precision builds, please issue e.g.,
module load gromacs/3.3.3
(the version number without the attached "d" indicates single precision).
-
On Itanium systems, the 3.3.x releases use assembler loops, which for many cases give considerable performance improvements. You can deactivate the assembler loops by setting the environment variable NOASSEMBLYLOOPS, but this will give much lower performance in most cases. The 4.0.x versions are built without assembler loops for Itanium, using the Fortran kernels instead. The assembler loops do not work any more on recent IA64 processors.
-
Note that in the GROMACS path there are automatic shell completion files available (completion.$SHELL) which add all GROMACS file extensions if you source them into your shell.
Setting up batch jobs
Gromacs on the Linux Cluster systems:
For long production runs, an SGE batch job should be used to run the program. The example batch scripts provided in this section require the input files speptide.top, after_pr.gro and full.mdp to be placed in ~/mydir before the run.
Further notes:
-
To run in batch mode, submit the script using the qsub command. To run small test cases interactively, start the script directly.
-
The parallel batch script uses the variable $NSLOTS which is either set by SGE or - if you run the script interactively - set to 4 CPUs. $NSLOTS is repeatedly referred to in the following lines of the SGE script
-
For parallel runs, Parastation does not find the executable, hence its full pathname is specified via "which".
-
For the 4.0.x versions, the grompp command does not accept the -np argument any more. Please consult the documentation for how to set up parallel runs in 4.0.x
-
For batch jobs, the nice switch is set to 0 for mdrun. Please omit this switch when running interactively, otherwise your job will be forcibly removed from the system after some time.
-
Please do not forget to replace the dummy e-Mail address in the example scripts by your own one.
Gromacs on the HLRB-II:
A PBS job script must be generated and submitted. An example is given at the end of the table below.
|
Serial processing |
|
x86-64 Cluster |
|---|
#!/bin/bash #$-o $HOME/mydir/gromacs.out -j y #$-N gromacs #$-S /bin/bash #$-l march=x86_64 #$-M wrzlprmft@mydomain . /etc/profile cd mydir module load gromacs grompp -v -f full -o full -c after_pr -p speptide mdrun -v -nice 0 -s full -e full -o full -c after_full -g flog |
|
Itanium Cluster |
#!/bin/bash #$-o $HOME/mydir/gromacs.out -j y #$-N gromacs #$-S /bin/bash #$-l march=ia64 #$-M wrzlprmft@mydomain . /etc/profile cd mydir module load gromacs grompp -v -f full -o full -c after_pr -p speptide mdrun -v -nice 0 -s full -e full -o full -c after_full -g flog |
|
Parallel processing |
|
2 or 4-way Itanium Cluster |
#!/bin/bash
#$-o $HOME/mydir/gromacs.out -j y
#$-N gromacs
#$-S /bin/bash
#$-l h_rt=24:00:00
# Please choose "mpi" for 4-way, "mpi_pe" for 2-way systems
#$-pe [mpi|mpi_pe] 20
#$-M wrzlprmft@mydomain
. /etc/profile
cd mydir
module load gromacs
if [ -z $NSLOTS ] ; then
export NSLOTS=4
fi
grompp -v -f full -o full -c after_pr -p speptide -np $NSLOTS
mpirun -np $NSLOTS $(which mdrun_mpi) -v -s full -e full \
-o full -c after_full -g flog -N $NSLOTS
|
|
SGI Altix (Linux-Cluster) |
#!/bin/bash
#$-o $HOME/mydir/gromacs.out -j y
#$-N gromacs
#$-S /bin/bash
#$-l h_rt=24:00:00
#$-pe numa* 16
#$-M wrzlprmft@mydomain
. /etc/profile
cd mydir
module load gromacs
if [ -z $NSLOTS ] ; then
export NSLOTS=4
fi
grompp -v -f full -o full -c after_pr -p speptide -np $NSLOTS
mpirun -np $NSLOTS $(which mdrun_altix) -v -s full -e full \
-o full -c after_full -g flog -N $NSLOTS
|
|
x86_64 Cluster |
#!/bin/bash #$-o $HOME/mydir/gromacs.out -j y #$-N gromacs #$-S /bin/bash #$-l h_rt=24:00:00 # use multiples of 8 as last argument in the following line #$-pe mpi_8 8 #$-M wrzlprmft@mydomain . /etc/profile cd mydir module load gromacs if [ -z $NSLOTS ] ; then export NSLOTS=4 fi grompp -v -f full -o full -c after_pr -p speptide -np $NSLOTS mpiexec -n $NSLOTS $(which mdrun_mpi) -v -s full -e full \ -o full -c after_full -g flog -N $NSLOTS |
|
Parallel processing on HLRB-II |
#!/bin/ksh
#PBS -o $HOME/mydir/gromacs.out
#PBS -N gromacs
#PBS -l select=16
#PBS -l walltime=10:00:00
#PBS -M <your email address>
. /etc/profile.d/modules.sh
cd mydir
module load gromacs
grompp -v -f full -o full -c after_pr -p speptide -np 16
mpiexec $(which mdrun_altix) -v -s full -e full \
-o full -c after_full -g flog -N 16
|
Expected performance
According to the internal performance evaluation of mdrun, around 1 GFlop/s are achieved by the small test system referred to in the example job scripts on both the IA32 (2.8 GHz) and Itanium (1.6 GHz) systems if dedicated CPUs are available and the double precision version is used. For parallel runs, scaling depends on the system size; the test system achieves a speedup of 2.1-2.2 with 4 CPUs. Please run tests of your own to optimize the number of processors for your input data.
The Gromacs benchmark results for the LRZ HPC systems are also available.
Documentation
After loading the environment module, the $GROMACS_DOC variable points to a directory containing documentation and tutorials.
For gromacs 4.0, parallel scalability has been much improved. The invocation of grompp and the parallel mdrun_mpi, mdrun_altix binaries has slightly changed.
For further information (including the man pages for all GROMACS subcommands), please refer to the GROMACS web site, in particular the documentation links.