C2115 Practical Introduction to
Supercomputing
8th Lesson
Petr Kulhánek, Jakub Štěpán
kulhanek@chemi.muni.cz
National Centre for Biomolecular Research, Faculty of Science Masaryk University, Kotlářská 2, CZ-61137 Brno
TRY OF EDUCATION.      OP Education h^^j
european I V I | f       \   % IHII |
SOCial fund in the I I MINISTRY OF EDUCATION,     OPEcucatlon **2^Jr^
CZeCh republic      EUROPEAN UNION YOUTH AND SPORTS for Competitivness l/4ttA^
INVESTMENTS IN EDUCATION DEVELOPMENT CZ.1.07/2.2.00/15.0233
radical Introduction to Supercomputing 8th Lesson
Contents
> Infinity
job, command overview, aliases
> Application submitting
sander, pmemd, gaussian, parallel runs
> Exercise
parallel runs efficiency of sander, pmemd, gaussian applications
radical Introduction to Supercomputing
8th Lesson
Infinity
> https://lcc.ncbr.muni.cz/whitezone/development/infinity/
> Job
> Command overview
> Aliases
radical Introduction to Supercomputing
8th Lesson
-3-
Command overview
Software management:
• site	logical resources activation
• module	software activation/deactivation
Job management:	
• pqueues	batch system queues accessible to user list
• pnodes	computational nodes accessible to user
• pqstat	list of all jobs in batch system
• pjobs	list of current user jobs in batch system
• psubmit	submitting job to batch system
• pinfo	job information
• pgo	connection to job main node
• paliases	aliases definition
ractical Introduction to Supercomputing
8th Lesson
Job
Job has to keep following conditions:
• Each job is submitted in separate directory
• All input data has to be placed in job directory
• Job directories must not be nested
• Job run is controlled by script (or input file for automatically detected jobs)
• Job script has to be in bash
• Job script must not use absolute paths, all paths are relative to job directory
	
	
	
ractical Introduction to Supercomputing
8th Lesson
Script may be introduced by standard bash interpreter       or special interpreter
, that ensures that script is not run except in batch system. Latter approach prevents damage/overwrite/deleting of already calculated data by unintended script start.
Job submit
Job is submitted in job directory by command psubmit.
psubmit destination job  [resources] [syncmode]
destination (where) is:
• queue_name
• machine_name@queue_name
job is:
• Job script name
• Input file in case of automatically recognized jobs
resources is description of requested job resources, if none given, then 1 CPU is default
syncmode defines way of data transfer mode between job directory and calculation node, default mode is "sync"
ractical Introduction to Supercomputing 8th Lesson
Job run monitoring
Job run may be monitored by command pinfo, this is run either from job directory or in computational node work directory. Other possibilities are commands pjobs and pqstat.
If job is in running mode on computational node, then command pgo can be used to access the node and change to job work directory there.
ractical Introduction to Supercomputing
8th Lesson
-8-
Management files
In job directory there are management files created upon job submission to batch system and its run and finalization. List of files and its contents follows:
• *.info        control file with information about job state
• *.infex       actual script (wrapper), that is run by batch system
• *.infout     standard output of *.infex script, in case of non-standard job termination contains useful informations
• *.nodes list of computational nodes dedicated to job
• *.gpus list of GPU cards dedicated to job
• *.key job unique identifier
• *.stdout standard output of job script
ractical Introduction to Supercomputing
8th Lesson
Running applications
> sander
> pmemd
> gaussian
> Parallel running
radical Introduction to Supercomputing
8th Lesson
-10-
sander program is dedicated to molecular modeling. Detail information is given in: http://ambermd.org
#!/bin/bash
# activate module amber containing applications
# sander a pmemd module add amber
# applications run
sander -0 -i prod.in -p topology.parm7 \
-c input.rst7
[115 Practical Introduction to Supercomputing
8th Lesson
pmemd program is dedicated to molecular modeling. Detail information is given in: http://ambermd.org
#!/bin/bash
# activate module amber containing applications
# sander a pmemd module add amber
# spuštěni aplikace
pmemd -O -i prod, in -p topology .parm7 \
-c input.rst7
[115 Practical Introduction to Supercomputing
8th Lesson
sander/pmemd
Simulation length:
Simulation length (calculation) is defined by keyword (nstlim) given in file prod.in, that is equal to number of integration steps.
Simulation results are in files:
mdout
mdinfo <-- contains statistics information, for example how many ns per day is
program able to calculate
mdcrd restrt
ractical Introduction to Supercomputing
8th Lesson
-13-
sander/pmemd - parallel run
Parallel run differs only in submission to batch system by command psubmit. Rest of input remains same! (same input file, job script).
$ psubmit short test_sander ncpus=l
May be omitted
*.stdout
Module build:  amber:12.0:x86 64:single
Calculation node:
S %CPU %MEM		TIME+ COMMAND
R		
R	EKH	
$ psubmit short test sander ncpus=2
*.stdout
Module build:  amber:12.0:x86 64:para
Calculation node:
%CPU %MEM      TIME+ COMMAND			
IS	.7 i		
ractical Introduction to Supercomputing
8th Lesson
Exercise
Input data:
/home/kulhanek/Data/2115/data/sander/small
1. Run job on 1CPU on cluster wolf.
2. Run job on 2CPU on cluster wolf.
ractical Introduction to Supercomputing
8th Lesson
gaussian
gaussian program is dedicated to quantum-chemical calculations. Detail information are to be found at: http://www.gaussian.com
# run application g09 input
#!/bin/bash
# activate modeule gaussian module add gaussian
Input file
no extension
>115 Practical Introduction to Supercomputing
8th Lesson
gaussian
Calculation length:
Calculation length is limited by maximum optimisation steps number (MaxCycle in in file).
Calculation result is in file:
input.log
ractical Introduction to Supercomputing
8th Lesson
gaussian - parallel run
Parallel run has to be set in resource definition for command psubmit
nd input file for
input.com (first line)
%NProcShared=l
input.com (first line)
%NProcShared=4
Identical number
$ psubmit short test gaussian ncpus=l       $ psubmit short test gaussian
pus=4
May be omitted
Calculation node:
100 1.2
0 0,1
1:01,25 1502,exe 1:38 . 57 pt>s_moin
Calculation node:
S SCPU %f
Rl 333 11.1 S\ 0/0,1
TIME+ COMMAND
0:43.38 I502.exe 0:00.30 init
>115 Practical Introduction to Supercomputing
8th Lesson
gaussian - parallel run, II
Auto detection may be used by gaussian calculation submiting.
No auto detection:
input.com (first line)
%NProcShared=4
Identical number!
$ psubmit short test gaussian ncpus=4
With auto detection:
input.com (does not need to contain
%NProcShared=4
$ psubmit short input.com ncpus=4
ractical Introduction to Supercomputing
8th Lesson
Exercise
Input data:
/home/kulhanek/Data/2115/data/gaussian
1. Run job with 1CPU on cluster wolf.
2. Run job with 2CPU on cluster wolf.
ractical Introduction to Supercomputing
8th Lesson
Exercise
> parallel runs efficiency of
> sander
> pmemd
> gaussian
radical Introduction to Supercomputing 8th Lesson
Exercise LVII.l
Exercise goal is to determine scaling of sander, pmemd, gaussian applicationon cluster SOKAR in range of CPU number 1, 2, 4, 8, 16, 32 a 64. For each CPU number select simulation length appropriate for particular CPU number. Then state theoretical simulation length, real speedup, and real CPU usage percent. Plot graph with real speedup as function of CPU number. Compare plot with ideal scaling graph.
Do testing calculation on cluster WOLF. Final calculations has to be done on cluster SOKAR. Make sure, that you have ssh keys setup done on SOKAR.
There are two nodes available with 64 CPU in queue long. Nodes has property c2115.
Request 64 CPU for all jobs during submit, proper CPU number will be selected "lowered" to desired value in script.
Input data are on cluster WOLF in directories:
/home/kulhanek/Data/2115/data /pmemd/medium/ /home/kulhanek/Data/2115/data /pmemd/small/ /home/kulhanek/Data/2115/data /pmemd/big/ /home/kulhanek/Data/2115/data /sander/medium/ /home/kulhanek/Data/2115/data /sander/small/ /home/kulhanek/Data/2115/data /sander/big/ /home/kulhanek/Data/2115/data /gaussian/
> Practical Introduction to Supercomputing 8th Lesson
CPU number adjusting for testing
N = desired CPU number
INF_NCPU=N AMS_NCPU=$INF_NCPU
# activate module amber containing applications
# sander a pmemd module add amber
$ psubmit long test_sander ncpus=64,props=c2115
N
Always 64
input.com (first line)
%NProcShared=N    <- N = desired CPU number
$ psubmit long test_gaussian ncpus=64,props=c2115
Always 64
radical Introduction to Supercomputing 8th Lesson
Exercise LVII.2
According to manual on MetaCentra web pages run job in gaussianu, take input file fro previous exercise.
ractical Introduction to Supercomputing
8th Lesson