Use on Caparmor

Warning

This part is dedicated to advanced Caparmor users. To run a MARS configuration, informations contained in the MARS environment are sufficient.

  • OMP run : XXXX lien vers manuel
  • MPI run : XXXX
  • hybrid run : XXXX

PBS command

qsub

to submit a job (under the form of shell_script) into a batch queue:

qsub –l select=1 :ncpus=8 –o res -q parallel8 shell_script

qstat

to watch your job:

qstat -u user

qdel

to kill a job:

qstat -u user   (to get the job ID)
qdel jobid
qdel -W force jobid

QSUB

-q: specify the name of the queue according to the number of cpus you want to use:

-q queue_name
Name of the queues on Caparmor
type of the job queue name nb of nodes max nb of cpus elapsed time
sequential sequentiel 1 1 300h
OMP parallel8 1 8 72h
MPI / hybrid parallel8 1 8 72h
MPI / hybrid parallel32 2 < nb <= 4 32 48h
MPI / hybrid parallel64 5 < nb <= 8 64 24h
MPI / hybrid parallel256 9 < nb <= 32 256 18h

You get the the total number of core (and consequently the name of the queue) from select*ncpus

select: number of nodes (if ncpus=8)
ncpus and mpiprocs are defined for each node (or each select number)
ncpus: * number of OMP threads (=ncpus/mpiprocs) when using OMP
(do not specify OMP_THREADS or set it to OMP_THREADS=ncpus/mpiprocs in the batch)
: * number of cores used per node (=mpiprocs if you want to make sure not to share de node with someone else)
mpiprocs: number of mpi cores per node
-np: total number of MPI cores (select*mpiprocs)

  • MPI run (best use):

    qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=8 batch_mpi   | -np = 16 in batch_mpi

  • hybrid (MPI+OMP) run (best use knowing there are 8 cores per node)

    8 OMP threads, 1 MPI core per node and 4 nodes
    qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=1 batch_mpiomp   | in batch_mpiomp -np = 4
                                                                 |                 setenv OMP_THREADS 8
    2 OMP threads per node, 8 nodes in total, different ways with -np 16
    qsub -q parallel32 -l select=16:ncpus=2:mpiprocs=1 batch_mpiomp  | in batch_mpiomp -np = 16
                                                                 |                 setenv OMP_THREADS 2

qsub -q parallel32 -l select=8:ncpus=4:mpiprocs=2 batch_mpiomp   | in batch_mpiomp -np = 16
                                                                 |                 setenv OMP_THREADS 2

qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=4 batch_mpiomp   | in batch_mpiomp -np = 16
                                                                 |                 setenv OMP_THREADS 2
with these last choice, you will share the node nobody

  • to specify the memory and make sure you will not share the memory with other runs (the maximum is 24Gb):

    -l select=8:ncpus=4:mpiprocs=4:mem=23.8gb

  • by default, if not specified mpiprocs=1

    qsub -q parallel256 -l select=64:ncpus=4           batch_mpiomp  | in batch_mpiomp -np = 64
                                                                 |                 setenv OMP_THREADS 4

if you do not want to share the node, prefer :
qsub -q parallel256 -l select=32:ncpus=4:mpiprocs=2 batch_mpiomp | in batch_mpiomp -np = 64
                                                                 |                 setenv OMP_THREADS 4

  • exemple with error

    qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=1 batch_mpiomp   | in batch_mpiomp -np = 32
                                                                 |                 setenv OMP_THREADS 8

choose -np 4 instead

  • how to use only 2 threads and 1 MPI cpus per node (and I freeze other cores for any other use)

    qsub -q parallel256 -l select=16:ncpus=8:mpiprocs=1 batch_mpiomp | in batch_mpiomp -np = 16
                                                                 |                 setenv OMP_THREADS 2

    In this last exemple you use only 32 cores but freeze 128 cores. Do not do that !

batch for MPI jobs

  • batch for intel MPI:

    #PBS -S /bin/csh
#--- Affectation du nom au job lance
#PBS -N name_job
##affectation du nom au fichier erreurs
#PBS -e erreur.txt
## Affectation du nom au fichier sorties
#PBS -o sortie.txt
#PBS -m e
#PBS -l walltime=06:00:00
#PBS -q parallel8
#PBS -l select=1:ncpus=8:mpiprocs=8

#----------------------------------------
#-- Chargement pour accès commande module
#----------------------------------------
source /usr/share/modules/init/ksh

#----------------------------------------
#-- Load the module
#----------------------------------------
module purge
module load intel-mpi/4.0.0.028
module load intel-comp/11.1.073

# cd to the directory from which you submitted your job
cd $PBS_O_WORKDIR

time runmpi -n 8 executable
time runmpi [-genv I_MPI_USE_DYNAMIC_CONNECTIONS 0] -np 8 time executable

#denistina

  • batch for MPT (SGI MPI):

    #PBS -S /bin/csh
#--- Affectation du nom au job lance
#PBS -N name_job
##affectation du nom au fichier erreurs
#PBS -e erreur.txt
## Affectation du nom au fichier sorties
#PBS -o sortie.txt
#PBS -m e
#PBS -l walltime=06:00:00
#PBS -q parallel8
#PBS -l select=1:ncpus=8:mpiprocs=8

#----------------------------------------
#-- Chargement pour accès commande module
#----------------------------------------
source /usr/share/modules/init/ksh

#-- Load the module
#
module purge
module load mpt/2.01
module load intel-comp/11.1.073

# environment variables for MPT
# number and depth of deribative type (do not compile if noti large enough)
setenv MPI_TYPE_DEPTH 50
setenv MPI_TYPE_MAX 50000
setenv MPI_DEFAULT_SINGLE_COPY_OFF "ON"
# optimisation
#setenv MPI_IB_RAILS 2
#setenv MPI_IB_XRC
#setenv MPI_FASTSTART
#
# cd to the directory you submitted your job
cd $PBS_O_WORKDIR

time mpiexec_mpt -np 8 time ./mars_exe

batch for hybrid jobs

Same as for MPI/MPT jobs except that you must specify

  • the number of OMP threads:

    setenv OMP_THREADS 4

  • OMP options:

    setenv OMP_SCHEDULE "dynamic,1"

  • and choose the mpiprocs number in consequence