.. _dvp.parallel.caparmor: **Use on Caparmor** =================== .. warning:: This part is dedicated to advanced Caparmor users. To run a MARS configuration, informations contained in the MARS environment are sufficient. * OMP run : XXXX lien vers manuel * MPI run : XXXX * hybrid run : XXXX PBS command ^^^^^^^^^^^ *qsub* ------ to submit a job (under the form of shell_script) into a batch queue:: qsub –l select=1 :ncpus=8 –o res -q parallel8 shell_script *qstat* ------- to watch your job:: qstat -u user *qdel* ------ to kill a job:: qstat -u user (to get the job ID) qdel jobid qdel -W force jobid QSUB ^^^^ **-q**: specify the name of the queue according to the number of cpus you want to use:: -q queue_name .. csv-table:: **Name of the queues on Caparmor** :header: "type of the job", "queue name", "nb of nodes","max nb of cpus","elapsed time" :widths: 20, 20, 20, 20, 20 "sequential", "sequentiel", "1", "1", "300h" "OMP", "parallel8", "1", "8", "72h" "MPI / hybrid", "parallel8", "1", "8", "72h" "MPI / hybrid", "parallel32", "2 < nb <= 4", "32", "48h" "MPI / hybrid", "parallel64", "5 < nb <= 8", "64", "24h" "MPI / hybrid", "parallel256", "9 < nb <= 32", "256", "18h" You get the the total number of core (and consequently the name of the queue) from **select*ncpus** | **select**: number of nodes (if ncpus=8) | **ncpus** and **mpiprocs** are defined for each node (or each **select** number) | **ncpus**: * number of OMP threads (**=ncpus/mpiprocs**) when using OMP | (do not specify OMP_THREADS or set it to **OMP_THREADS=ncpus/mpiprocs** in the batch) | : * number of cores used per node (**=mpiprocs** if you want to make sure not to share de node with someone else) | **mpiprocs**: number of mpi cores per node | **-np**: total number of MPI cores (**select*mpiprocs**) * **MPI run** (best use):: qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=8 batch_mpi | -np = 16 in batch_mpi * **hybrid (MPI+OMP) run** (best use knowing there are 8 cores per node) | 8 OMP threads, 1 MPI core per node and 4 nodes :: qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=1 batch_mpiomp | in batch_mpiomp -np = 4 | setenv OMP_THREADS 8 | 2 OMP threads per node, 8 nodes in total, different ways with -np 16 :: qsub -q parallel32 -l select=16:ncpus=2:mpiprocs=1 batch_mpiomp | in batch_mpiomp -np = 16 | setenv OMP_THREADS 2 qsub -q parallel32 -l select=8:ncpus=4:mpiprocs=2 batch_mpiomp | in batch_mpiomp -np = 16 | setenv OMP_THREADS 2 qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=4 batch_mpiomp | in batch_mpiomp -np = 16 | setenv OMP_THREADS 2 with these last choice, you will share the node nobody * to specify the **memory and make sure you will not share the memory** with other runs (the maximum is 24Gb):: -l select=8:ncpus=4:mpiprocs=4:mem=23.8gb * by default, if not specified **mpiprocs=1** :: qsub -q parallel256 -l select=64:ncpus=4 batch_mpiomp | in batch_mpiomp -np = 64 | setenv OMP_THREADS 4 if you do not want to share the node, prefer : qsub -q parallel256 -l select=32:ncpus=4:mpiprocs=2 batch_mpiomp | in batch_mpiomp -np = 64 | setenv OMP_THREADS 4 * exemple with error :: qsub -q parallel32 -l select=4:ncpus=8:mpiprocs=1 batch_mpiomp | in batch_mpiomp -np = 32 | setenv OMP_THREADS 8 choose -np 4 instead * how to use only 2 threads and 1 MPI cpus per node (and I freeze other cores for any other use) :: qsub -q parallel256 -l select=16:ncpus=8:mpiprocs=1 batch_mpiomp | in batch_mpiomp -np = 16 | setenv OMP_THREADS 2 In this last exemple you use only 32 cores but freeze 128 cores. Do not do that ! batch for MPI jobs ^^^^^^^^^^^^^^^^^^ - batch for intel MPI:: #PBS -S /bin/csh #--- Affectation du nom au job lance #PBS -N name_job ##affectation du nom au fichier erreurs #PBS -e erreur.txt ## Affectation du nom au fichier sorties #PBS -o sortie.txt #PBS -m e #PBS -l walltime=06:00:00 #PBS -q parallel8 #PBS -l select=1:ncpus=8:mpiprocs=8 #---------------------------------------- #-- Chargement pour accès commande module #---------------------------------------- source /usr/share/modules/init/ksh #---------------------------------------- #-- Load the module #---------------------------------------- module purge module load intel-mpi/4.0.0.028 module load intel-comp/11.1.073 # cd to the directory from which you submitted your job cd $PBS_O_WORKDIR time runmpi -n 8 executable time runmpi [-genv I_MPI_USE_DYNAMIC_CONNECTIONS 0] -np 8 time executable #denistina - batch for MPT (SGI MPI):: #PBS -S /bin/csh #--- Affectation du nom au job lance #PBS -N name_job ##affectation du nom au fichier erreurs #PBS -e erreur.txt ## Affectation du nom au fichier sorties #PBS -o sortie.txt #PBS -m e #PBS -l walltime=06:00:00 #PBS -q parallel8 #PBS -l select=1:ncpus=8:mpiprocs=8 #---------------------------------------- #-- Chargement pour accès commande module #---------------------------------------- source /usr/share/modules/init/ksh #-- Load the module # module purge module load mpt/2.01 module load intel-comp/11.1.073 # environment variables for MPT # number and depth of deribative type (do not compile if noti large enough) setenv MPI_TYPE_DEPTH 50 setenv MPI_TYPE_MAX 50000 setenv MPI_DEFAULT_SINGLE_COPY_OFF "ON" # optimisation #setenv MPI_IB_RAILS 2 #setenv MPI_IB_XRC #setenv MPI_FASTSTART # # cd to the directory you submitted your job cd $PBS_O_WORKDIR time mpiexec_mpt -np 8 time ./mars_exe batch for hybrid jobs ^^^^^^^^^^^^^^^^^^^^^ Same as for MPI/MPT jobs except that you must specify * the number of OMP threads:: setenv OMP_THREADS 4 * OMP options:: setenv OMP_SCHEDULE "dynamic,1" * and choose the mpiprocs number in consequence