Running jobs on Clusty

Clusty resources and scheduling jobs are governed by torque and maui. These two services make sure that all resources available to Clusty are shared and all jobs submitted to Clusty scheduled fairly.

To submit a job to Clusty, you need to write a Portable Batch System (PBS) shell script. The table below lists the most useful directives.

Option:
Description:
#PBS -N jobNameAssign the name of the job. Job stdout and stderr will have this name.
#PBS -l nodes=nodename:ppn=procsDelegate job to the node nodename and procs number of processors.
#PBS -l walltime=hh:mm:ssAssign the maximum wall time for the job.
#PBS -Vexport all environment variables to the job.
#PBS -M <user@email>provide an email to which job alerts should be sent.
#PBS -m abejob alerts to be emailed: b for begin, e for end, a for abort.
#PBS -o /path/to/stdoutprovide a path to the job's standard output file (written upon job completion).
#PBS -e /path/to/stderrprovide a path to the job's standard error file (written upon job completion).


A typical example for a PBS script (run.sh) would be:

#PBS -V
#PBS -N myMPIjob
#PBS -l nodes=node3:ppn=48
#PBS -l walltime=24:00:00
#PBS -M john.doe@somewhere.net
#PBS -m abe

cd $PBS_O_WORKDIR
mpirun python my_mpi_job.py

This delegates a job to node3 and requests 48 processors. It also sets the maximum wall time of 1 day.

To submit this job to Clusty, use the qsub command:

qsub run.sh

The table below lists the most common commands for managing and monitoring your job:

Command:
Description:
qnodes Displays information on all nodes (alias: pbsnodes)
showbf Displays number of free processors
showstart job.ID Provides an estimate of the expected start time of the named job
checkjob job.ID Displays a detailed description of the named job
qstat -a job.ID Displays statistics for the named job
qstat -f job.ID Displays all the gory detailes for the named job
qdel job.ID Interrupts the running job and removes it from the queue
qhold job.ID Hold the job in queue
qrls job.ID Release the job from hold


Some interesting situations:

Q: I don't care which nodes end up doing the work; how do I submit a job to any free node?
A: Use qsub -l nodes=N:ppn=M (or, alternatively, #PBS -l nodes=N:ppn=M in the script), where N is a number of requested nodes and M is a number of requested processors. Thus, qsub -l nodes=4:ppn=4 will submit a job to any 4 nodes and request 4 processors on each.
Q: Can I submit a job to n processors irrespective of the node, i.e. to whichever ones are free?
A: Yes, by providing -l nodes=procs. Torque is clever enough (after faking the number of nodes under the hood) to interpret the passed number of nodes as virtual nodes, i.e. procs. Thus, if showbf shows N free processors, -l nodes=N will schedule a job for them no matter which node(s) they are on.
Q: I started a job but I either forgot to set or underestimated the wall time for my job. How can I increase it on the fly?
A: You can't, but I can. Email me the job.ID and the desired wall time, and I'll change it. Note, however, that the maximum wall time is 2 days. If your program requires more time than 2 days, rewrite it into smaller blocks.
Q: Is there anything I should be aware of when running my program on Clusty?
A: Yes! Don't run anything directly, least of all mpirun. That will circumvent the scheduler and invoke everyone's wrath. The head node can be used for interactive programs (IDL, IRAF, ...), but as soon as you are running a compute job, you must submit it via qsub.
Q: I tried to run mpirun through a PBS script but stderr gives me a bunch of ORTE_ERROR_LOG File open failure errors. What gives?
A: That happens if you don't export the environment (the -V switch or #PBS -V). If that doesn't help, export the shell explicitly (-S /bin/bash). If that doesn't help, please report it. It happened to me a couple of times but I'm not entirely sure what exactly solved it.
Q: This is so awesome! What can I do to make it up to you?
A: I accept Belgian beer and Californian full-bodied red wine donations. ;)

How to contact me? Email me!

Finally, stuff that probably doesn't interest you as a user but might interest you as a prospective maintainer.