by David E. Konerding, Ph.D. and Conrad Huang
In the past CGL has provided a command called "submit" (type "man submit" to learn more about it) which was used for running long-term jobs. submit allows users to submit a job and log out, while the job continues running at lowered priority. With the advent of the multi-node socrates cluster, a more sophisticated job submission process is in place: the "OpenPBS" batch queueing system. We recommend submitting jobs that require over an hour of processing be run in the batch queueing system.
The "submit" command has been replaced with a script that emulates the old behavior, but we strongly recommend that you use the new "qsub" command instead to maximize the functionality of PBS.
There are two commonly used queues for executing jobs: "regular" and "verylong". The "regular" queue is for jobs that require less than 24 hours of CPU time, while "verylong" can handle jobs requiring up to 28 days of CPU time. Note that CPU time differs from wall clock time, with wall clock time being typically longer (i.e., your job will not get 100% of the CPU all the time, so it takes longer to run than actual CPU time). An exception to this rule occurs when your job uses multiple processors (e.g., running "blast" with "-p 3" uses three CPUs for the job); in this case, the wall clock time may be as low as one third of the CPU time because
There may be filled with up to seven active jobs in the "regular" queue and up to five active jobs in the "verylong" queue. Subsequent jobs are placed in a "wait" state and only started when an active job terminates. There is no user-specific limits on the number of jobs, either active or submitted. However, we expect users to demonstrate discretion when using PBS. Running a few active job at once when usage is light is probably fine; submitting 50 jobs at once and locking out other users will result in draconian actions by CGL staff.wall clock time ~= (total CPU time) / (number of active CPUs)
Both the "regular" and "verylong" queues only handle jobs that run on a single node. If you need to run a multi-node job, please contact CGL staff.
Here is an example job script file which simply echos a string to the standard output:> qsub script
If you create the file "script" with the contents above, and "qsub" it, you will almost immediately get an email message saying that the job has been started, and almost immediately after, that the job has completed. If a lot of other people have jobs running, your job might not start immediately. You can check on the status of your job by using the "qstat" command:## The next line is an instruction to PBS: it tells PBS to email ## you when your job "b"egins, "a"borts, and "e"nds. #PBS -m bae echo "Hello world!" ## end of batch script
In this case, only one job is running: its name is "7998.socrates", it's being run for user "dek", and has been running for 11 hours and 50 minutes. The "state" of the job is "R" which is short for running. If the job has not yet started running, the state will be "Q" (for queued). If you happen to type "qstat" just as your job is finishing, the state will be "E" (for ending).> qstat Job id Name User Time Use S Queue ---------------- ---------------- ---------------- -------- - ----- 7998.socrates script dek 11:50:10 R verylong
PBS will place the job in the "verylong" queue and execute it. Notice that the job need not actually take 40 hours (the job above will take almost no time at all), but the CPU time estimate will force it into the "verylong" queue. If a submitted job were to run for entire estimated required CPU time, it would be terminated by PBS, so it is always better to overestimate the CPU time requirement than to underestimate.## The next line is an instruction to PBS: it tells PBS ## that your job will take 40 hours, or 240 minutes. #PBS -l cput=40:00:00 #PBS -m bae date ls date ## end of batch script
The command line you would type to do such a search might look like:
The -a 3 flag tells blast to run with 3 CPUs. -d nr uses the nonredundant database. -p blastp tells blast it's a protein sequence. -i myseq.fa uses a query sequence stored in the file myseq.fa. -o hits.out saves the output to the file hits.out.> blastall -a 3 -d nr -p blastp -i myseq.fa -o hits.out
When you submit a job to PBS, by default you get to use only 1 CPU. But there is a way to ask PBS you want to use 3 CPUs on one node. The job script would look like this:
As you can see the job script file has a new line:#PBS -m e #PBS -l nodes=1:ppn=3,cput=36:00:00 blastall -a 3 -d nr -p blastp -i myseq.fa -o hits.out ## end of batch script
This says to PBS: "I am requesting that my job run on 1 node, with 3 processors per node" and the job will take up to 36 CPU hours. PBS will automatically route this job to the "verylong" queue because the "regular" queue only handles jobs taking up to 24 hours of CPU time. If your job will take more than 24 CPU hours and you omit the "cput" estimate, then it will be run in the "regular" queue and be terminated after 24 CPU hours.#PBS -l nodes=1:ppn=3,cput=36:00:00
now your script will look like:cd $PBS_O_WORKDIR
This will cause your script to go to the directory that you submitted the job from.#PBS -m e #PBS -l nodes=1:ppn=3 cd $PBS_O_WORKDIR blastall -a 3 -d nr -p blastp -i myseq.fa -o hits.out ## end of batch script
When run, the job script will be executed on one of the hosts, and it will have an environment variable called PBS_NODEFILE. If you look at the contents of the file listed in PBS_NODEFILE it might look like this:#PBS -l nodes=1:ppn=3
As you can see, thymine have been assigned to the job, and is listed 3 times. This is because we asked for 3 processors. Since your program just takes a list of hosts on its command line, your job script could have the following:> cat $PBS_NODEFILE thymine.cgl.ucsf.edu thymine.cgl.ucsf.edu thymine.cgl.ucsf.edu
this would expand to:myprogram `cat $PBS_NODEFILE`
and your program would end up using 3 cpus on thymine.myprogram thymine.cgl.ucsf.edu thymine.cgl.ucsf.edu thymine.cgl.ucsf.edu
Technically speaking, this process does not actually assign a specific CPU to a specific application; it's merely a side effect of the multiprocessor scheduler that the operating system provides. This means, unfortunately, that if somebody else logs into the system and runs some job that uses up a whole CPU, you won't actually get 3 CPUs worth of processing, so you may get suboptimal performance. It is in everybody's best interest not to abuse the queueing system for his own short term interests. Further, CGL has set up queueing policies which try to avoid having system service processes like the web server from interfering with PBS jobs.
An example script which would always run on uracil (you must use the fully qualified name, "uracil.cgl.ucsf.edu"):thymine.cgl.ucsf.edu state = free np = 4 properties = socrates ntype = cluster jobs = 0/636.socrates.cgl.ucsf.edu uracil.cgl.ucsf.edu state = free np = 4 properties = socrates ntype = cluster cytosine.cgl.ucsf.edu state = free np = 3 properties = socrates ntype = cluster adenine.cgl.ucsf.edu state = free np = 2 properties = socrates ntype = cluster
An example script which would always run on an ES45 node:#PBS -m e #PBS -l nodes=uracil.cgl.ucsf.edu echo `hostname` # end of PBS script
#PBS -m e #PBS -l nodes=es45 echo `hostname` # end of PBS script
It provides information on what environment variables to use, what options are available, how input and output streams are handled, and much more.man qsub