Submissons are halted when the potential consumption by your running jobs plus the consumption of your completed jobs reach your current submission limit.
Here is an example:
Assume a user has a submission limit of 25,000 cpu hours; that is, a subscription at level 3.
The user has already run some jobs that have all completed, and their total cpu consumption is 10,050 cpu hours. The user decides they would like to run 4 BEAST jobs and configures them to run for 120 hours each. These jobs each require 4 gpus.
The consumption of a running job is calculated assuming it will run for the maximum run time. When the user submits the first job, the maximum possible total consumption will be (120 hours) * (4 gpus = 84 cpus) = 10080 cpu hours. The application assumes the job will run for the maximum run time when it is submitted, so the user's consumption will increase to 20130 cpu hours (10,050 + 10,080).
Similarly, when a second identical job is submitted, the user’s consumption will increase to 30210 (20,130 + 10,080) hours, which is above the maximum time allowed (25,000) and the submission willc be blocked.
If the first job completes after 10 hours, the cpu hours consumed will be adjusted to the actual amount of time consumed. The first job will have consumed (10 hours) * (4 gpus = 84 cpus) = 840 cpu hours. Accordingly, the consumption for completed jobs will be 10,050+ 840 = 10,890. Now the user can submit their second job, which if it is configured as before, will be assumed to require 10080 hours. It should be clear that you can decrease the maximum run time if you want to submit more jobs, as long as the jobs complete in the specified time, or if they can be restarted.
If this is not clear, please contact us.
CIPRES – Cyberinfrastructure for Phylogenic Research