The accounting system monitors usage in real time. Usage is calculated as the sum of usage by jobs that have completed, and potential usage by jobs that are running currently. When a job is submitted, the application reserves the maximum time it might consume in your account. When it completes, any unused time is released to your account for use. Note that accounts are denominated in CPU hours, and GPU usage is charged at 20 x CPU; i.e. 1 GPU hr = 20 CPU hrs. If a submisison is blocked due to lack of time, you can decrease the maximum run time if you want to submit more jobs. The ley is that the jobs complete in the specified time, or can be restarted if they do not complete in the alotted maximum time.
Example 1. Let's say you have 30,000 hours in your account and you want to submit 4 jobs that require 84 cores each, and they are configured with a maximum run time of 10 hours. When you submit all four jobs, the task page will show you have consumed 3,360 CPU hours initally; this represents the maximum amount of time these jobs would consume (10 x 84 x 4) if they run for the configured maximum time. While all the jobs are running, 3,360 hours will be reserved in your allocation. If all four jobs complete after 30 minutes, the charge will be 0.5 hours x 84 cores x 4 jobs = 168 CPU hours. At that time, all but 168 of the 3,360 hours held in reserve are released, and your account balance will increase by 3,360 - 168 = 3,192 hours. The total amount charged is the amount actually used by the jobs.
Example 2. On the other hand, say you have 30,000 hours in your account, and you want to submit the same four jobs to run for 168 rather than 10 hours. The application assumes that they will run for the full 168 hours, so each job you launch will reserve 14,122 core hours (168 x 84 x 1) against the balance in your account. The application will allow you to submit the first two jobs, bringing you to a potential consumption of 28,244 core hours. The value 28,244 will appear as the amount of CPU hours consumed in your task page while the jobs are running. When you try to submit the third job, the application will present an error message, saying you have reached your limit. If both jobs run the full 168 hours, you won't be able to submit the third job unless you purchase access to more time. However, if both jobs end after 1 hour, the jobs will have consumed 168 hours (1 hour x 84 cores x 2 jobs), this is charged to your account, but the time remaining in your account will increase by 28,076 (28,244 - 168) hours; this unused time is released back to your account. At that time, you would be able to submit the remaining two jobs.
Example 3. Suppose you have a subscription at level 4, 50,000 CPU hours. You have run some jobs that have completed, and the total consumption on your account is 30,850 CPU hours; your available account balance is now 19,150 CPU hours. You want to run 4 BEAST jobs configured for 120 hours each, and these jobs require 1 GPU. When you submit the first job, the maximum possible total consumption will be (120 hours * 4 GPU = 80 CPUs) = 9,600 CPU hours. The application assumes the job will run for the maximum run time, so it reserves this amount of time in your account. Your consumption (displayed in the interface) will increase to 40,450 CPU hours (30,850 + 9,600). If you try to submit a second identical job, consumption would increase to 50,050 (40,450 + 9,600) hours, which is above the allocation balance (50,000), so the submission will be blocked. If the first job completes after 10 hours, the balance will be adjusted. The first job consumed (10 hours * 4 GPUs = 80 CPUs) = 800 CPU hours, and the balance will adjust to 30,850 + 800 = 31,650. Now you can submit the second job, which requires 9,600 hours
If this is not clear, please let us know.
CIPRES – Cyberinfrastructure for Phylogenic Research