airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Chau <>
Subject Job Submission Limit
Date Mon, 03 Aug 2015 15:03:46 GMT
Hey Devs,

Just wanted to get some input on our to plan to implement the queue
throttling feature.

Batch Queue Throttling:
- in Orchestrator, the current submit() function in GFACPassiveJobSubmitter
publishes jobs to rabbitmq immediately
- instead of publishing immediately we should pass the messages to a new
component, call it BatchQueueClass.
- we need a new component BatchQueueClass to periodically check to see when
we can unload jobs to submit

Adding BatchQueueClass
- setup a new table(s) to contain compute resource names and their
corresponding queues' current job numbers and maximum job limits
- data models in airavata have information on maximum job submission limits
for a queue but no data on how many jobs are currently running
- the current job number will effectively act as a counter, which will be
incremented when a job is submitted, and when a job is completed
- once that is done, BatchQueueClass needs to periodically check new table
to see if the user's requested queue's current job number < queue job
limit. If it is then we can pop jobs off and submit them until we hit the
job limit; if not, then we wait until the we're under the job limit.

How does this sound?


View raw message