airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Chau <dch...@binghamton.edu>
Subject Job Submission Limit
Date Mon, 03 Aug 2015 14:50:54 GMT
Hey Devs,

Just wanted to get some input on our plan to implement the queue throttling feature.

Batch Queue Throttling:
- in Orchestrator the current submit() method in GFACPassiveJobSubmitter publishes to rabbitmq
immediately
- instead of publishing immediately, we should have the submit method pass the message to
a new component, BatchQueueClass (tentative name), to check when we can unload jobs to submit

Adding BatchQueueClass
- setup a new table(s) to contain computer resource names and their corresponding queues’
current job numbers and maximum job limits
- current data models only have information on max job submission limit for a queue but no
information on how many jobs are currently submitted, so the idea is to implement a increment/decrement
counter in the new table so when a job is submitted, a call to increment the counter will
be made, and when a job is finished a call to decrement the counter will be made
- once that is complete, BatchQueueClass needs to periodically check the new table to see
if the current job number < queue job limit. If it is then we can just pop jobs off to
submit them until we hit the job limit

How does this sound?

Doug
Mime
View raw message