airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pierce, Marlon" <>
Subject Re: Job Submission Limit
Date Mon, 03 Aug 2015 17:19:05 GMT
This seems like a good thing to summarize in a diagram.  In general, you should use or extend
existing Airavata components for storing job information and tracking job state unless there
is a really compelling reason not too.


From: Raminderjeet Singh <<>>
Reply-To: "<>" <<>>
Date: Monday, August 3, 2015 at 1:05 PM
To: "<>" <<>>
Subject: Re: Job Submission Limit

I see you point. Other thing can be user can run few jobs manually or using a different service.
Its a good idea to get it from the resource but we need to be careful about our poll as we
don't want to do it to frequently.

On Mon, Aug 3, 2015 at 12:57 PM, John Weachock <<>>
Sorry - my email might have been misworded slightly. I was referring to the number of submitted
jobs, not the limit. For example, if we record the number of submitted jobs independently,
we may run into issues where our count doesn't match up with the actual number of submitted
jobs. I think it's best that we poll for the number of submitted jobs.

On Mon, Aug 3, 2015 at 12:53 PM, Raminderjeet Singh <<>>
Usually these limits are set as a policy by the resource provider and do not change usually.
As long as we have a place holder to configure/change it in Airavata for a user/gateway, we
don't need to get it from a resource.

On Mon, Aug 3, 2015 at 12:33 PM, John Weachock <<>>
I think it would be best for us to not maintain our own record of the job limit - we need
to remember that jobs will be submitted to these resources using the community accounts through
other methods as well. I think I remember someone mentioning that it would be ideal to poll
the resources for their limits - can anyone confirm that we can do this?

On Mon, Aug 3, 2015 at 12:24 PM, Douglas Chau <<>>
Hmm @shameera, that's very true. Perhaps, we can store the submission requests in registry.
In the event that orchestrator goes down we can recover them through registry afterwards.

@Yoshimito, I didn't think about that - will take it into consideration.Thanks for the insight!

On Mon, Aug 3, 2015 at 12:11 PM, K Yoshimoto <<>>

I think you also want to put in a check for successful submission,
then take appropriate action on failed submission.  It can be
difficult to keep the submission limit up-to-date.

On Mon, Aug 03, 2015 at 11:03:46AM -0400, Douglas Chau wrote:
> Hey Devs,
> Just wanted to get some input on our to plan to implement the queue
> throttling feature.
> Batch Queue Throttling:
> - in Orchestrator, the current submit() function in GFACPassiveJobSubmitter
> publishes jobs to rabbitmq immediately
> - instead of publishing immediately we should pass the messages to a new
> component, call it BatchQueueClass.
> - we need a new component BatchQueueClass to periodically check to see when
> we can unload jobs to submit
> Adding BatchQueueClass
> - setup a new table(s) to contain compute resource names and their
> corresponding queues' current job numbers and maximum job limits
> - data models in airavata have information on maximum job submission limits
> for a queue but no data on how many jobs are currently running
> - the current job number will effectively act as a counter, which will be
> incremented when a job is submitted, and when a job is completed
> - once that is done, BatchQueueClass needs to periodically check new table
> to see if the user's requested queue's current job number < queue job
> limit. If it is then we can pop jobs off and submit them until we hit the
> job limit; if not, then we wait until the we're under the job limit.
> How does this sound?
> Doug

View raw message