airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Orchestrator Real time Job submission improvement
Date Thu, 11 Sep 2014 18:42:40 GMT
Hi Shameera,

Can you please map this to the diagram at [1]? Will the HPCPullMonitor be equivalent to the
BufferedQueue we discussed on the architecture list? 

Suresh
[1] - https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Metascheduler

On Sep 11, 2014, at 10:29 AM, Shameera Rathnayaka <shameerainfo@gmail.com> wrote:

> Hi devs, 
> 
> I am going to implement the $Subject
> 
> Requirement: Introduce a max job submission count for a given resource under a given
username.
> 
> Abstraction: When user submits a new experiment to the airavata, user selects the resource
(Machine) where airavata should run that experiment (Job). That resource may have job count
restriction like under one user there can only be have X number of jobs either in Q or R state.
So we need to handle this at Orchestrator level rather than handing over the experiment to
GFac to submit the jobs where it gets rejected because of that restriction. To do that Orchestrator
need to know the job count of particular user in that given resource.  
>  
> 
> Implementation:  HPCPullMonitor will write stat data to zookeeper, zookeeper path would
be something like /stat/{username}/{machine}/jobs/{count}. Orchestrator will register a watcher
for this data change and that watcher will trigger when any GFac node(Monitor component) update
the job status realtime. Finished jobs will immediately decrement the count and these changes
will replicate in Orchestrator with ZK watches.
> 
> Thanks,
> Shameera.
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/


Mime
View raw message