airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anuj Bhandar <bhandar.a...@gmail.com>
Subject Re: Building a Resource Allocation Service For Airavata
Date Tue, 07 Feb 2017 22:22:32 GMT
Hello dev,

I have made some progress on this module, here is a brief description:

The skeleton code for this micro-service is uploaded to 
https://github.com/apache/airavata/tree/allocation-manager

As the use-cases were pretty exhaustive, I have started on developing 
the data models for this service, this documents ( 
https://drive.google.com/open?id=1Dxat8ewk7GQ8qzrmEQsueKORgw6IPlYShVVByoAOAys 
) will summarize the preliminary structure, Both GridChem and XRAS 
allocation request models were studied to develop these, Please review 
the document and feel free to comment about the choices made.

Jira Link: https://issues.apache.org/jira/browse/AIRAVATA-2320

I believe the data model will share more light on the functionalities 
and requirements for this service, and will surely help any developer to 
make contributions.


Thanks and best regards,

*Anuj Bhandar*


On 9/23/16 4:20 PM, Anuj Bhandar wrote:
>
> Hello Mark,
>
> First of all, thank you for the reply !
>
> The design you briefed me, a very good way to control the allocations, 
> the insight will definitely help me address few issue's for this service.
>
> I have derived some use-cases after discussing with Sudhakar, Are 
> these use-cases exhaustive enough to generalize them to CIPRES, Please 
> review and suggest improvements if any.
>
> Use-cases can be found here : 
> https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Resource+Allocation+Service
>
>
> Yes Suresh,
>
> I have created a page to document the development phase for this 
> module, link is mentioned above.
>
>
> Thanks and best regards,
>
> Anuj Bhandar
>
>
> On 09/23/2016 03:38 PM, Miller, Mark wrote:
>>
>> Hi Anuj,
>>
>> Just to follow up on Suresh’s comment.
>>
>> We do not use an allocation protocol, though Sudhakar does, and 
>> probably can help with that.
>>
>> We do enforce both storage and usage limits on users.
>>
>> The former is done at login, and users over a certain threshold 
>> aren’t allowed to submit jobs until their storage levels are reduced.
>>
>> The latter is enforced, but manually in our main site (at least at 
>> present).
>>
>> However, our rest site has a better and more sophisticated method.
>>
>> Our policy is to allow users to access 50,000 su if they are at a US 
>> institution, and 30,000 su if they are at a foreign institution.
>>
>> When the user hits 10,20, 30, and 50 thousand SU, they receive an 
>> email to warn them about the process.
>>
>> If they go above, I can turn off their account manually, but I 
>> usually ask them how they are doing first.
>>
>> In the REST site, each time a user submits a job, we calculate the 
>> amount of total SUs it would consume if it ran to completion.
>>
>> We subtract this amount from the amount of allocation the user has 
>> remaining. If the potential cost is greater than the amount they have 
>> remaining, the submission is rejected. That way, a user cannot submit 
>> 100 jobs each requiring 6800 SU. Which is possible, though it would 
>> take a long time.
>>
>> The idea of holding a reserve against potential expenses by a running 
>> job is not a new idea, but it prevents things from getting out of hand.
>>
>> Let me know if you have any questions about how this works.
>>
>> Best,
>>
>> Mark
>>
>> *From:*Suresh Marru [mailto:smarru@apache.org]
>> *Sent:* Friday, September 23, 2016 12:00 PM
>> *To:* Airavata Dev <dev@airavata.apache.org>
>> *Subject:* Re: Building a Resource Allocation Service For Airavata
>>
>> Hi Anuj,
>>
>> This is great summary. Since this is a new capability for Airavata, 
>> it will be useful to have dedicated wiki space to record all 
>> information. We can start with a high-level problem description (you 
>> have some of that in the form of use cases), architecture and design 
>> thoughts and Implementation plan (pointers to JIRA’s).
>>
>> May be you can put it under - 
>> https://cwiki.apache.org/confluence/display/AIRAVATA/Architecture+Documentation
>>
>> Mark and Terri might have some input on this task since CIPRES 
>> enforces soft quotas for users (but I guess there is no formal 
>> allocation request process). Once you have the wiki we can gather the 
>> feedback also there.
>>
>> Thanks,
>>
>> Suresh
>>
>>     On Sep 23, 2016, at 2:39 PM, Anuj Bhandar <bhandar.anuj@gmail.com
>>     <mailto:bhandar.anuj@gmail.com>> wrote:
>>
>>     Hi Dev,
>>
>>     I have undertaken this project of designing and developing an
>>     resource allocation service for Airavata, to give you some
>>     context, this service takes inspiration from XRAS (XSEDE Resource
>>     Allocation Service).
>>
>>     Here is the preliminary design document, which was written as a
>>     part of GSOC-2016,
>>     https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGsoc+Proposal%5D+Resource+Allocation+Manager+for+Apache+Airavata.
>>
>>     I have derived some use cases by discussing with Sudhakar
>>     Pamidighantam ( PI - GridChem), here is the link :
>>     https://issues.apache.org/jira/browse/AIRAVATA-2108.
>>
>>     Any comments and improvements are gladly welcome.
>>
>>     Thanks and best regards,
>>
>>     Anuj Bhandar
>>
>


Mime
View raw message