airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mangirish Wagle <vaglomangir...@gmail.com>
Subject Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
Date Thu, 24 Mar 2016 01:10:52 GMT
Thanks Shameera for the info and sharing the JIRA Epic details.

I have drafted my GSOC Proposal for the project and I request you to please
review the same:-

https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+Proposal-+Cloud+Based+Clusters+for+Apache+Airavata

I shall submit this on the GSOC portal by tomorrow, once I get my
enrollment verification proof.

Regards,
Mangirish



On Wed, Mar 23, 2016 at 12:29 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Mangirish,
>
> Yes your above understanding is right. Gfac is like task executor which
> execute what ever task given by Orchestrator.
>
> Here is the epic https://issues.apache.org/jira/browse/AIRAVATA-1924,
> Open stack integration is part of this epic, you can create a new top level
> jira ticket and create subtask under that ticket.
>
> Regards,
> Shameera.
>
> On Wed, Mar 23, 2016 at 12:20 PM Mangirish Wagle <vaglomangirish@gmail.com>
> wrote:
>
>> Thanks Marlon for the info. So what I get is that the Orchestrator would
>> decide if the job needs to be submitted to cloud based cluster and route it
>> to GFAC which would have a separate interfacing with the cloud cluster
>> service.
>>
>> Also I wanted to know if there is any Story/ Epic created in JIRA for
>> this project which I can use to create and track tasks? If not can I create
>> one?
>>
>> Thanks.
>>
>> Regards,
>> Mangirish
>>
>> On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>>
>>> The Application Factory component is called “gfac” in the code base.
>>> This is the part that handles the interfacing to the remote resource (most
>>> often by ssh but other providers exist). The Orchestrator routes jobs to
>>> GFAC instances.
>>>
>>> From: Mangirish Wagle <vaglomangirish@gmail.com>
>>> Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
>>> Date: Wednesday, March 23, 2016 at 11:56 AM
>>> To: "dev@airavata.apache.org" <dev@airavata.apache.org>
>>> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
>>>
>>> Hello Team,
>>>
>>> I was drafting the GSOC proposal and I just had a quick question about
>>> the integration of the project with Apache Airavata.
>>>
>>> Which is the component in Airavata that would call the service to
>>> provision the cloud cluster?
>>>
>>> I am looking at the Airavata architecture diagram and my understanding
>>> is that this would be treated as a new Application and would have a
>>> separate application interface in 'Application Factory' component. Also the
>>> workflow orchestrator would be having the intelligence to figure out which
>>> jobs to be submitted to cloud based clusters.
>>>
>>> Please let me know whether my understanding is correct.
>>>
>>> Thank you.
>>>
>>> Best Regards,
>>> Mangirish Wagle
>>>
>>> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>>>
>>>> Hi Mangirish, please add your proposal to the GSOC 2016 site.
>>>>
>>>> From: Mangirish Wagle <vaglomangirish@gmail.com>
>>>> Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
>>>> Date: Thursday, March 17, 2016 at 3:35 PM
>>>> To: "dev@airavata.apache.org" <dev@airavata.apache.org>
>>>> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata
>>>>
>>>> Hello Dev Team,
>>>>
>>>> I had the opportunity to interact with Suresh and Shameera wherein we
>>>> discussed an open requirement in Airavata to be addressed. The requirement
>>>> is to expand the capabilities of Apache Airavata to submit jobs to cloud
>>>> based clusters in addition to HPC/ HTC clusters.
>>>>
>>>> The idea is to dynamically provision a cloud cluster in an environment
>>>> like Jetstream, based on the configuration figured out by Airavata, which
>>>> would be operated by a distributed system management software like Mesos.
>>>> An initial high level goals would be:-
>>>>
>>>>    1. Airavata categorizes certain jobs to be run on cloud based
>>>>    clusters and figure out the required hardware config for the cluster.
>>>>    2. The proposed service would provision the cluster with the
>>>>    required resources.
>>>>    3. An ansible script would configure a Mesos cluster with the
>>>>    resources provisioned.
>>>>    4. Airavata submits the job to the Mesos cluster.
>>>>    5. Mesos then figures out the efficient resource allocation within
>>>>    the cluster and runs the job and fetches the result.
>>>>    6. The cluster is then deprovisioned automatically when not in use.
>>>>
>>>> The project would mainly focus on point 2 and 6 above.
>>>>
>>>> To start with, I am currently trying to get a working prototype of
>>>> setting up compute nodes on an openstack environment using JClouds
>>>> (Targetted for Jetstream). Also, I am planning to explore the option of
>>>> using Openstack Heat engine to orchestrate the cluster. However, going
>>>> ahead Airavata would be supporting other clouds like Amazon EC2 or Comet
>>>> cluster, so we need to have a generic solution for achieving the goal.
>>>>
>>>> Another approach which might be efficient in terms of performance and
>>>> time is using a container based clouds using Docker, Kubernetes which would
>>>> have substantially less bootstrap time compared to cloud VMs. This would
be
>>>> a future prospect as we may not have all the clusters supporting
>>>> containerization.
>>>>
>>>> This has been considered as a potential GSOC project and I would be
>>>> working on drafting a proposal on this idea.
>>>>
>>>> Any inputs/ comments/ suggestions would be very helpful.
>>>>
>>>> Best Regards,
>>>> Mangirish Wagle
>>>>
>>>
>>>
>> --
> Shameera Rathnayaka
>

Mime
View raw message