airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mangirish Wagle <>
Subject Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
Date Thu, 24 Mar 2016 01:10:52 GMT
Thanks Shameera for the info and sharing the JIRA Epic details.

I have drafted my GSOC Proposal for the project and I request you to please
review the same:-

I shall submit this on the GSOC portal by tomorrow, once I get my
enrollment verification proof.


On Wed, Mar 23, 2016 at 12:29 PM, Shameera Rathnayaka <> wrote:

> Hi Mangirish,
> Yes your above understanding is right. Gfac is like task executor which
> execute what ever task given by Orchestrator.
> Here is the epic,
> Open stack integration is part of this epic, you can create a new top level
> jira ticket and create subtask under that ticket.
> Regards,
> Shameera.
> On Wed, Mar 23, 2016 at 12:20 PM Mangirish Wagle <>
> wrote:
>> Thanks Marlon for the info. So what I get is that the Orchestrator would
>> decide if the job needs to be submitted to cloud based cluster and route it
>> to GFAC which would have a separate interfacing with the cloud cluster
>> service.
>> Also I wanted to know if there is any Story/ Epic created in JIRA for
>> this project which I can use to create and track tasks? If not can I create
>> one?
>> Thanks.
>> Regards,
>> Mangirish
>> On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon <> wrote:
>>> The Application Factory component is called “gfac” in the code base.
>>> This is the part that handles the interfacing to the remote resource (most
>>> often by ssh but other providers exist). The Orchestrator routes jobs to
>>> GFAC instances.
>>> From: Mangirish Wagle <>
>>> Reply-To: "" <>
>>> Date: Wednesday, March 23, 2016 at 11:56 AM
>>> To: "" <>
>>> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
>>> Hello Team,
>>> I was drafting the GSOC proposal and I just had a quick question about
>>> the integration of the project with Apache Airavata.
>>> Which is the component in Airavata that would call the service to
>>> provision the cloud cluster?
>>> I am looking at the Airavata architecture diagram and my understanding
>>> is that this would be treated as a new Application and would have a
>>> separate application interface in 'Application Factory' component. Also the
>>> workflow orchestrator would be having the intelligence to figure out which
>>> jobs to be submitted to cloud based clusters.
>>> Please let me know whether my understanding is correct.
>>> Thank you.
>>> Best Regards,
>>> Mangirish Wagle
>>> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon <> wrote:
>>>> Hi Mangirish, please add your proposal to the GSOC 2016 site.
>>>> From: Mangirish Wagle <>
>>>> Reply-To: "" <>
>>>> Date: Thursday, March 17, 2016 at 3:35 PM
>>>> To: "" <>
>>>> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata
>>>> Hello Dev Team,
>>>> I had the opportunity to interact with Suresh and Shameera wherein we
>>>> discussed an open requirement in Airavata to be addressed. The requirement
>>>> is to expand the capabilities of Apache Airavata to submit jobs to cloud
>>>> based clusters in addition to HPC/ HTC clusters.
>>>> The idea is to dynamically provision a cloud cluster in an environment
>>>> like Jetstream, based on the configuration figured out by Airavata, which
>>>> would be operated by a distributed system management software like Mesos.
>>>> An initial high level goals would be:-
>>>>    1. Airavata categorizes certain jobs to be run on cloud based
>>>>    clusters and figure out the required hardware config for the cluster.
>>>>    2. The proposed service would provision the cluster with the
>>>>    required resources.
>>>>    3. An ansible script would configure a Mesos cluster with the
>>>>    resources provisioned.
>>>>    4. Airavata submits the job to the Mesos cluster.
>>>>    5. Mesos then figures out the efficient resource allocation within
>>>>    the cluster and runs the job and fetches the result.
>>>>    6. The cluster is then deprovisioned automatically when not in use.
>>>> The project would mainly focus on point 2 and 6 above.
>>>> To start with, I am currently trying to get a working prototype of
>>>> setting up compute nodes on an openstack environment using JClouds
>>>> (Targetted for Jetstream). Also, I am planning to explore the option of
>>>> using Openstack Heat engine to orchestrate the cluster. However, going
>>>> ahead Airavata would be supporting other clouds like Amazon EC2 or Comet
>>>> cluster, so we need to have a generic solution for achieving the goal.
>>>> Another approach which might be efficient in terms of performance and
>>>> time is using a container based clouds using Docker, Kubernetes which would
>>>> have substantially less bootstrap time compared to cloud VMs. This would
>>>> a future prospect as we may not have all the clusters supporting
>>>> containerization.
>>>> This has been considered as a potential GSOC project and I would be
>>>> working on drafting a proposal on this idea.
>>>> Any inputs/ comments/ suggestions would be very helpful.
>>>> Best Regards,
>>>> Mangirish Wagle
>> --
> Shameera Rathnayaka

View raw message