airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mangirish Wagle <>
Subject Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
Date Wed, 23 Mar 2016 16:20:13 GMT
Thanks Marlon for the info. So what I get is that the Orchestrator would
decide if the job needs to be submitted to cloud based cluster and route it
to GFAC which would have a separate interfacing with the cloud cluster

Also I wanted to know if there is any Story/ Epic created in JIRA for this
project which I can use to create and track tasks? If not can I create one?



On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon <> wrote:

> The Application Factory component is called “gfac” in the code base.  This
> is the part that handles the interfacing to the remote resource (most often
> by ssh but other providers exist). The Orchestrator routes jobs to GFAC
> instances.
> From: Mangirish Wagle <>
> Reply-To: "" <>
> Date: Wednesday, March 23, 2016 at 11:56 AM
> To: "" <>
> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
> Hello Team,
> I was drafting the GSOC proposal and I just had a quick question about the
> integration of the project with Apache Airavata.
> Which is the component in Airavata that would call the service to
> provision the cloud cluster?
> I am looking at the Airavata architecture diagram and my understanding is
> that this would be treated as a new Application and would have a separate
> application interface in 'Application Factory' component. Also the workflow
> orchestrator would be having the intelligence to figure out which jobs to
> be submitted to cloud based clusters.
> Please let me know whether my understanding is correct.
> Thank you.
> Best Regards,
> Mangirish Wagle
> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon <> wrote:
>> Hi Mangirish, please add your proposal to the GSOC 2016 site.
>> From: Mangirish Wagle <>
>> Reply-To: "" <>
>> Date: Thursday, March 17, 2016 at 3:35 PM
>> To: "" <>
>> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata
>> Hello Dev Team,
>> I had the opportunity to interact with Suresh and Shameera wherein we
>> discussed an open requirement in Airavata to be addressed. The requirement
>> is to expand the capabilities of Apache Airavata to submit jobs to cloud
>> based clusters in addition to HPC/ HTC clusters.
>> The idea is to dynamically provision a cloud cluster in an environment
>> like Jetstream, based on the configuration figured out by Airavata, which
>> would be operated by a distributed system management software like Mesos.
>> An initial high level goals would be:-
>>    1. Airavata categorizes certain jobs to be run on cloud based
>>    clusters and figure out the required hardware config for the cluster.
>>    2. The proposed service would provision the cluster with the required
>>    resources.
>>    3. An ansible script would configure a Mesos cluster with the
>>    resources provisioned.
>>    4. Airavata submits the job to the Mesos cluster.
>>    5. Mesos then figures out the efficient resource allocation within
>>    the cluster and runs the job and fetches the result.
>>    6. The cluster is then deprovisioned automatically when not in use.
>> The project would mainly focus on point 2 and 6 above.
>> To start with, I am currently trying to get a working prototype of
>> setting up compute nodes on an openstack environment using JClouds
>> (Targetted for Jetstream). Also, I am planning to explore the option of
>> using Openstack Heat engine to orchestrate the cluster. However, going
>> ahead Airavata would be supporting other clouds like Amazon EC2 or Comet
>> cluster, so we need to have a generic solution for achieving the goal.
>> Another approach which might be efficient in terms of performance and
>> time is using a container based clouds using Docker, Kubernetes which would
>> have substantially less bootstrap time compared to cloud VMs. This would be
>> a future prospect as we may not have all the clusters supporting
>> containerization.
>> This has been considered as a potential GSOC project and I would be
>> working on drafting a proposal on this idea.
>> Any inputs/ comments/ suggestions would be very helpful.
>> Best Regards,
>> Mangirish Wagle

View raw message