airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
Date Thu, 24 Mar 2016 23:18:53 GMT
Hi Mangirish,

Your proposal has all the required good detail. One optional addition you can clarify on if
you can expand or contract resources to a previously provisioned cloud. 

Suresh

> On Mar 23, 2016, at 9:10 PM, Mangirish Wagle <vaglomangirish@gmail.com> wrote:
> 
> Thanks Shameera for the info and sharing the JIRA Epic details.
> 
> I have drafted my GSOC Proposal for the project and I request you to please review the
same:-
> 
> https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+Proposal-+Cloud+Based+Clusters+for+Apache+Airavata
<https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+Proposal-+Cloud+Based+Clusters+for+Apache+Airavata>
> 
> I shall submit this on the GSOC portal by tomorrow, once I get my enrollment verification
proof.
> 
> Regards,
> Mangirish
> 
> 
> 
> On Wed, Mar 23, 2016 at 12:29 PM, Shameera Rathnayaka <shameerainfo@gmail.com <mailto:shameerainfo@gmail.com>>
wrote:
> Hi Mangirish, 
> 
> Yes your above understanding is right. Gfac is like task executor which execute what
ever task given by Orchestrator. 
> 
> Here is the epic https://issues.apache.org/jira/browse/AIRAVATA-1924 <https://issues.apache.org/jira/browse/AIRAVATA-1924>,
Open stack integration is part of this epic, you can create a new top level jira ticket and
create subtask under that ticket. 
> 
> Regards, 
> Shameera.
> 
> On Wed, Mar 23, 2016 at 12:20 PM Mangirish Wagle <vaglomangirish@gmail.com <mailto:vaglomangirish@gmail.com>>
wrote:
> Thanks Marlon for the info. So what I get is that the Orchestrator would decide if the
job needs to be submitted to cloud based cluster and route it to GFAC which would have a separate
interfacing with the cloud cluster service.
> 
> Also I wanted to know if there is any Story/ Epic created in JIRA for this project which
I can use to create and track tasks? If not can I create one?
> 
> Thanks.
> 
> Regards,
> Mangirish
> 
> On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon <marpierc@iu.edu <mailto:marpierc@iu.edu>>
wrote:
> The Application Factory component is called “gfac” in the code base.  This is the
part that handles the interfacing to the remote resource (most often by ssh but other providers
exist). The Orchestrator routes jobs to GFAC instances.
> 
> From: Mangirish Wagle <vaglomangirish@gmail.com <mailto:vaglomangirish@gmail.com>>
> Reply-To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org>" <dev@airavata.apache.org
<mailto:dev@airavata.apache.org>>
> Date: Wednesday, March 23, 2016 at 11:56 AM
> To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org>" <dev@airavata.apache.org
<mailto:dev@airavata.apache.org>>
> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
> 
> Hello Team,
> 
> I was drafting the GSOC proposal and I just had a quick question about the integration
of the project with Apache Airavata.
> 
> Which is the component in Airavata that would call the service to provision the cloud
cluster?
> 
> I am looking at the Airavata architecture diagram and my understanding is that this would
be treated as a new Application and would have a separate application interface in 'Application
Factory' component. Also the workflow orchestrator would be having the intelligence to figure
out which jobs to be submitted to cloud based clusters.
> 
> Please let me know whether my understanding is correct.
> 
> Thank you.
> 
> Best Regards,
> Mangirish Wagle
> 
> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon <marpierc@iu.edu <mailto:marpierc@iu.edu>>
wrote:
> Hi Mangirish, please add your proposal to the GSOC 2016 site.
> 
> From: Mangirish Wagle <vaglomangirish@gmail.com <mailto:vaglomangirish@gmail.com>>
> Reply-To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org>" <dev@airavata.apache.org
<mailto:dev@airavata.apache.org>>
> Date: Thursday, March 17, 2016 at 3:35 PM
> To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org>" <dev@airavata.apache.org
<mailto:dev@airavata.apache.org>>
> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata
> 
> Hello Dev Team,
> 
> I had the opportunity to interact with Suresh and Shameera wherein we discussed an open
requirement in Airavata to be addressed. The requirement is to expand the capabilities of
Apache Airavata to submit jobs to cloud based clusters in addition to HPC/ HTC clusters.
> 
> The idea is to dynamically provision a cloud cluster in an environment like Jetstream,
based on the configuration figured out by Airavata, which would be operated by a distributed
system management software like Mesos. An initial high level goals would be:-
> Airavata categorizes certain jobs to be run on cloud based clusters and figure out the
required hardware config for the cluster.
> The proposed service would provision the cluster with the required resources.
> An ansible script would configure a Mesos cluster with the resources provisioned.
> Airavata submits the job to the Mesos cluster.
> Mesos then figures out the efficient resource allocation within the cluster and runs
the job and fetches the result.
> The cluster is then deprovisioned automatically when not in use.
> The project would mainly focus on point 2 and 6 above.
> 
> To start with, I am currently trying to get a working prototype of setting up compute
nodes on an openstack environment using JClouds (Targetted for Jetstream). Also, I am planning
to explore the option of using Openstack Heat engine to orchestrate the cluster. However,
going ahead Airavata would be supporting other clouds like Amazon EC2 or Comet cluster, so
we need to have a generic solution for achieving the goal.
> 
> Another approach which might be efficient in terms of performance and time is using a
container based clouds using Docker, Kubernetes which would have substantially less bootstrap
time compared to cloud VMs. This would be a future prospect as we may not have all the clusters
supporting containerization.
> 
> This has been considered as a potential GSOC project and I would be working on drafting
a proposal on this idea.
> 
> Any inputs/ comments/ suggestions would be very helpful.
> 
> Best Regards,
> Mangirish Wagle
> 
> 
> -- 
> Shameera Rathnayaka
> 


Mime
View raw message