airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Rattihalli <gratt...@binghamton.edu>
Subject Re: Design of Mesos/Aurora integration with Airavata on Jetstream
Date Wed, 27 Apr 2016 18:57:33 GMT
Hi Suresh,

First of all I would like to commend you for starting this discussion. This
> is exactly kind of the mailing list engagement we would like to see as part
> of your GSOC. You already drew Emre’s attention, often discussions done in
> isolation undermine this impact. So good job on a start and we will expect
> over the summer you will engage lot more.
>
>
Thanks. Yes, I plan to actively use this mailing list to keep my design
focused on real use cases, and also to get regular feedback.


> Application dependency libraries, compiler versions can quickly get
> tangled. You can choose to deploy Lmod (https://github.com/TACC/Lmod)
> within a VM to alleviate this but that will be an over kill. Eventually
> containers will hopefully alleviate this problem.
>
> The worker VMs need to have the Mesos libraries installed and the Mesos
> master needs to be configured with the IPs of the worker VMs. So, if new
> VMs are created each time, we will develop the playbook for the creation of
> the Mesos cluster on the fly, and then launch the application on it.
>
>
> Can the slave not register itself dynamically and report its IP Address?
> With either multiple applications per VM or one per VM, you will need to
> look into the capability of a persistent master and short lived elastic
> slave instances right? May be I am mis-understating your questions. Can you
> help clarify this more?
>
>

Yes, this is possible, and was the first of the two options I had emailed.
To support this kind of elasticity, we will work on developing a
script/playbook to install the Mesos libraries on the new slaves/agents and
connect them with the master.

Thanks,
Gourav

On Apr 27, 2016, at 1:21 PM, Emre Brookes <emre@biochem.uthscsa.edu> wrote:

Pierce, Marlon wrote:

Thanks, Gourav, this is great. How will the design differ in the two cases?

Since installing the applications is not a simple task and there aren’t
good docker libraries (for example) for many of them, I would assume that
we would have a collection of VMs that have the codes already installed.

FWIW, that model would work well with GenApp usage also.  Although, I can
imagine a scenario where a part of job initialization script (running under
the already "code installed" VM) checks for updates to the code and
installs them, but that can be independent of Airavata/Aurora/Mesos's
knowledge.

-Emre


Marlon


From: Gourav Rattihalli <grattih1@binghamton.edu <
mailto:grattih1@binghamton.edu <grattih1@binghamton.edu>>>
Reply-To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org
<dev@airavata.apache.org>>" <dev@airavata.apache.org <
mailto:dev@airavata.apache.org <dev@airavata.apache.org>>>
Date: Wednesday, April 27, 2016 at 1:01 PM
To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org
<dev@airavata.apache.org>>" <dev@airavata.apache.org <
mailto:dev@airavata.apache.org <dev@airavata.apache.org>>>
Subject: Design of Mesos/Aurora integration with Airavata on Jetstream

Hi Dev's,

I have been working on the integration of Apache Aurora and Mesos with
Airavata and Jetstream. Using Mangirish's latest code, VMs can now be
created using on Jetstream. I want to understand how Airavata is likely to
use the VMs for applications. Will airavata re-use existing VMs for
successive applications for a given community, or will new VMs be created
for each application? This will decide how we design the automated creation
of a Mesos cluster using the new VMs that are created. I understand that we
will have a Mesos master VM that will run all the time on Jetstream.

Here is the Job submission flow that I am assuming:

Airavata Client--> Airavata
Server-->(Orchestrator-->GFAC)-->My-Apache-Aurora-module-->(A script will
create .aurora configuration file that will be used to launch the
job)-->Aurora/Mesos-->VMs

So, I will design a module that will work with Orchestrator-->GFAC to
generate the appropriate Apache Aurora script to be submitted to the Aurora
master.

Please let me know if there are any comments, suggestions on this plan.

-- 
Regards,
Gourav Rattihalli






-- 
Regards,
Gourav Rattihalli

Mime
View raw message