airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pierce, Marlon" <marpi...@iu.edu>
Subject Re: launching a job through Airavata to Mesos cluster
Date Wed, 28 Oct 2015 17:00:54 GMT
The “hello world” for us is usually to just run some linux command on the remote queuing
system. So that is something you could easily wrap via docker and submit via Mesos (I hope
it is easy…).  Related to this is determining that the job has gone through various states
(queued, executing, completed) and communicating that back to Airavata.


Marlon


From: Pankaj Saha <psaha4@binghamton.edu<mailto:psaha4@binghamton.edu>>
Date: Wednesday, October 28, 2015 at 12:54 PM
To: Marlon Pierce <marpierc@iu.edu<mailto:marpierc@iu.edu>>
Cc: Suresh Marru <smarru@apache.org<mailto:smarru@apache.org>>, Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Marlon,
"we more typically use Airavata to run scientific applications on clusters" - Our plan is
to run dockerized scientific application through Airavata.
Definitely our target is not to run a Tomcat webserver, I just said that as an example.

Give me some time, after I discuss with Prof. Madhu and Renan I will try to reply your queries
in details.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpierc@iu.edu<mailto:marpierc@iu.edu>>
wrote:
Hi Pankaj—

I may not have the full background on what you want to do, but we more typically use Airavata
to run scientific applications on clusters. This goes through a scheduling and queuing system
(Torque/MOAB or SLURM).  If you want to do this, you would first need to say how you would
submit a dockerized application using Mesos as your queueing system. You could do this for
a Tomcat server, but that is a little different from the jobs we normally run.

We don’t use Airavata to bootstrap itself, although that may be an interesting way to think
of things.  There it is interesting to use Mesos + Marathon to run Airavata services (which
may themselves by dockerized) but that would be done today outside Airavata.

Marlon


From: Suresh Marru <smarru@apache.org<mailto:smarru@apache.org>>
Date: Wednesday, October 28, 2015 at 12:25 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Cc: Marlon Pierce <marpierc@iu.edu<mailto:marpierc@iu.edu>>

Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj,

You cannot do this out of the box now, but with some guidance you can easily write a task
to run  Dockerized application on a Mesos cluster, please use 0.16 for this. As Shameera explained
this is a improvised architecture and any contributions you will make will carry forward.
Note: This will require you to understand under the hood details of Airavata Architecture
but I am sure you will motivate others to work with you to speed it up.

Suresh

On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psaha4@binghamton.edu<mailto:psaha4@binghamton.edu>>
wrote:

“launch a dockerized job”  means submitting a docker container for any specific job (say
a java webserver docker container) through Airavata.
What I remember Airavata php gateway can not submit any docker container(correct me if I am
wrong). What I want to figure out initially is whether its feasible to submit a docker container
with help of Airavata into a Mesos cluster.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpierc@iu.edu<mailto:marpierc@iu.edu>>
wrote:
I’ll add: if submitting a job to (for example) a SLURM queuing system, we need to create
the correct SLURM submission script and submit it by executing the correct command line operation
(sbatch).

From: Marlon Pierce <marpierc@iu.edu<mailto:marpierc@iu.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 12:08 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>, Suresh
Marru <smarru@apache.org<mailto:smarru@apache.org>>, Pankaj Saha <psaha4@binghamton.edu<mailto:psaha4@binghamton.edu>>

Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj, can you say more about what you mean by “launch a dockerized job”?

Marlon


From: Pankaj Saha <psaha4@binghamton.edu<mailto:psaha4@binghamton.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 11:56 AM
To: Suresh Marru <smarru@apache.org<mailto:smarru@apache.org>>
Cc: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Suresh,

My initial understanding is, I have to launch a dockerized job through Airavata which will
be run in the Mesos cluster.  I was looking for the code which submits jobs and wanted to
make changes such a way that it can submit docker containers to Mesos/Marathon cluster.

I can use 0.15 branch and I have no idea about data transfer protocol and job submission protocols
that Shameera has mentioned. I may want to submit jobs by submitting a JSON through command
line or any other way that you guys feel is more appropriate.

I can talk to Prof. Madhu and let you know more on the requirement.

Thanks
Pankaj







On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <smarru@apache.org<mailto:smarru@apache.org>>
wrote:
Pankaj can you clarify the following:

Do you want an Airavata instance to run some dockerized applications scheduled by Mesos? Or
do you just need a client which will connect to Airavata hosted and managed by Mesos/Marathon?

Suresh

On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <shameerainfo@gmail.com<mailto:shameerainfo@gmail.com>>
wrote:

Hi Pankaj,

Wich version of Airavata you are working on?  what is the data transfer protocol? What is
the job submission protocol?

Short answer:  if you are using Airavata 0.15 then you need to write new Provider implementation
to submit the request to Mesos/Marathon cluster.  But if you are using Airavata 16.0 which
is current master, then you need to write JobSubmissionTask implementation. Either case you
can go through the existing implementations, for Provider implementation see  SSHProvider
 and JobSubmissionTask implementation see SSHJobSubmissionTask.

If I get the answers to my questions then i can provide exactly what you need to do. BTW we
have cleaned our internal architecture in Airavata 16.0, as a developer you would find it
easy to work with Airavata 16.0 that Airavata 15.0. But notice master is not yet stable as
Airavata 15.0.

Regards,
Shameera.


On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psaha4@binghamton.edu<mailto:psaha4@binghamton.edu>>
wrote:
Hello Shameera,
I am working on jet-stream project, where I have to find out a way to submit a job in mesos/marathon
cluster through Airavata client. I don't have much idea from where to start looking into.
Can you please give some clue so that I can start working and making changes to java code
for the same.

Thanks
Pankaj

--
Shameera Rathnayaka





Mime
View raw message