airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: launching a job through Airavata to Mesos cluster
Date Wed, 28 Oct 2015 16:25:37 GMT
Hi Pankaj,

You cannot do this out of the box now, but with some guidance you can easily write a task
to run  Dockerized application on a Mesos cluster, please use 0.16 for this. As Shameera explained
this is a improvised architecture and any contributions you will make will carry forward.
Note: This will require you to understand under the hood details of Airavata Architecture
but I am sure you will motivate others to work with you to speed it up. 

Suresh

> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psaha4@binghamton.edu> wrote:
> 
> “launch a dockerized job”  means submitting a docker container for any specific job
(say a java webserver docker container) through Airavata. 
> What I remember Airavata php gateway can not submit any docker container(correct me if
I am wrong). What I want to figure out initially is whether its feasible to submit a docker
container with help of Airavata into a Mesos cluster.
> 
> Thanks
> Pankaj
> 
> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpierc@iu.edu <mailto:marpierc@iu.edu>>
wrote:
> I’ll add: if submitting a job to (for example) a SLURM queuing system, we need to create
the correct SLURM submission script and submit it by executing the correct command line operation
(sbatch).
> 
> From: Marlon Pierce <marpierc@iu.edu <mailto:marpierc@iu.edu>>
> Reply-To: Airavata Dev <dev@airavata.apache.org <mailto:dev@airavata.apache.org>>
> Date: Wednesday, October 28, 2015 at 12:08 PM
> To: Airavata Dev <dev@airavata.apache.org <mailto:dev@airavata.apache.org>>,
Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>, Pankaj Saha <psaha4@binghamton.edu
<mailto:psaha4@binghamton.edu>>
> 
> Subject: Re: launching a job through Airavata to Mesos cluster
> 
> Hi Pankaj, can you say more about what you mean by “launch a dockerized job”?   
> 
> Marlon
> 
> 
> From: Pankaj Saha <psaha4@binghamton.edu <mailto:psaha4@binghamton.edu>>
> Reply-To: Airavata Dev <dev@airavata.apache.org <mailto:dev@airavata.apache.org>>
> Date: Wednesday, October 28, 2015 at 11:56 AM
> To: Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>
> Cc: Airavata Dev <dev@airavata.apache.org <mailto:dev@airavata.apache.org>>
> Subject: Re: launching a job through Airavata to Mesos cluster
> 
> Hi Suresh,
> 
> My initial understanding is, I have to launch a dockerized job through Airavata which
will be run in the Mesos cluster.  I was looking for the code which submits jobs and wanted
to make changes such a way that it can submit docker containers to Mesos/Marathon cluster.
> 
> I can use 0.15 branch and I have no idea about data transfer protocol and job submission
protocols that Shameera has mentioned. I may want to submit jobs by submitting a JSON through
command line or any other way that you guys feel is more appropriate.
> 
> I can talk to Prof. Madhu and let you know more on the requirement.
> 
> Thanks
> Pankaj
> 
> 
> 
> 
> 
> 
> 
> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <smarru@apache.org <mailto:smarru@apache.org>>
wrote:
> Pankaj can you clarify the following:
> 
> Do you want an Airavata instance to run some dockerized applications scheduled by Mesos?
Or do you just need a client which will connect to Airavata hosted and managed by Mesos/Marathon?

> 
> Suresh
> 
>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <shameerainfo@gmail.com <mailto:shameerainfo@gmail.com>>
wrote:
>> 
>> Hi Pankaj, 
>> 
>> Wich version of Airavata you are working on?  what is the data transfer protocol?
What is the job submission protocol? 
>> 
>> Short answer:  if you are using Airavata 0.15 then you need to write new Provider
implementation to submit the request to Mesos/Marathon cluster.  But if you are using Airavata
16.0 which is current master, then you need to write JobSubmissionTask implementation. Either
case you can go through the existing implementations, for Provider implementation see  SSHProvider
 and JobSubmissionTask implementation see SSHJobSubmissionTask.
>> 
>> If I get the answers to my questions then i can provide exactly what you need to
do. BTW we have cleaned our internal architecture in Airavata 16.0, as a developer you would
find it easy to work with Airavata 16.0 that Airavata 15.0. But notice master is not yet stable
as Airavata 15.0.
>> 
>> Regards,
>> Shameera.
>>  
>> 
>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psaha4@binghamton.edu <mailto:psaha4@binghamton.edu>>
wrote:
>> Hello Shameera,
>> I am working on jet-stream project, where I have to find out a way to submit a job
in mesos/marathon cluster through Airavata client. I don't have much idea from where to start
looking into. Can you please give some clue so that I can start working and making changes
to java code for the same.
>> 
>> Thanks
>> Pankaj
>> 
>> -- 
>> Shameera Rathnayaka
> 
> 
> 


Mime
View raw message