airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Saha <psa...@binghamton.edu>
Subject Re: launching a job through Airavata to Mesos cluster
Date Wed, 28 Oct 2015 16:54:59 GMT
Hi Marlon,
"we more typically use Airavata to run scientific applications on clusters"
- Our plan is to run dockerized scientific application through Airavata.
Definitely our target is not to run a Tomcat webserver, I just said that as
an example.

Give me some time, after I discuss with Prof. Madhu and Renan I will try to
reply your queries in details.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpierc@iu.edu> wrote:

> Hi Pankaj—
>
> I may not have the full background on what you want to do, but we more
> typically use Airavata to run scientific applications on clusters. This
> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
> you want to do this, you would first need to say how you would submit a
> dockerized application using Mesos as your queueing system. You could do
> this for a Tomcat server, but that is a little different from the jobs we
> normally run.
>
> We don’t use Airavata to bootstrap itself, although that may be an
> interesting way to think of things.  There it is interesting to use Mesos +
> Marathon to run Airavata services (which may themselves by dockerized) but
> that would be done today outside Airavata.
>
> Marlon
>
>
> From: Suresh Marru <smarru@apache.org>
> Date: Wednesday, October 28, 2015 at 12:25 PM
> To: Airavata Dev <dev@airavata.apache.org>
> Cc: Marlon Pierce <marpierc@iu.edu>
>
> Subject: Re: launching a job through Airavata to Mesos cluster
>
> Hi Pankaj,
>
> You cannot do this out of the box now, but with some guidance you can
> easily write a task to run  Dockerized application on a Mesos cluster,
> please use 0.16 for this. As Shameera explained this is a improvised
> architecture and any contributions you will make will carry forward. Note:
> This will require you to understand under the hood details of Airavata
> Architecture but I am sure you will motivate others to work with you to
> speed it up.
>
> Suresh
>
> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psaha4@binghamton.edu> wrote:
>
> “launch a dockerized job”  means submitting a docker container for any
> specific job (say a java webserver docker container) through Airavata.
> What I remember Airavata php gateway can not submit any docker
> container(correct me if I am wrong). What I want to figure out initially is
> whether its feasible to submit a docker container with help of Airavata
> into a Mesos cluster.
>
> Thanks
> Pankaj
>
> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>
>> I’ll add: if submitting a job to (for example) a SLURM queuing system, we
>> need to create the correct SLURM submission script and submit it by
>> executing the correct command line operation (sbatch).
>>
>> From: Marlon Pierce <marpierc@iu.edu>
>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 12:08 PM
>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru <
>> smarru@apache.org>, Pankaj Saha <psaha4@binghamton.edu>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Pankaj, can you say more about what you mean by “launch a dockerized
>> job”?
>>
>> Marlon
>>
>>
>> From: Pankaj Saha <psaha4@binghamton.edu>
>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 11:56 AM
>> To: Suresh Marru <smarru@apache.org>
>> Cc: Airavata Dev <dev@airavata.apache.org>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Suresh,
>>
>> My initial understanding is, I have to launch a dockerized job through
>> Airavata which will be run in the Mesos cluster.  I was looking for the
>> code which submits jobs and wanted to make changes such a way that it can
>> submit docker containers to Mesos/Marathon cluster.
>>
>> I can use 0.15 branch and I have no idea about data transfer protocol and
>> job submission protocols that Shameera has mentioned. I may want to submit
>> jobs by submitting a JSON through command line or any other way that you
>> guys feel is more appropriate.
>>
>> I can talk to Prof. Madhu and let you know more on the requirement.
>>
>> Thanks
>> Pankaj
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <smarru@apache.org> wrote:
>>
>>> Pankaj can you clarify the following:
>>>
>>> Do you want an Airavata instance to run some dockerized applications
>>> scheduled by Mesos? Or do you just need a client which will connect to
>>> Airavata hosted and managed by Mesos/Marathon?
>>>
>>> Suresh
>>>
>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>> Hi Pankaj,
>>>
>>> Wich version of Airavata you are working on?  what is the data transfer
>>> protocol? What is the job submission protocol?
>>>
>>> Short answer:  if you are using Airavata 0.15 then you need to write new
>>> Provider implementation to submit the request to Mesos/Marathon cluster.
>>> But if you are using Airavata 16.0 which is current master, then you need
>>> to write JobSubmissionTask implementation. Either case you can go through
>>> the existing implementations, for Provider implementation see  SSHProvider
>>>  and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>
>>> If I get the answers to my questions then i can provide exactly what you
>>> need to do. BTW we have cleaned our internal architecture in Airavata 16.0,
>>> as a developer you would find it easy to work with Airavata 16.0 that
>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>>>
>>> Regards,
>>> Shameera.
>>>
>>>
>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psaha4@binghamton.edu>
>>> wrote:
>>>
>>>> Hello Shameera,
>>>> I am working on jet-stream project, where I have to find out a way to
>>>> submit a job in mesos/marathon cluster through Airavata client. I don't
>>>> have much idea from where to start looking into. Can you please give some
>>>> clue so that I can start working and making changes to java code for the
>>>> same.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>> --
>>> Shameera Rathnayaka
>>>
>>>
>>>
>>
>
>

Mime
View raw message