airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Saha <psa...@binghamton.edu>
Subject Re: launching a job through Airavata to Mesos cluster
Date Wed, 28 Oct 2015 17:48:21 GMT
As Suresh said I think this is the time  when I have to understand under
the hood details of Airavata Architecture, which I could not gain much
during GSoC.

On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpierc@iu.edu> wrote:

> The “hello world” for us is usually to just run some linux command on the
> remote queuing system. So that is something you could easily wrap via
> docker and submit via Mesos (I hope it is easy…).  Related to this is
> determining that the job has gone through various states (queued,
> executing, completed) and communicating that back to Airavata.
>
>
> Marlon
>
>
> From: Pankaj Saha <psaha4@binghamton.edu>
> Date: Wednesday, October 28, 2015 at 12:54 PM
> To: Marlon Pierce <marpierc@iu.edu>
> Cc: Suresh Marru <smarru@apache.org>, Airavata Dev <
> dev@airavata.apache.org>
>
> Subject: Re: launching a job through Airavata to Mesos cluster
>
> Hi Marlon,
> "we more typically use Airavata to run scientific applications on
> clusters" - Our plan is to run dockerized scientific application through A
> iravata.
> Definitely our target is not to run a Tomcat webserver, I just said that
> as an example.
>
> Give me some time, after I discuss with Prof. Madhu and Renan I will try
> to reply your queries in details.
>
> Thanks
> Pankaj
>
> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>
>> Hi Pankaj—
>>
>> I may not have the full background on what you want to do, but we more
>> typically use Airavata to run scientific applications on clusters. This
>> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
>> you want to do this, you would first need to say how you would submit a
>> dockerized application using Mesos as your queueing system. You could do
>> this for a Tomcat server, but that is a little different from the jobs we
>> normally run.
>>
>> We don’t use Airavata to bootstrap itself, although that may be an
>> interesting way to think of things.  There it is interesting to use Mesos +
>> Marathon to run Airavata services (which may themselves by dockerized) but
>> that would be done today outside Airavata.
>>
>> Marlon
>>
>>
>> From: Suresh Marru <smarru@apache.org>
>> Date: Wednesday, October 28, 2015 at 12:25 PM
>> To: Airavata Dev <dev@airavata.apache.org>
>> Cc: Marlon Pierce <marpierc@iu.edu>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Pankaj,
>>
>> You cannot do this out of the box now, but with some guidance you can
>> easily write a task to run  Dockerized application on a Mesos cluster,
>> please use 0.16 for this. As Shameera explained this is a improvised
>> architecture and any contributions you will make will carry forward. Note:
>> This will require you to understand under the hood details of Airavata
>> Architecture but I am sure you will motivate others to work with you to
>> speed it up.
>>
>> Suresh
>>
>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psaha4@binghamton.edu> wrote:
>>
>> “launch a dockerized job”  means submitting a docker container for any
>> specific job (say a java webserver docker container) through Airavata.
>> What I remember Airavata php gateway can not submit any docker
>> container(correct me if I am wrong). What I want to figure out initially is
>> whether its feasible to submit a docker container with help of Airavata
>> into a Mesos cluster.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>>
>>> I’ll add: if submitting a job to (for example) a SLURM queuing system,
>>> we need to create the correct SLURM submission script and submit it by
>>> executing the correct command line operation (sbatch).
>>>
>>> From: Marlon Pierce <marpierc@iu.edu>
>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru <
>>> smarru@apache.org>, Pankaj Saha <psaha4@binghamton.edu>
>>>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Pankaj, can you say more about what you mean by “launch a dockerized
>>> job”?
>>>
>>> Marlon
>>>
>>>
>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>> To: Suresh Marru <smarru@apache.org>
>>> Cc: Airavata Dev <dev@airavata.apache.org>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Suresh,
>>>
>>> My initial understanding is, I have to launch a dockerized job through
>>> Airavata which will be run in the Mesos cluster.  I was looking for the
>>> code which submits jobs and wanted to make changes such a way that it can
>>> submit docker containers to Mesos/Marathon cluster.
>>>
>>> I can use 0.15 branch and I have no idea about data transfer protocol
>>> and job submission protocols that Shameera has mentioned. I may want to
>>> submit jobs by submitting a JSON through command line or any other way that
>>> you guys feel is more appropriate.
>>>
>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>
>>> Thanks
>>> Pankaj
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <smarru@apache.org>
>>> wrote:
>>>
>>>> Pankaj can you clarify the following:
>>>>
>>>> Do you want an Airavata instance to run some dockerized applications
>>>> scheduled by Mesos? Or do you just need a client which will connect to
>>>> Airavata hosted and managed by Mesos/Marathon?
>>>>
>>>> Suresh
>>>>
>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>> shameerainfo@gmail.com> wrote:
>>>>
>>>> Hi Pankaj,
>>>>
>>>> Wich version of Airavata you are working on?  what is the data transfer
>>>> protocol? What is the job submission protocol?
>>>>
>>>> Short answer:  if you are using Airavata 0.15 then you need to write
>>>> new Provider implementation to submit the request to Mesos/Marathon
>>>> cluster.  But if you are using Airavata 16.0 which is current master, then
>>>> you need to write JobSubmissionTask implementation. Either case you can go
>>>> through the existing implementations, for Provider implementation see
>>>>  SSHProvider  and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>
>>>> If I get the answers to my questions then i can provide exactly what
>>>> you need to do. BTW we have cleaned our internal architecture in Airavata
>>>> 16.0, as a developer you would find it easy to work with Airavata 16.0 that
>>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>>>>
>>>> Regards,
>>>> Shameera.
>>>>
>>>>
>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psaha4@binghamton.edu>
>>>> wrote:
>>>>
>>>>> Hello Shameera,
>>>>> I am working on jet-stream project, where I have to find out a way to
>>>>> submit a job in mesos/marathon cluster through Airavata client. I don't
>>>>> have much idea from where to start looking into. Can you please give
some
>>>>> clue so that I can start working and making changes to java code for
the
>>>>> same.
>>>>>
>>>>> Thanks
>>>>> Pankaj
>>>>>
>>>>> --
>>>> Shameera Rathnayaka
>>>>
>>>>
>>>>
>>>
>>
>>
>

Mime
View raw message