airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Saha <psa...@binghamton.edu>
Subject Re: launching a job through Airavata to Mesos cluster
Date Sat, 14 Nov 2015 19:06:36 GMT
Hi Suresh and Marlon
I am trying to launch a hello world sample application from my local docker
Airavata PGA. Can I get a demo admin role for few days so that I can create
an application and test some sample experiments? It looks like without
admin role I can not create applications.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <marpierc@iu.edu> wrote:

> But you could answer my implied question below with zero knowledge of
> Airavata.
>
> From: Pankaj Saha <psaha4@binghamton.edu>
> Reply-To: Airavata Dev <dev@airavata.apache.org>
> Date: Wednesday, October 28, 2015 at 1:48 PM
>
> To: Marlon Pierce <marpierc@iu.edu>
> Cc: Suresh Marru <smarru@apache.org>, Airavata Dev <
> dev@airavata.apache.org>
> Subject: Re: launching a job through Airavata to Mesos cluster
>
> As Suresh said I think this is the time  when I have to understand under
> the hood details of Airavata Architecture, which I could not gain much
> during GSoC.
>
> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>
>> The “hello world” for us is usually to just run some linux command on the
>> remote queuing system. So that is something you could easily wrap via
>> docker and submit via Mesos (I hope it is easy…).  Related to this is
>> determining that the job has gone through various states (queued,
>> executing, completed) and communicating that back to Airavata.
>>
>>
>> Marlon
>>
>>
>> From: Pankaj Saha <psaha4@binghamton.edu>
>> Date: Wednesday, October 28, 2015 at 12:54 PM
>> To: Marlon Pierce <marpierc@iu.edu>
>> Cc: Suresh Marru <smarru@apache.org>, Airavata Dev <
>> dev@airavata.apache.org>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Marlon,
>> "we more typically use Airavata to run scientific applications on
>> clusters" - Our plan is to run dockerized scientific application through
>> Airavata.
>> Definitely our target is not to run a Tomcat webserver, I just said that
>> as an example.
>>
>> Give me some time, after I discuss with Prof. Madhu and Renan I will try
>> to reply your queries in details.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>>
>>> Hi Pankaj—
>>>
>>> I may not have the full background on what you want to do, but we more
>>> typically use Airavata to run scientific applications on clusters. This
>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
>>> you want to do this, you would first need to say how you would submit a
>>> dockerized application using Mesos as your queueing system. You could do
>>> this for a Tomcat server, but that is a little different from the jobs we
>>> normally run.
>>>
>>> We don’t use Airavata to bootstrap itself, although that may be an
>>> interesting way to think of things.  There it is interesting to use Mesos +
>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>> that would be done today outside Airavata.
>>>
>>> Marlon
>>>
>>>
>>> From: Suresh Marru <smarru@apache.org>
>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>> To: Airavata Dev <dev@airavata.apache.org>
>>> Cc: Marlon Pierce <marpierc@iu.edu>
>>>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Pankaj,
>>>
>>> You cannot do this out of the box now, but with some guidance you can
>>> easily write a task to run  Dockerized application on a Mesos cluster,
>>> please use 0.16 for this. As Shameera explained this is a improvised
>>> architecture and any contributions you will make will carry forward. Note:
>>> This will require you to understand under the hood details of Airavata
>>> Architecture but I am sure you will motivate others to work with you to
>>> speed it up.
>>>
>>> Suresh
>>>
>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psaha4@binghamton.edu> wrote:
>>>
>>> “launch a dockerized job”  means submitting a docker container for any
>>> specific job (say a java webserver docker container) through Airavata.
>>> What I remember Airavata php gateway can not submit any docker
>>> container(correct me if I am wrong). What I want to figure out initially is
>>> whether its feasible to submit a docker container with help of Airavata
>>> into a Mesos cluster.
>>>
>>> Thanks
>>> Pankaj
>>>
>>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpierc@iu.edu>
>>> wrote:
>>>
>>>> I’ll add: if submitting a job to (for example) a SLURM queuing system,
>>>> we need to create the correct SLURM submission script and submit it by
>>>> executing the correct command line operation (sbatch).
>>>>
>>>> From: Marlon Pierce <marpierc@iu.edu>
>>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru <
>>>> smarru@apache.org>, Pankaj Saha <psaha4@binghamton.edu>
>>>>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Pankaj, can you say more about what you mean by “launch a dockerized
>>>> job”?
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>>> To: Suresh Marru <smarru@apache.org>
>>>> Cc: Airavata Dev <dev@airavata.apache.org>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Suresh,
>>>>
>>>> My initial understanding is, I have to launch a dockerized job through
>>>> Airavata which will be run in the Mesos cluster.  I was looking for the
>>>> code which submits jobs and wanted to make changes such a way that it can
>>>> submit docker containers to Mesos/Marathon cluster.
>>>>
>>>> I can use 0.15 branch and I have no idea about data transfer protocol
>>>> and job submission protocols that Shameera has mentioned. I may want to
>>>> submit jobs by submitting a JSON through command line or any other way that
>>>> you guys feel is more appropriate.
>>>>
>>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <smarru@apache.org>
>>>> wrote:
>>>>
>>>>> Pankaj can you clarify the following:
>>>>>
>>>>> Do you want an Airavata instance to run some dockerized applications
>>>>> scheduled by Mesos? Or do you just need a client which will connect to
>>>>> Airavata hosted and managed by Mesos/Marathon?
>>>>>
>>>>> Suresh
>>>>>
>>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>>> shameerainfo@gmail.com> wrote:
>>>>>
>>>>> Hi Pankaj,
>>>>>
>>>>> Wich version of Airavata you are working on?  what is the data
>>>>> transfer protocol? What is the job submission protocol?
>>>>>
>>>>> Short answer:  if you are using Airavata 0.15 then you need to write
>>>>> new Provider implementation to submit the request to Mesos/Marathon
>>>>> cluster.  But if you are using Airavata 16.0 which is current master,
then
>>>>> you need to write JobSubmissionTask implementation. Either case you can
go
>>>>> through the existing implementations, for Provider implementation see
>>>>>  SSHProvider  and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>>
>>>>> If I get the answers to my questions then i can provide exactly what
>>>>> you need to do. BTW we have cleaned our internal architecture in Airavata
>>>>> 16.0, as a developer you would find it easy to work with Airavata 16.0
that
>>>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>>>>>
>>>>> Regards,
>>>>> Shameera.
>>>>>
>>>>>
>>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psaha4@binghamton.edu>
>>>>> wrote:
>>>>>
>>>>>> Hello Shameera,
>>>>>> I am working on jet-stream project, where I have to find out a way
to
>>>>>> submit a job in mesos/marathon cluster through Airavata client. I
don't
>>>>>> have much idea from where to start looking into. Can you please give
some
>>>>>> clue so that I can start working and making changes to java code
for the
>>>>>> same.
>>>>>>
>>>>>> Thanks
>>>>>> Pankaj
>>>>>>
>>>>>> --
>>>>> Shameera Rathnayaka
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message