airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Saha <psa...@binghamton.edu>
Subject Re: launching a job through Airavata to Mesos cluster
Date Sat, 14 Nov 2015 20:05:22 GMT
Hi Supun,
I need a small clarification here. I am using local docker for PGA and
servers but I am using the sam credential service that you guys are using.
So do you think making local changes in the pga_config will do the work?

Thanks
Pankaj

On Sat, Nov 14, 2015 at 3:01 PM, Supun Nakandala <supun.nakandala@gmail.com>
wrote:

> Hi Pankaj,
>
> You can set to admin role to Internal/everyone in pga_config.php (in
> <PGA_ROOT>/app/config/pga_config.php) and you will admin permissions.
>
> On Sat, Nov 14, 2015 at 2:06 PM, Pankaj Saha <psaha4@binghamton.edu>
> wrote:
>
>> Hi Suresh and Marlon
>> I am trying to launch a hello world sample application from my local
>> docker Airavata PGA. Can I get a demo admin role for few days so that I can
>> create an application and test some sample experiments? It looks like
>> without admin role I can not create applications.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>>
>>> But you could answer my implied question below with zero knowledge of
>>> Airavata.
>>>
>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>> Date: Wednesday, October 28, 2015 at 1:48 PM
>>>
>>> To: Marlon Pierce <marpierc@iu.edu>
>>> Cc: Suresh Marru <smarru@apache.org>, Airavata Dev <
>>> dev@airavata.apache.org>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> As Suresh said I think this is the time  when I have to understand
>>> under the hood details of Airavata Architecture, which I could not gain
>>> much during GSoC.
>>>
>>> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>>>
>>>> The “hello world” for us is usually to just run some linux command on
>>>> the remote queuing system. So that is something you could easily wrap via
>>>> docker and submit via Mesos (I hope it is easy…).  Related to this is
>>>> determining that the job has gone through various states (queued,
>>>> executing, completed) and communicating that back to Airavata.
>>>>
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>>> Date: Wednesday, October 28, 2015 at 12:54 PM
>>>> To: Marlon Pierce <marpierc@iu.edu>
>>>> Cc: Suresh Marru <smarru@apache.org>, Airavata Dev <
>>>> dev@airavata.apache.org>
>>>>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Marlon,
>>>> "we more typically use Airavata to run scientific applications on
>>>> clusters" - Our plan is to run dockerized scientific application
>>>> through Airavata.
>>>> Definitely our target is not to run a Tomcat webserver, I just said
>>>> that as an example.
>>>>
>>>> Give me some time, after I discuss with Prof. Madhu and Renan I will
>>>> try to reply your queries in details.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpierc@iu.edu>
>>>> wrote:
>>>>
>>>>> Hi Pankaj—
>>>>>
>>>>> I may not have the full background on what you want to do, but we more
>>>>> typically use Airavata to run scientific applications on clusters. This
>>>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM).
 If
>>>>> you want to do this, you would first need to say how you would submit
a
>>>>> dockerized application using Mesos as your queueing system. You could
do
>>>>> this for a Tomcat server, but that is a little different from the jobs
we
>>>>> normally run.
>>>>>
>>>>> We don’t use Airavata to bootstrap itself, although that may be an
>>>>> interesting way to think of things.  There it is interesting to use Mesos
+
>>>>> Marathon to run Airavata services (which may themselves by dockerized)
but
>>>>> that would be done today outside Airavata.
>>>>>
>>>>> Marlon
>>>>>
>>>>>
>>>>> From: Suresh Marru <smarru@apache.org>
>>>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>>>> To: Airavata Dev <dev@airavata.apache.org>
>>>>> Cc: Marlon Pierce <marpierc@iu.edu>
>>>>>
>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>
>>>>> Hi Pankaj,
>>>>>
>>>>> You cannot do this out of the box now, but with some guidance you can
>>>>> easily write a task to run  Dockerized application on a Mesos cluster,
>>>>> please use 0.16 for this. As Shameera explained this is a improvised
>>>>> architecture and any contributions you will make will carry forward.
Note:
>>>>> This will require you to understand under the hood details of Airavata
>>>>> Architecture but I am sure you will motivate others to work with you
to
>>>>> speed it up.
>>>>>
>>>>> Suresh
>>>>>
>>>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psaha4@binghamton.edu>
>>>>> wrote:
>>>>>
>>>>> “launch a dockerized job”  means submitting a docker container for
any
>>>>> specific job (say a java webserver docker container) through Airavata.
>>>>> What I remember Airavata php gateway can not submit any docker
>>>>> container(correct me if I am wrong). What I want to figure out initially
is
>>>>> whether its feasible to submit a docker container with help of Airavata
>>>>> into a Mesos cluster.
>>>>>
>>>>> Thanks
>>>>> Pankaj
>>>>>
>>>>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpierc@iu.edu>
>>>>> wrote:
>>>>>
>>>>>> I’ll add: if submitting a job to (for example) a SLURM queuing
>>>>>> system, we need to create the correct SLURM submission script and
submit it
>>>>>> by executing the correct command line operation (sbatch).
>>>>>>
>>>>>> From: Marlon Pierce <marpierc@iu.edu>
>>>>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>>>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>>>>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru <
>>>>>> smarru@apache.org>, Pankaj Saha <psaha4@binghamton.edu>
>>>>>>
>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>
>>>>>> Hi Pankaj, can you say more about what you mean by “launch a
>>>>>> dockerized job”?
>>>>>>
>>>>>> Marlon
>>>>>>
>>>>>>
>>>>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>>>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>>>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>>>>> To: Suresh Marru <smarru@apache.org>
>>>>>> Cc: Airavata Dev <dev@airavata.apache.org>
>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>
>>>>>> Hi Suresh,
>>>>>>
>>>>>> My initial understanding is, I have to launch a dockerized job
>>>>>> through Airavata which will be run in the Mesos cluster.  I was looking
for
>>>>>> the code which submits jobs and wanted to make changes such a way
that it
>>>>>> can submit docker containers to Mesos/Marathon cluster.
>>>>>>
>>>>>> I can use 0.15 branch and I have no idea about data transfer protocol
>>>>>> and job submission protocols that Shameera has mentioned. I may want
to
>>>>>> submit jobs by submitting a JSON through command line or any other
way that
>>>>>> you guys feel is more appropriate.
>>>>>>
>>>>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>>>>
>>>>>> Thanks
>>>>>> Pankaj
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <smarru@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Pankaj can you clarify the following:
>>>>>>>
>>>>>>> Do you want an Airavata instance to run some dockerized applications
>>>>>>> scheduled by Mesos? Or do you just need a client which will connect
to
>>>>>>> Airavata hosted and managed by Mesos/Marathon?
>>>>>>>
>>>>>>> Suresh
>>>>>>>
>>>>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Pankaj,
>>>>>>>
>>>>>>> Wich version of Airavata you are working on?  what is the data
>>>>>>> transfer protocol? What is the job submission protocol?
>>>>>>>
>>>>>>> Short answer:  if you are using Airavata 0.15 then you need to
write
>>>>>>> new Provider implementation to submit the request to Mesos/Marathon
>>>>>>> cluster.  But if you are using Airavata 16.0 which is current
master, then
>>>>>>> you need to write JobSubmissionTask implementation. Either case
you can go
>>>>>>> through the existing implementations, for Provider implementation
see
>>>>>>>  SSHProvider  and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>>>>
>>>>>>> If I get the answers to my questions then i can provide exactly
what
>>>>>>> you need to do. BTW we have cleaned our internal architecture
in Airavata
>>>>>>> 16.0, as a developer you would find it easy to work with Airavata
16.0 that
>>>>>>> Airavata 15.0. But notice master is not yet stable as Airavata
15.0.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Shameera.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psaha4@binghamton.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello Shameera,
>>>>>>>> I am working on jet-stream project, where I have to find
out a way
>>>>>>>> to submit a job in mesos/marathon cluster through Airavata
client. I don't
>>>>>>>> have much idea from where to start looking into. Can you
please give some
>>>>>>>> clue so that I can start working and making changes to java
code for the
>>>>>>>> same.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Pankaj
>>>>>>>>
>>>>>>>> --
>>>>>>> Shameera Rathnayaka
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Thank you
> Supun Nakandala
> Dept. Computer Science and Engineering
> University of Moratuwa
>

Mime
View raw message