airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Supun Nakandala <supun.nakand...@gmail.com>
Subject Re: launching a job through Airavata to Mesos cluster
Date Sat, 14 Nov 2015 20:26:28 GMT
yes Pankaj. Even though the roles information are in IS the capabilities
for different roles are defined in the PGA. Every user by default has the
role "Internal/everyone". So when you PGA admin-role to that you will gain
admin capabilities in your PGA.

On Sat, Nov 14, 2015 at 3:05 PM, Pankaj Saha <psaha4@binghamton.edu> wrote:

> Hi Supun,
> I need a small clarification here. I am using local docker for PGA and
> servers but I am using the sam credential service that you guys are using.
> So do you think making local changes in the pga_config will do the work?
>
> Thanks
> Pankaj
>
> On Sat, Nov 14, 2015 at 3:01 PM, Supun Nakandala <
> supun.nakandala@gmail.com> wrote:
>
>> Hi Pankaj,
>>
>> You can set to admin role to Internal/everyone in pga_config.php (in
>> <PGA_ROOT>/app/config/pga_config.php) and you will admin permissions.
>>
>> On Sat, Nov 14, 2015 at 2:06 PM, Pankaj Saha <psaha4@binghamton.edu>
>> wrote:
>>
>>> Hi Suresh and Marlon
>>> I am trying to launch a hello world sample application from my local
>>> docker Airavata PGA. Can I get a demo admin role for few days so that I can
>>> create an application and test some sample experiments? It looks like
>>> without admin role I can not create applications.
>>>
>>> Thanks
>>> Pankaj
>>>
>>> On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <marpierc@iu.edu> wrote:
>>>
>>>> But you could answer my implied question below with zero knowledge of
>>>> Airavata.
>>>>
>>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>>> Date: Wednesday, October 28, 2015 at 1:48 PM
>>>>
>>>> To: Marlon Pierce <marpierc@iu.edu>
>>>> Cc: Suresh Marru <smarru@apache.org>, Airavata Dev <
>>>> dev@airavata.apache.org>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> As Suresh said I think this is the time  when I have to understand
>>>> under the hood details of Airavata Architecture, which I could not gain
>>>> much during GSoC.
>>>>
>>>> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpierc@iu.edu>
>>>> wrote:
>>>>
>>>>> The “hello world” for us is usually to just run some linux command
on
>>>>> the remote queuing system. So that is something you could easily wrap
via
>>>>> docker and submit via Mesos (I hope it is easy…).  Related to this
is
>>>>> determining that the job has gone through various states (queued,
>>>>> executing, completed) and communicating that back to Airavata.
>>>>>
>>>>>
>>>>> Marlon
>>>>>
>>>>>
>>>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>>>> Date: Wednesday, October 28, 2015 at 12:54 PM
>>>>> To: Marlon Pierce <marpierc@iu.edu>
>>>>> Cc: Suresh Marru <smarru@apache.org>, Airavata Dev <
>>>>> dev@airavata.apache.org>
>>>>>
>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>
>>>>> Hi Marlon,
>>>>> "we more typically use Airavata to run scientific applications on
>>>>> clusters" - Our plan is to run dockerized scientific application
>>>>> through Airavata.
>>>>> Definitely our target is not to run a Tomcat webserver, I just said
>>>>> that as an example.
>>>>>
>>>>> Give me some time, after I discuss with Prof. Madhu and Renan I will
>>>>> try to reply your queries in details.
>>>>>
>>>>> Thanks
>>>>> Pankaj
>>>>>
>>>>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpierc@iu.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Pankaj—
>>>>>>
>>>>>> I may not have the full background on what you want to do, but we
>>>>>> more typically use Airavata to run scientific applications on clusters.
>>>>>> This goes through a scheduling and queuing system (Torque/MOAB or
SLURM).
>>>>>> If you want to do this, you would first need to say how you would
submit a
>>>>>> dockerized application using Mesos as your queueing system. You could
do
>>>>>> this for a Tomcat server, but that is a little different from the
jobs we
>>>>>> normally run.
>>>>>>
>>>>>> We don’t use Airavata to bootstrap itself, although that may be
an
>>>>>> interesting way to think of things.  There it is interesting to use
Mesos +
>>>>>> Marathon to run Airavata services (which may themselves by dockerized)
but
>>>>>> that would be done today outside Airavata.
>>>>>>
>>>>>> Marlon
>>>>>>
>>>>>>
>>>>>> From: Suresh Marru <smarru@apache.org>
>>>>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>>>>> To: Airavata Dev <dev@airavata.apache.org>
>>>>>> Cc: Marlon Pierce <marpierc@iu.edu>
>>>>>>
>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>
>>>>>> Hi Pankaj,
>>>>>>
>>>>>> You cannot do this out of the box now, but with some guidance you
can
>>>>>> easily write a task to run  Dockerized application on a Mesos cluster,
>>>>>> please use 0.16 for this. As Shameera explained this is a improvised
>>>>>> architecture and any contributions you will make will carry forward.
Note:
>>>>>> This will require you to understand under the hood details of Airavata
>>>>>> Architecture but I am sure you will motivate others to work with
you to
>>>>>> speed it up.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psaha4@binghamton.edu>
>>>>>> wrote:
>>>>>>
>>>>>> “launch a dockerized job”  means submitting a docker container
for
>>>>>> any specific job (say a java webserver docker container) through
Airavata.
>>>>>> What I remember Airavata php gateway can not submit any docker
>>>>>> container(correct me if I am wrong). What I want to figure out initially
is
>>>>>> whether its feasible to submit a docker container with help of Airavata
>>>>>> into a Mesos cluster.
>>>>>>
>>>>>> Thanks
>>>>>> Pankaj
>>>>>>
>>>>>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpierc@iu.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> I’ll add: if submitting a job to (for example) a SLURM queuing
>>>>>>> system, we need to create the correct SLURM submission script
and submit it
>>>>>>> by executing the correct command line operation (sbatch).
>>>>>>>
>>>>>>> From: Marlon Pierce <marpierc@iu.edu>
>>>>>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>>>>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>>>>>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru
<
>>>>>>> smarru@apache.org>, Pankaj Saha <psaha4@binghamton.edu>
>>>>>>>
>>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>>
>>>>>>> Hi Pankaj, can you say more about what you mean by “launch
a
>>>>>>> dockerized job”?
>>>>>>>
>>>>>>> Marlon
>>>>>>>
>>>>>>>
>>>>>>> From: Pankaj Saha <psaha4@binghamton.edu>
>>>>>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>>>>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>>>>>> To: Suresh Marru <smarru@apache.org>
>>>>>>> Cc: Airavata Dev <dev@airavata.apache.org>
>>>>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>>>>
>>>>>>> Hi Suresh,
>>>>>>>
>>>>>>> My initial understanding is, I have to launch a dockerized job
>>>>>>> through Airavata which will be run in the Mesos cluster.  I was
looking for
>>>>>>> the code which submits jobs and wanted to make changes such a
way that it
>>>>>>> can submit docker containers to Mesos/Marathon cluster.
>>>>>>>
>>>>>>> I can use 0.15 branch and I have no idea about data transfer
>>>>>>> protocol and job submission protocols that Shameera has mentioned.
I may
>>>>>>> want to submit jobs by submitting a JSON through command line
or any other
>>>>>>> way that you guys feel is more appropriate.
>>>>>>>
>>>>>>> I can talk to Prof. Madhu and let you know more on the requirement.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Pankaj
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <smarru@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Pankaj can you clarify the following:
>>>>>>>>
>>>>>>>> Do you want an Airavata instance to run some dockerized
>>>>>>>> applications scheduled by Mesos? Or do you just need a client
which will
>>>>>>>> connect to Airavata hosted and managed by Mesos/Marathon?
>>>>>>>>
>>>>>>>> Suresh
>>>>>>>>
>>>>>>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Pankaj,
>>>>>>>>
>>>>>>>> Wich version of Airavata you are working on?  what is the
data
>>>>>>>> transfer protocol? What is the job submission protocol?
>>>>>>>>
>>>>>>>> Short answer:  if you are using Airavata 0.15 then you need
to
>>>>>>>> write new Provider implementation to submit the request to
Mesos/Marathon
>>>>>>>> cluster.  But if you are using Airavata 16.0 which is current
master, then
>>>>>>>> you need to write JobSubmissionTask implementation. Either
case you can go
>>>>>>>> through the existing implementations, for Provider implementation
see
>>>>>>>>  SSHProvider  and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>>>>>>
>>>>>>>> If I get the answers to my questions then i can provide exactly
>>>>>>>> what you need to do. BTW we have cleaned our internal architecture
in
>>>>>>>> Airavata 16.0, as a developer you would find it easy to work
with Airavata
>>>>>>>> 16.0 that Airavata 15.0. But notice master is not yet stable
as Airavata
>>>>>>>> 15.0.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Shameera.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psaha4@binghamton.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hello Shameera,
>>>>>>>>> I am working on jet-stream project, where I have to find
out a way
>>>>>>>>> to submit a job in mesos/marathon cluster through Airavata
client. I don't
>>>>>>>>> have much idea from where to start looking into. Can
you please give some
>>>>>>>>> clue so that I can start working and making changes to
java code for the
>>>>>>>>> same.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Pankaj
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Shameera Rathnayaka
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Thank you
>> Supun Nakandala
>> Dept. Computer Science and Engineering
>> University of Moratuwa
>>
>
>


-- 
Thank you
Supun Nakandala
Dept. Computer Science and Engineering
University of Moratuwa

Mime
View raw message