airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: OODT File Manager - Airavata Integration Document
Date Fri, 30 Oct 2015 05:43:39 GMT
Hi Supun,
1) Do you have a test file that you can share and I will try?
2) How long is it taking?
3) Can you make your log available with DEBUG mode turned on so I can see
what is going on?
Thanks,
Lewis

On Mon, Oct 26, 2015 at 11:52 AM, Supun Nakandala <supun.nakandala@gmail.com
> wrote:

> OODT Devs,
>
> I'm experiencing unacceptable slowness when ingesting some what large
> files(>10MB) using OODT file manager. When I check the OODT server side
> logs I see multiple attempts to overwrite the same input file again and
> again and failing. I am
> using org.apache.oodt.cas.filemgr.ingest.StdIngester.
>
> Any clue on what I am doing wrong here?
>
> Thanks
>
> On Mon, Oct 12, 2015 at 4:58 PM, Chris Mattmann <chris.mattmann@gmail.com>
> wrote:
>
> > Yep it can do either.
> >
> > 1. Use Product.STRUCTURE_HIERARCHICAL for directories.
> > This feature works, and also with data movement, but it’s preferred
> > to do option 2.
> > 2. Manage the hierarchy simply with Product.STRUCTURE_FLAT products,
> > but with versioning to place and co-locate the files according to the
> > organized hierarchy, and then link them with metadata keys.
> >
> > Make sense?
> >
> > Cheers,
> > Chris
> >
> > —
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Suresh Marru <smarru@apache.org>
> > Reply-To: <dev@oodt.apache.org>
> > Date: Monday, October 12, 2015 at 1:47 PM
> > To: <dev@airavata.apache.org>, <dev@oodt.apache.org>
> > Subject: Re: OODT File Manager - Airavata Integration Document
> >
> > >Thanks Chris this is helpful to get a clarification. The only reason we
> > >want to do at a service level so we can have more fine grain control
> based
> > >on users/groups and their roles. We will start by looking at the tomcat
> > >layer and will ask more questions along the way.
> > >
> > >Paraphrasing Supun: Seems like OODT currently supports files organized
> in
> > >a
> > >flat structure. Any suggestions on how to support data products
> organized
> > >in hierarchal directories. As an example, say an Experiment has a
> Workflow
> > >which runs bunch of simulations all organized as inputs and outputs
> under
> > >a
> > >parent experiment. Can we point the file manager to such a organization
> or
> > >is it expected that the hierarchy be managed outside (through metadata
> key
> > >value pairs or otherwise)?
> > >
> > >Thanks,
> > >Suresh
> > >
> > >On Mon, Oct 12, 2015 at 4:32 PM Chris Mattmann <
> chris.mattmann@gmail.com>
> > >wrote:
> > >
> > >> Hey Suresh,
> > >>
> > >> Sure. Right now I would recommend not trying to do auth/auth
> > >> at the File Manager Level, and instead using e.g., Tomcat or
> > >> something to set up like an OAuth realm, and doing it that way.
> > >> Is there a specific need to lock it down at the service level?
> > >> At their core they are XML-RPC, and I think it will be more difficult
> > >> to implement there than trying to do in an app server that sits
> > >> in front.
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >> —
> > >> Chris Mattmann
> > >> chris.mattmann@gmail.com
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Suresh Marru <smarru@apache.org>
> > >> Reply-To: <dev@oodt.apache.org>
> > >> Date: Monday, October 12, 2015 at 1:09 PM
> > >> To: <dev@airavata.apache.org>, <dev@oodt.apache.org>
> > >> Subject: Re: OODT File Manager - Airavata Integration Document
> > >>
> > >> >OODT Dev's,
> > >> >
> > >> >Can we plugin custom authentication to CAS File Manager? I see that
> you
> > >> >could potentially use the OODT single sign on package and have it
> talk
> > >>to
> > >> >a
> > >> >LDAP but wondering if we could do OAuth2 like authentication and
> > >> >authorization.
> > >> >
> > >> >Thanks,
> > >> >Suresh
> > >> >
> > >> >On Tue, Sep 1, 2015 at 12:03 AM DImuthu Upeksha
> > >> ><dimuthu.upeksha2@gmail.com>
> > >> >wrote:
> > >> >
> > >> >> Hi Chris,
> > >> >>
> > >> >> Thank you for your feedback. This is a POC level implementation
> done
> > >>by
> > >> >>me
> > >> >> and there is a lot of room for improvement. It would be great
if
> both
> > >> >> Airavata and OODT teams can go through the design and decide
> whether
> > >> >> Airavata can be integrated with OODT File manager. But my idea
is,
> > >>yes
> > >> >>it
> > >> >> is possible.
> > >> >>
> > >> >> Thanks,
> > >> >> Dimuthu
> > >> >>
> > >> >> On Tue, Sep 1, 2015 at 5:07 AM, Mattmann, Chris A (3980) <
> > >> >> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >> >>
> > >> >>> Great work Dimuthu!
> > >> >>>
> > >> >>> So this means now Airavata and OODT’s File Manager are integrated?
> > >> >>> :=)
> > >> >>>
> > >> >>> CC/dev@oodt.a.o
> > >> >>>
> > >> >>> Cheers,
> > >> >>> Chris
> > >> >>>
> > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> >>> Chris Mattmann, Ph.D.
> > >> >>> Chief Architect
> > >> >>> Instrument Software and Science Data Systems Section (398)
> > >> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >> >>> Office: 168-519, Mailstop: 168-527
> > >> >>> Email: chris.a.mattmann@nasa.gov
> > >> >>> WWW:  http://sunset.usc.edu/~mattmann/
> > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> >>> Adjunct Associate Professor, Computer Science Department
> > >> >>> University of Southern California, Los Angeles, CA 90089 USA
> > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> -----Original Message-----
> > >> >>> From: DImuthu Upeksha <dimuthu.upeksha2@gmail.com>
> > >> >>> Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> > >> >>> Date: Saturday, August 29, 2015 at 1:27 AM
> > >> >>> To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> > >> >>> Subject: OODT File Manager - Airavata Integration Document
> > >> >>>
> > >> >>> >Hi all,
> > >> >>> >
> > >> >>> >
> > >> >>> >Please find the document [1] that describes the steps
followed in
> > >>the
> > >> >>> >attempt of integrating OODT File Manager and AIravata.
This setup
> > >>is
> > >> >>> >currently working on gw77.
> > >> >>> >
> > >> >>> >
> > >> >>> >[1]
> > >> >>> >
> > >> >>>
> > >> >>>
> > >>
> > https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWrP
> > >> >>>-M
> > >> >>> >OFUDw/edit?usp=sharing
> > >> >>> >
> > >> >>> >
> > >> >>> >Thanks
> > >> >>> >Dimuthu
> > >> >>> >
> > >> >>> >
> > >> >>> >--
> > >> >>> >Regards
> > >> >>> >W.Dimuthu Upeksha
> > >> >>> >Undergraduate
> > >> >>> >Department of Computer Science And Engineering
> > >> >>> >University of Moratuwa, Sri Lanka
> > >> >>> >
> > >> >>> >
> > >> >>> >
> > >> >>>
> > >> >>>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Regards
> > >> >>
> > >> >> W.Dimuthu Upeksha
> > >> >> Undergraduate
> > >> >> Department of Computer Science And Engineering
> > >> >>
> > >> >> University of Moratuwa, Sri Lanka
> > >> >>
> > >>
> > >>
> > >>
> >
> >
> >
>
>
> --
> Thank you
> Supun Nakandala
> Dept. Computer Science and Engineering
> University of Moratuwa
>



-- 
*Lewis*

Mime
View raw message