airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Supun Nakandala <supun.nakand...@gmail.com>
Subject Re: OODT File Manager - Airavata Integration Document
Date Mon, 26 Oct 2015 18:52:57 GMT
OODT Devs,

I'm experiencing unacceptable slowness when ingesting some what large
files(>10MB) using OODT file manager. When I check the OODT server side
logs I see multiple attempts to overwrite the same input file again and
again and failing. I am
using org.apache.oodt.cas.filemgr.ingest.StdIngester.

Any clue on what I am doing wrong here?

Thanks

On Mon, Oct 12, 2015 at 4:58 PM, Chris Mattmann <chris.mattmann@gmail.com>
wrote:

> Yep it can do either.
>
> 1. Use Product.STRUCTURE_HIERARCHICAL for directories.
> This feature works, and also with data movement, but it’s preferred
> to do option 2.
> 2. Manage the hierarchy simply with Product.STRUCTURE_FLAT products,
> but with versioning to place and co-locate the files according to the
> organized hierarchy, and then link them with metadata keys.
>
> Make sense?
>
> Cheers,
> Chris
>
> —
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
>
>
> -----Original Message-----
> From: Suresh Marru <smarru@apache.org>
> Reply-To: <dev@oodt.apache.org>
> Date: Monday, October 12, 2015 at 1:47 PM
> To: <dev@airavata.apache.org>, <dev@oodt.apache.org>
> Subject: Re: OODT File Manager - Airavata Integration Document
>
> >Thanks Chris this is helpful to get a clarification. The only reason we
> >want to do at a service level so we can have more fine grain control based
> >on users/groups and their roles. We will start by looking at the tomcat
> >layer and will ask more questions along the way.
> >
> >Paraphrasing Supun: Seems like OODT currently supports files organized in
> >a
> >flat structure. Any suggestions on how to support data products organized
> >in hierarchal directories. As an example, say an Experiment has a Workflow
> >which runs bunch of simulations all organized as inputs and outputs under
> >a
> >parent experiment. Can we point the file manager to such a organization or
> >is it expected that the hierarchy be managed outside (through metadata key
> >value pairs or otherwise)?
> >
> >Thanks,
> >Suresh
> >
> >On Mon, Oct 12, 2015 at 4:32 PM Chris Mattmann <chris.mattmann@gmail.com>
> >wrote:
> >
> >> Hey Suresh,
> >>
> >> Sure. Right now I would recommend not trying to do auth/auth
> >> at the File Manager Level, and instead using e.g., Tomcat or
> >> something to set up like an OAuth realm, and doing it that way.
> >> Is there a specific need to lock it down at the service level?
> >> At their core they are XML-RPC, and I think it will be more difficult
> >> to implement there than trying to do in an app server that sits
> >> in front.
> >>
> >> Cheers,
> >> Chris
> >>
> >> —
> >> Chris Mattmann
> >> chris.mattmann@gmail.com
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Suresh Marru <smarru@apache.org>
> >> Reply-To: <dev@oodt.apache.org>
> >> Date: Monday, October 12, 2015 at 1:09 PM
> >> To: <dev@airavata.apache.org>, <dev@oodt.apache.org>
> >> Subject: Re: OODT File Manager - Airavata Integration Document
> >>
> >> >OODT Dev's,
> >> >
> >> >Can we plugin custom authentication to CAS File Manager? I see that you
> >> >could potentially use the OODT single sign on package and have it talk
> >>to
> >> >a
> >> >LDAP but wondering if we could do OAuth2 like authentication and
> >> >authorization.
> >> >
> >> >Thanks,
> >> >Suresh
> >> >
> >> >On Tue, Sep 1, 2015 at 12:03 AM DImuthu Upeksha
> >> ><dimuthu.upeksha2@gmail.com>
> >> >wrote:
> >> >
> >> >> Hi Chris,
> >> >>
> >> >> Thank you for your feedback. This is a POC level implementation done
> >>by
> >> >>me
> >> >> and there is a lot of room for improvement. It would be great if both
> >> >> Airavata and OODT teams can go through the design and decide whether
> >> >> Airavata can be integrated with OODT File manager. But my idea is,
> >>yes
> >> >>it
> >> >> is possible.
> >> >>
> >> >> Thanks,
> >> >> Dimuthu
> >> >>
> >> >> On Tue, Sep 1, 2015 at 5:07 AM, Mattmann, Chris A (3980) <
> >> >> chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >>
> >> >>> Great work Dimuthu!
> >> >>>
> >> >>> So this means now Airavata and OODT’s File Manager are integrated?
> >> >>> :=)
> >> >>>
> >> >>> CC/dev@oodt.a.o
> >> >>>
> >> >>> Cheers,
> >> >>> Chris
> >> >>>
> >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >>> Chris Mattmann, Ph.D.
> >> >>> Chief Architect
> >> >>> Instrument Software and Science Data Systems Section (398)
> >> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> >>> Office: 168-519, Mailstop: 168-527
> >> >>> Email: chris.a.mattmann@nasa.gov
> >> >>> WWW:  http://sunset.usc.edu/~mattmann/
> >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >>> Adjunct Associate Professor, Computer Science Department
> >> >>> University of Southern California, Los Angeles, CA 90089 USA
> >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> -----Original Message-----
> >> >>> From: DImuthu Upeksha <dimuthu.upeksha2@gmail.com>
> >> >>> Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> >> >>> Date: Saturday, August 29, 2015 at 1:27 AM
> >> >>> To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> >> >>> Subject: OODT File Manager - Airavata Integration Document
> >> >>>
> >> >>> >Hi all,
> >> >>> >
> >> >>> >
> >> >>> >Please find the document [1] that describes the steps followed
in
> >>the
> >> >>> >attempt of integrating OODT File Manager and AIravata. This
setup
> >>is
> >> >>> >currently working on gw77.
> >> >>> >
> >> >>> >
> >> >>> >[1]
> >> >>> >
> >> >>>
> >> >>>
> >>
> https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWrP
> >> >>>-M
> >> >>> >OFUDw/edit?usp=sharing
> >> >>> >
> >> >>> >
> >> >>> >Thanks
> >> >>> >Dimuthu
> >> >>> >
> >> >>> >
> >> >>> >--
> >> >>> >Regards
> >> >>> >W.Dimuthu Upeksha
> >> >>> >Undergraduate
> >> >>> >Department of Computer Science And Engineering
> >> >>> >University of Moratuwa, Sri Lanka
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> Regards
> >> >>
> >> >> W.Dimuthu Upeksha
> >> >> Undergraduate
> >> >> Department of Computer Science And Engineering
> >> >>
> >> >> University of Moratuwa, Sri Lanka
> >> >>
> >>
> >>
> >>
>
>
>


-- 
Thank you
Supun Nakandala
Dept. Computer Science and Engineering
University of Moratuwa

Mime
View raw message