Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@oodt.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAFwzmVDsJ9MAytUCo7BsR7gAAB=nzMPDVfqKR3UEG+e=qBG91g@mail.gmail.com>
References: 
 <CAHzXhOjxtjWJ1hiDSngvu44FTJxzwHF1gtt__YdvE6cD37x3dQ@mail.gmail.com>
	<D20A3314.3F99B%chris.a.mattmann@jpl.nasa.gov>
	<CAHzXhOgpRxFBKtDek_RGwDfkWKQ4gNwP4f06j_qsZESnE3VXjQ@mail.gmail.com>
	<CAMtCqM0ZEkbm+W23jzOLWSgfXCGeMwYq8md29imO-uapNjgVhA@mail.gmail.com>
	<D2416678.4B3A8%chris.mattmann@gmail.com>
	<CAMtCqM31PeN3v7=_JQRsdSPao4u7uU1bOOpoj0c8XNrn=H6iFw@mail.gmail.com>
	<D2416CC3.4B414%chris.mattmann@gmail.com>
	<CAFwzmVDsJ9MAytUCo7BsR7gAAB=nzMPDVfqKR3UEG+e=qBG91g@mail.gmail.com>
Date: Thu, 29 Oct 2015 22:43:39 -0700
Message-ID: 
 <CAGaRif20kSiV8WyLiOBPyjjkV3nm=Dp_pFJeuYUkqxoz9EZ7CQ@mail.gmail.com>
Subject: Re: OODT File Manager - Airavata Integration Document
From: Lewis John Mcgibbney <lewis.mcgibbney@gmail.com>
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Cc: Airavata Dev <dev@airavata.apache.org>
Content-Type: multipart/alternative; boundary=001a113ad1bcdf296c05234be81f

--001a113ad1bcdf296c05234be81f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Supun,
1) Do you have a test file that you can share and I will try?
2) How long is it taking?
3) Can you make your log available with DEBUG mode turned on so I can see
what is going on?
Thanks,
Lewis

On Mon, Oct 26, 2015 at 11:52 AM, Supun Nakandala <supun.nakandala@gmail.co=
m
> wrote:

> OODT Devs,
>
> I'm experiencing unacceptable slowness when ingesting some what large
> files(>10MB) using OODT file manager. When I check the OODT server side
> logs I see multiple attempts to overwrite the same input file again and
> again and failing. I am
> using org.apache.oodt.cas.filemgr.ingest.StdIngester.
>
> Any clue on what I am doing wrong here?
>
> Thanks
>
> On Mon, Oct 12, 2015 at 4:58 PM, Chris Mattmann <chris.mattmann@gmail.com=
>
> wrote:
>
> > Yep it can do either.
> >
> > 1. Use Product.STRUCTURE_HIERARCHICAL for directories.
> > This feature works, and also with data movement, but it=E2=80=99s prefe=
rred
> > to do option 2.
> > 2. Manage the hierarchy simply with Product.STRUCTURE_FLAT products,
> > but with versioning to place and co-locate the files according to the
> > organized hierarchy, and then link them with metadata keys.
> >
> > Make sense?
> >
> > Cheers,
> > Chris
> >
> > =E2=80=94
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Suresh Marru <smarru@apache.org>
> > Reply-To: <dev@oodt.apache.org>
> > Date: Monday, October 12, 2015 at 1:47 PM
> > To: <dev@airavata.apache.org>, <dev@oodt.apache.org>
> > Subject: Re: OODT File Manager - Airavata Integration Document
> >
> > >Thanks Chris this is helpful to get a clarification. The only reason w=
e
> > >want to do at a service level so we can have more fine grain control
> based
> > >on users/groups and their roles. We will start by looking at the tomca=
t
> > >layer and will ask more questions along the way.
> > >
> > >Paraphrasing Supun: Seems like OODT currently supports files organized
> in
> > >a
> > >flat structure. Any suggestions on how to support data products
> organized
> > >in hierarchal directories. As an example, say an Experiment has a
> Workflow
> > >which runs bunch of simulations all organized as inputs and outputs
> under
> > >a
> > >parent experiment. Can we point the file manager to such a organizatio=
n
> or
> > >is it expected that the hierarchy be managed outside (through metadata
> key
> > >value pairs or otherwise)?
> > >
> > >Thanks,
> > >Suresh
> > >
> > >On Mon, Oct 12, 2015 at 4:32 PM Chris Mattmann <
> chris.mattmann@gmail.com>
> > >wrote:
> > >
> > >> Hey Suresh,
> > >>
> > >> Sure. Right now I would recommend not trying to do auth/auth
> > >> at the File Manager Level, and instead using e.g., Tomcat or
> > >> something to set up like an OAuth realm, and doing it that way.
> > >> Is there a specific need to lock it down at the service level?
> > >> At their core they are XML-RPC, and I think it will be more difficul=
t
> > >> to implement there than trying to do in an app server that sits
> > >> in front.
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >> =E2=80=94
> > >> Chris Mattmann
> > >> chris.mattmann@gmail.com
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Suresh Marru <smarru@apache.org>
> > >> Reply-To: <dev@oodt.apache.org>
> > >> Date: Monday, October 12, 2015 at 1:09 PM
> > >> To: <dev@airavata.apache.org>, <dev@oodt.apache.org>
> > >> Subject: Re: OODT File Manager - Airavata Integration Document
> > >>
> > >> >OODT Dev's,
> > >> >
> > >> >Can we plugin custom authentication to CAS File Manager? I see that
> you
> > >> >could potentially use the OODT single sign on package and have it
> talk
> > >>to
> > >> >a
> > >> >LDAP but wondering if we could do OAuth2 like authentication and
> > >> >authorization.
> > >> >
> > >> >Thanks,
> > >> >Suresh
> > >> >
> > >> >On Tue, Sep 1, 2015 at 12:03 AM DImuthu Upeksha
> > >> ><dimuthu.upeksha2@gmail.com>
> > >> >wrote:
> > >> >
> > >> >> Hi Chris,
> > >> >>
> > >> >> Thank you for your feedback. This is a POC level implementation
> done
> > >>by
> > >> >>me
> > >> >> and there is a lot of room for improvement. It would be great if
> both
> > >> >> Airavata and OODT teams can go through the design and decide
> whether
> > >> >> Airavata can be integrated with OODT File manager. But my idea is=
,
> > >>yes
> > >> >>it
> > >> >> is possible.
> > >> >>
> > >> >> Thanks,
> > >> >> Dimuthu
> > >> >>
> > >> >> On Tue, Sep 1, 2015 at 5:07 AM, Mattmann, Chris A (3980) <
> > >> >> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >> >>
> > >> >>> Great work Dimuthu!
> > >> >>>
> > >> >>> So this means now Airavata and OODT=E2=80=99s File Manager are i=
ntegrated?
> > >> >>> :=3D)
> > >> >>>
> > >> >>> CC/dev@oodt.a.o
> > >> >>>
> > >> >>> Cheers,
> > >> >>> Chris
> > >> >>>
> > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
++
> > >> >>> Chris Mattmann, Ph.D.
> > >> >>> Chief Architect
> > >> >>> Instrument Software and Science Data Systems Section (398)
> > >> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >> >>> Office: 168-519, Mailstop: 168-527
> > >> >>> Email: chris.a.mattmann@nasa.gov
> > >> >>> WWW:  http://sunset.usc.edu/~mattmann/
> > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
++
> > >> >>> Adjunct Associate Professor, Computer Science Department
> > >> >>> University of Southern California, Los Angeles, CA 90089 USA
> > >> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
++
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> -----Original Message-----
> > >> >>> From: DImuthu Upeksha <dimuthu.upeksha2@gmail.com>
> > >> >>> Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> > >> >>> Date: Saturday, August 29, 2015 at 1:27 AM
> > >> >>> To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> > >> >>> Subject: OODT File Manager - Airavata Integration Document
> > >> >>>
> > >> >>> >Hi all,
> > >> >>> >
> > >> >>> >
> > >> >>> >Please find the document [1] that describes the steps followed =
in
> > >>the
> > >> >>> >attempt of integrating OODT File Manager and AIravata. This set=
up
> > >>is
> > >> >>> >currently working on gw77.
> > >> >>> >
> > >> >>> >
> > >> >>> >[1]
> > >> >>> >
> > >> >>>
> > >> >>>
> > >>
> > https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWr=
P
> > >> >>>-M
> > >> >>> >OFUDw/edit?usp=3Dsharing
> > >> >>> >
> > >> >>> >
> > >> >>> >Thanks
> > >> >>> >Dimuthu
> > >> >>> >
> > >> >>> >
> > >> >>> >--
> > >> >>> >Regards
> > >> >>> >W.Dimuthu Upeksha
> > >> >>> >Undergraduate
> > >> >>> >Department of Computer Science And Engineering
> > >> >>> >University of Moratuwa, Sri Lanka
> > >> >>> >
> > >> >>> >
> > >> >>> >
> > >> >>>
> > >> >>>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Regards
> > >> >>
> > >> >> W.Dimuthu Upeksha
> > >> >> Undergraduate
> > >> >> Department of Computer Science And Engineering
> > >> >>
> > >> >> University of Moratuwa, Sri Lanka
> > >> >>
> > >>
> > >>
> > >>
> >
> >
> >
>
>
> --
> Thank you
> Supun Nakandala
> Dept. Computer Science and Engineering
> University of Moratuwa
>


--=20
*Lewis*

--001a113ad1bcdf296c05234be81f--