oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Etienne Koen <koe...@gmail.com>
Subject Re: filemgr ingestion
Date Wed, 10 Sep 2014 16:37:28 GMT
Thanks for clearing that out Chris,

I think that the capability of the filemgr should be considered as a test
on its own and does not actually fall into the category which we are
currently benchmarking for the baseline WAN/LAN performance of the tools.

There should be a separate test section for archiving...

Thanks for the help!

Etienne
On Sep 10, 2014 6:15 PM, "Chris Mattmann" <mattmann@apache.org> wrote:

> Hi Etienne,
>
> Thanks for your query. The default protocol used by the
> RemoteDataTransfer is XML-RPC. It's a call-back mechanism
> that I baked up myself with a configurable chunk size parameter.
>
> My suggestion is rather than use that DataTransfer, I would leverage:
>
> (1) a distributed filesystem like HadoopFS, Tachyon (from BDAS), GlusterFS,
> or even NFS that logically mounts a set of local shared nothing disks as a
> virtual global mount
>
> (2) use the LocalDataTransfer in the File Manager, with #1 in place
>
> The main reason for this is that it allows the people who work on
> distributed
> and reliable filesystems that solve those problems in a great way, it
> allows
> us in the OODT community to take advantage of that work without having to
> write
> all the complex functionality ourselves. That way we just have
> light-weight insulated
> pieces/components in OODT that rely on the workhorses that do things right.
>
> HTH!
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Etienne Koen <koenej@gmail.com>
> Date: Wednesday, September 10, 2014 1:57 AM
> To: Shakeh Khudikyan <Shakeh.E.Khudikyan@jpl.nasa.gov>
> Cc: Chris Mattmann <chris.mattmann@gmail.com>
> Subject: filemgr ingestion
>
> >Hi Shakeh,
> >What is the default protocol that filemgr uses for RemoteDataTransfer
> >ingestion?
> >
> >Also, is the a way to specify the protocol?
> >
> >Cheers
> >Etienne
> >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message