oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Bennett <lmzxq....@gmail.com>
Subject Re: Metadata based versioning
Date Fri, 30 May 2014 13:26:11 GMT
Hey Chris,

Thanks for your reply.

I think you may have clarified some of my understanding of oodt under the
hood. Woot. (or is that woodt?)

Firstly, from OODT-72 I can see how the design decisions were made. It just
so happens that I'm wanting the filename for versioning and suddenly I
understand why FinalFileLocationExtractor is needed. I will now use it with
confidence :-).

Your use of the term 'client side data movement' confused me at first, so I
had to think about it a bit. I was always under the impression (a naive
misconception) that if your file manager existed on a "machine B" you would
need to do a remote data transfer to use that file manager.

But what you're saying is that the following setup is possible:

   - Machine A (client): crawler + repository path + local data transfer
   (i.e. machine A, or the 'client' does not need a file manager running and
   does not need to remote data transfer to the machine B)
   - Machine B (server): file manager (does not need the repository path to
   archive files)

Have I got the right idea?


On 30 May 2014 07:01, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Hey Tom,
> You've correctly discovered this. This was an intentional by-design
> artifact of my belief that versioning and data movement should be
> sort of co-located on the same machine. So if you do client side
> data movement (which most people do), then the versioning should
> happen alongside of it, and thus any metadata extraction present
> there should be available during versioning for use in e.g., Metadata
> based versioning.
> The rub comes in the issue where the metadata is generated on the
> server side and you expect versioning to be available to the system.
> One way of getting around this is taking a look at the way that
> the FinalFileLocationExtractor [1] grabs the latest version of the
> CoreMetKeys.FILE_LOCATION property and then makes it available for e.g.,
> versioning.
> See discussion too in OODT-72 [2] for some rationale behind my
> sentiments there. Happy to discuss!
> Cheers,
> Chris
> [1] http://s.apache.org/bvd
> [2] https://issues.apache.org/jira/browse/OODT-72

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message