oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mattmann <mattm...@apache.org>
Subject Re: Doing a file move in the LocalDataTransferer
Date Fri, 21 Oct 2016 20:04:44 GMT
Another option:

1. Move the files to where they should be in controlled access storage
2. Use InPlaceDataTransfer (no move, copy, anything, just cataloging!)




On 10/21/16, 1:01 PM, "Tom Barber" <tom.barber@meteorite.bi> wrote:

    Yeah I figured there was some data safety to it which is why I asked. Okay,
    well it would certainly help the performance with ingesting large files in
    bulk, and we have 1TB to ingest in one go, so I'll give it a stab early
    next week.
    
    Tom
    
    On Fri, Oct 21, 2016 at 8:57 PM, BW <webb@apache.org> wrote:
    
    > +1
    >
    > On Friday, October 21, 2016, Chris Mattmann <mattmann@apache.org> wrote:
    >
    > > The rationale is to not make the staging area a moving target, whilst
    > > building controlled
    > > access storage. Creating a copy is always more safe in the area of
    > > preservation and provenance,
    > > than making everyting a moving target.
    > >
    > > I would strongly caution against doing a .moveFile() unless you add
    > > facilities in LocalDataTransfer to:
    > >
    > > 1. Make it configurable (by default off) maybe something like
    > > org.apache.oodt.cas.filemgr.datatransferer.local.move
    > > and settable in filemgr.properties
    > > 2. Preserve (in metadata) the original file location
    > > 3. Add locking to the addMetadata facilities and addReferences since they
    > > may be called in different control flows
    > >
    > > I would like to see a design that handles the above and unit tests before
    > > +1’ing such a change.
    > >
    > >
    > >
    > > On 10/21/16, 11:35 AM, "Tom Barber" <tom.barber@meteorite.bi
    > > <javascript:;>> wrote:
    > >
    > >     Hello folks
    > >
    > >     I'm asking here just incase someone knows a reason why this is a bad
    > > idea:
    > >
    > >     We have a bunch of large files on a slow NFS mount which we're
    > > ingesting in
    > >     bulk. Its using the LocalDataTransferer to do the ingestion move and
    > in
    > >     that code all the move calls are really file copies.
    > >
    > >     As you'll all know in doing a copy the drive is actually writing bits
    > > to
    > >     disk, where as doing a move is just moving the file pointer.
    > >
    > >     Is there a reason why the moveFile method actually uses
    > >      FileUtils.copyFile() before I try FileUtils.moveFile() ?
    > >
    > >     Because 1 min for a copy vs 0.06 seconds for a move is far more
    > > preferable.
    > >
    > >     Thanks
    > >
    > >     Tom
    > >
    > >
    > >
    > >
    >
    



Mime
View raw message