oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mattmann <mattm...@apache.org>
Subject Re: Doing a file move in the LocalDataTransferer
Date Fri, 21 Oct 2016 19:55:20 GMT
The rationale is to not make the staging area a moving target, whilst building controlled
access storage. Creating a copy is always more safe in the area of preservation and provenance,
than making everyting a moving target.

I would strongly caution against doing a .moveFile() unless you add facilities in LocalDataTransfer
to:
  
1. Make it configurable (by default off) maybe something like org.apache.oodt.cas.filemgr.datatransferer.local.move

and settable in filemgr.properties
2. Preserve (in metadata) the original file location
3. Add locking to the addMetadata facilities and addReferences since they may be called in
different control flows

I would like to see a design that handles the above and unit tests before +1’ing such a
change.



On 10/21/16, 11:35 AM, "Tom Barber" <tom.barber@meteorite.bi> wrote:

    Hello folks
    
    I'm asking here just incase someone knows a reason why this is a bad idea:
    
    We have a bunch of large files on a slow NFS mount which we're ingesting in
    bulk. Its using the LocalDataTransferer to do the ingestion move and in
    that code all the move calls are really file copies.
    
    As you'll all know in doing a copy the drive is actually writing bits to
    disk, where as doing a move is just moving the file pointer.
    
    Is there a reason why the moveFile method actually uses
     FileUtils.copyFile() before I try FileUtils.moveFile() ?
    
    Because 1 min for a copy vs 0.06 seconds for a move is far more preferable.
    
    Thanks
    
    Tom
    



Mime
View raw message