Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@oodt.apache.org
User-Agent: Microsoft-MacOutlook/f.1a.0.160910
Date: Fri, 21 Oct 2016 12:55:20 -0700
Subject: Re: Doing a file move in the LocalDataTransferer
From: Chris Mattmann <mattmann@apache.org>
To: <dev@oodt.apache.org>
Message-ID: <0F7354C7-23D3-4A16-861C-F618508ABBAB@gmail.com>
Thread-Topic: Doing a file move in the LocalDataTransferer
References: <CABML50dWfz1aVS5w=WAqG9w+POBx5aG3cTMsgitgX5a8bS=2ng@mail.gmail.com>
In-Reply-To: <CABML50dWfz1aVS5w=WAqG9w+POBx5aG3cTMsgitgX5a8bS=2ng@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="UTF-8"
Content-transfer-encoding: quoted-printable
archived-at: Fri, 21 Oct 2016 19:55:29 -0000

The rationale is to not make the staging area a moving target, whilst build=
ing controlled
access storage. Creating a copy is always more safe in the area of preserva=
tion and provenance,
than making everyting a moving target.

I would strongly caution against doing a .moveFile() unless you add facilit=
ies in LocalDataTransfer to:
 =20
1. Make it configurable (by default off) maybe something like org.apache.oo=
dt.cas.filemgr.datatransferer.local.move=20
and settable in filemgr.properties
2. Preserve (in metadata) the original file location
3. Add locking to the addMetadata facilities and addReferences since they m=
ay be called in different control flows

I would like to see a design that handles the above and unit tests before +=
1=E2=80=99ing such a change.


On 10/21/16, 11:35 AM, "Tom Barber" <tom.barber@meteorite.bi> wrote:

    Hello folks
   =20
    I'm asking here just incase someone knows a reason why this is a bad id=
ea:
   =20
    We have a bunch of large files on a slow NFS mount which we're ingestin=
g in
    bulk. Its using the LocalDataTransferer to do the ingestion move and in
    that code all the move calls are really file copies.
   =20
    As you'll all know in doing a copy the drive is actually writing bits t=
o
    disk, where as doing a move is just moving the file pointer.
   =20
    Is there a reason why the moveFile method actually uses
     FileUtils.copyFile() before I try FileUtils.moveFile() ?
   =20
    Because 1 min for a copy vs 0.06 seconds for a move is far more prefera=
ble.
   =20
    Thanks
   =20
    Tom
   =20