reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Inglis <>
Subject Re: [PROPOSAL REEF.NET] Adding an overwrite variable to IFileSystem CopyFromLocal
Date Thu, 12 Jul 2018 20:00:51 GMT
Thats a great question.

So, yes my thought is that it would throw an exception, but it should be a
custom REEF exception since different file system implementations can throw
different exceptions -- and I don't think the app developer should be
responsible for handling exceptions from Azure, HDFS, etc. So the
underlying file system implementation should catch the expected exception
but then throw or wrap the exception into a common REEF exception like
FileOperationException or FileCopyException. The implementation can also
add additional info into the exception via the message if needed.

On Wed, Jul 11, 2018 at 11:16 PM, Julia Wang (QIUHE) <> wrote:

> That make sense. So if override is false and remote file already exist,
> will it throw exception?
> -Julia
> -----Original Message-----
> From: Scott Inglis <>
> Sent: Wednesday, July 11, 2018 4:27 PM
> To:
> Subject: [PROPOSAL REEF.NET] Adding an overwrite variable to IFileSystem
> CopyFromLocal
> Hello,
> I noticed that some of the file implementations like LocalFileSystem and
> Azure Blob will overwrite the file on a remote file system when copying
> from the client. However HDFS file implementation does not overwrite.
> This should have a consistent behavior. Also I want to control if the
> overwrite is done or not. So I would like to make the following proposal:
> 1) Modify the IFileSystem.CopyFromLocal API to take in an overwrite
> boolean parameter
> 2) The overwrite parameter will default to true
> Here is what the api will look like:
> void CopyFromLocal(string localFileName, Uri remoteFileUri, bool overwrite
> = true);
> NOTE: With this change the current behavior in HDFS as overwrite will now
> be true.
> Thoughts?
> Thanks,
> Scott

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message