hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-11452) Revisit FileSystem.rename(path, path, options)
Date Thu, 05 Jan 2017 14:38:58 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-11452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-11452:
    Attachment: HADOOP-11452-002.patch

Patch 002

move the error strings from FSDirRenameOp into FSExceptionMessages, then  use them in the
base FileSystem.rename/3, the latter adding all the checks from the HDFS impl. 

This means the base impl fails on root directories, fails if dest==source, doesn't delete
the source if it == dest. 

Also, pulled the check for an empty dir out of the method for an override point. Why? Allows
object stores to implement a more efficient check for a path being a non-empty dir. The listFiles()
command returns an array of all children, which is very expensive on any DFS with many children,
but awfully expensive on an object store. Even if blobstores don't override the whole method,
they can override this (protected) method for performance.

*No tests*. This is not ready to go in, just shows what changes are needed in the functional
code. The checks in the base implementation now should match that in HDFS, though possibly
in a different order. The docs need to be synced up, and a contract test suite for this rename
written. Issue: add it to the existing rename tests, or create a new suite? I'm minded to
just add them to the existing test case, so everything automatically picks it up. All base
class implementations should be have similarly: if one passes, they all should

> Revisit FileSystem.rename(path, path, options)
> ----------------------------------------------
>                 Key: HADOOP-11452
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11452
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: fs
>    Affects Versions: 2.7.3
>            Reporter: Yi Liu
>            Assignee: Steve Loughran
>         Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected and with _deprecated_
annotation. And the default implementation is not atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a good and
atomic implementation. (Also an interesting thing in {{DFSClient}}, the _deprecated_ annotations
for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since it's atomic
for rename+overwrite, also it saves RPC calls if user desires rename+overwrite.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message