hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andras Bokor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15361) RawLocalFileSystem should use Java nio framework for rename
Date Sun, 08 Apr 2018 02:01:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429576#comment-16429576

Andras Bokor commented on HADOOP-15361:


I am bit confused how to resolve the caveats.
{quote}The compatibility is the troublespot here. How does it relate to what we have in filesystem.md?
There are some caveats and differences between RawLocal and HDFS that could be affected:
* filsystem.md states that if the source does not exist we should throw FileNotFoundException
but HDFS does not throw exception and contract test also expects only a false (FileSystemContractBaseTest#testRenameNonExistentPath).
 * local filesystem is able to replace a file but HDFS does not
 * if the parent folder of the destination does not exist HDFS fails but local filesystem
creates the missing directories.

What is best strategy here? Should we keep the sync with filesystem.md or follow HDFS and
the contract tests?

For me it seems filesystem.md just states what is happening, these behaviors are not intended.

> RawLocalFileSystem should use Java nio framework for rename
> -----------------------------------------------------------
>                 Key: HADOOP-15361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15361
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Andras Bokor
>            Assignee: Andras Bokor
>            Priority: Major
>              Labels: incompatibleChange
>         Attachments: HADOOP-15361.01.patch
> Currently RawLocalFileSystem uses a fallback logic for cross-volume renames. The fallback
logic is a copy-on-fail logic so when rename fails it copies the source then delete it.
>  An additional fallback logic was needed for Windows to provide POSIX rename behavior.
> Due to the fallback logic RawLocalFileSystem does not pass the contract tests (HADOOP-13082).
> With using Java nio framework both could be eliminated since it is not platform dependent
and provides cross-volume rename.
> In addition the fallback logic for Windows is not correct since Java io overrides the
destination only if the source is also a directory but handleEmptyDstDirectoryOnWindows method
checks only the destination. That means rename allows to override a directory with a file
on Windows but not on Unix.
> File#renameTo and Files#move are not 100% compatible:
>  If the source is a directory and the destination is an empty directory File#renameTo
overrides the source but Files#move is does not. We have to use {{StandardCopyOption.REPLACE_EXISTING}}
but it overrides the destination even if the source or the destination is a file. So to make
them compatible we have to check that the either the source or the destination is a directory
before we add the copy option.
> I think the correct strategy is
>  * Where the contract test passed so far it should pass after this
>  * Where the contract test failed because of Java specific think and not because of the
fallback logic we should keep the original behavior.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message