hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13082) RawLocalFileSystem does not fail when moving file to a non-existing directory
Date Wed, 04 May 2016 21:56:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271503#comment-15271503

Chris Nauroth commented on HADOOP-13082:

HADOOP-9805 is mostly just a refactoring of a patch I wrote for HADOOP-9507 to provide POSIX-like
behavior on Windows by allowing full replacement of a destination directory if it's empty.
 I think the only logical difference of HADOOP-9805 is that it added an explicit check for
running on Windows.  I'd be surprised if HADOOP-9805 introduced the problem.

I don't think either HADOOP-9507 or HADOOP-9805 are the root cause of the current problem
though.  Instead, I think the behavior you see is a result of the final {{FileUtil#copy}}
call.  That line of code goes waaaay back.  (I stopped digging through the git log at 2008,
but the code even pre-dates that.)

We have to assume that there are wide ecosystem dependencies on this behavior for {{RawLocalFileSystem}},
so I don't think it's realistic to change that behavior right now.  This is different from
HDFS semantics, where the rename would fail.  The contract tests are largely derived from
HDFS semantics.  The local file system is an unusual one-off in several ways, and clients
have different expectations for its semantics compared to HDFS or an alternative Hadoop-compatible
file system.

To address this test failure, I think it would be appropriate for the test suite subclass
to override {{testRenameFileMoveToNonExistentDirectory}} and annotate it with {{@Ignore}}.
 The semantics exercised by the test are not applicable to the local file system, so it makes
sense to skip it.  Alternatively, we could introduce another contract flag to control the
test's behavior.  I think that would be overkill though, because I'm pretty sure this is going
to be a one-off for the local file system.

> RawLocalFileSystem does not fail when moving file to a non-existing directory
> -----------------------------------------------------------------------------
>                 Key: HADOOP-13082
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13082
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Andras Bokor
>            Assignee: Andras Bokor
> FileSystemContractBaseTest#testRenameFileMoveToNonExistentDirectory: creates a file then
move it to a non-existing directory. It should fail but it will not (with RawLocalFileSystem)
because in RawLocalFileSystem#rename(Path, Path) method we have a fallback behavior that accomplishes
the rename by a full copy. The full copy will create the new directory and copy the file there.
> I see two possible solutions here:
> # Remove the fallback full copy behavior
> # Before full cp we should check whether the parent directory exists or not. If not return
false an do not do the full copy.
> The fallback logic was added by [HADOOP-9805|https://issues.apache.org/jira/browse/HADOOP-9805].

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message