hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13082) RawLocalFileSystem does not fail when moving file to a non-existing directory
Date Thu, 05 May 2016 10:30:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272168#comment-15272168
] 

Steve Loughran commented on HADOOP-13082:
-----------------------------------------


Short term, lets skip those tests. As {{FileSystemContractBaseTest}} is a Junit 3 test, it
can't be @Ignored; you have to just override those test cases with empty ones...it doesn't
show up in the test reporter.

Other things to note

# really FS.rename() should follow the semantics of the posix {{rename()}} command [http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html]
*not that of {{mv}}*. 
# HDFS doesn't quite get it right
# an other things vary

This is something we allude to in the filesystem.md specification file [http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html].

Looking at it, it actually calls out what HDFS does compared to Local FS, stating "this is
what happens". I didn't look into the details of why locafs was doing that when I wrote that
doc and the contract tests, just mentioned it happened and added a flag for the contract tests
{{"rename-creates-dest-dirs"}}, which should be set to indicate behaviour.


Short term: just set that option in the contract-test-options XML file to get through the
tests.

Mid term: we do need to look at this; moving to java 7 IO would be a start ... there's a JIRA
there which nobody has every fully addressed. Maybe that'll be something to work on *after*
getting all these tests in.

(BTW: thanks for your due diligence in determining the history of this feature and what other
code gets up to —your thoroughness is appreciated)

> RawLocalFileSystem does not fail when moving file to a non-existing directory
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-13082
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13082
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Andras Bokor
>            Assignee: Andras Bokor
>
> FileSystemContractBaseTest#testRenameFileMoveToNonExistentDirectory: creates a file then
move it to a non-existing directory. It should fail but it will not (with RawLocalFileSystem)
because in RawLocalFileSystem#rename(Path, Path) method we have a fallback behavior that accomplishes
the rename by a full copy. The full copy will create the new directory and copy the file there.
> I see two possible solutions here:
> # Remove the fallback full copy behavior
> # Before full cp we should check whether the parent directory exists or not. If not return
false an do not do the full copy.
> The fallback logic was added by [HADOOP-9805|https://issues.apache.org/jira/browse/HADOOP-9805].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message