hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andras Bokor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13082) RawLocalFileSystem does not fail when moving file to a non-existing directory
Date Thu, 05 May 2016 08:55:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272089#comment-15272089
] 

Andras Bokor commented on HADOOP-13082:
---------------------------------------

[~stevel@apache.org], [~cnauroth],

Thank you guys for getting back to me.

First of all, yes, fallback logic was added way before HADOOP-9805.

I looked into it a bit deeper and it seems this rename situation is not that simple. We have
9 methods in FileSystemContractBaseTest which is broken and 6 of 9 is broken by fallback logic.
I have a small list about it (it was my notes during the debugging):
{code}testRenameFileMoveToNonExistentDirectory
	Fallback creates the missing directory
testRenameDirectoryAsExistingFile
	Move fails but fallback behavior throws exception. Test expects only a false
testRenameDirectoryMoveToNonExistentDirectory
	Fallback creates the missing directory
testRenameChildDirForbidden
	Fallback will throw an exception (cannot copy x to ist subdirectory y) but contract expects
just a false
testMoveDirUnderParent
	renameTo return false (which is good) but fallback throws exception while test does not expect
exception
testRenameNonExistentPath
	rename returns with false but fallback throws exception
{code}

bq. I'm not sure now, after all, the raw OS doesn't do this, does it?
Yes, it does. In case of one volume mv command do just a rename. The inode will not change.
In case of multi-volume the OS does a copy-delete.

I checked Jakarta IO commons and Guava how do they handle this and they use the same solution
as we do in RawLocalFileSystem (if renameTo false, do a copy-delete). It is not good for us
because we do not know why the renameTo returned false (so we can't decide whether the false
was returned due to cross-volume move or other problem).
>From JDK 7 a new method was introduced. [Files.move|https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move%28java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...%29]
do more or less the same as renameTo and it handles the cross-volume situation.

Now I am confused what to do:
# Skip this one and solve others
# Skip all the rename related contracts
# Solve all the broken contracts with FileUtil#copy
# Solve all the contracts with [Files.move|https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move%28java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...%29]
# Other

What do you guys think?

> RawLocalFileSystem does not fail when moving file to a non-existing directory
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-13082
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13082
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Andras Bokor
>            Assignee: Andras Bokor
>
> FileSystemContractBaseTest#testRenameFileMoveToNonExistentDirectory: creates a file then
move it to a non-existing directory. It should fail but it will not (with RawLocalFileSystem)
because in RawLocalFileSystem#rename(Path, Path) method we have a fallback behavior that accomplishes
the rename by a full copy. The full copy will create the new directory and copy the file there.
> I see two possible solutions here:
> # Remove the fallback full copy behavior
> # Before full cp we should check whether the parent directory exists or not. If not return
false an do not do the full copy.
> The fallback logic was added by [HADOOP-9805|https://issues.apache.org/jira/browse/HADOOP-9805].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message