hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-303) Make contracts of LocalFileSystem and DistributedFileSystem consistent
Date Tue, 01 Sep 2009 18:58:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750046#action_12750046
] 

Konstantin Shvachko commented on HDFS-303:
------------------------------------------

> 4. When renaming a file or directory to a non-existent directory (e.g. /a/b to /c/d,
where /c doesn't exist) LocalFileSystem succeeds (returns true) while HDFS fails (false).
>
> HDFS seems more correct here.

Not sure. When HDFS creates a file under a non-existing path it calls {{mkdirs()}} to create
missing directories on the path. I think the rename should follow the same pattern.

> 5. When renaming a file (or directory) as an existing file (or directory) LocalFileSystem
succeeds (returns true) while HDFS fails (false).

The question is whether the behavior of LocalFileSystem is consistent across different native
local file systems. E.g. are the results the same on Windows and ext3?

> The question here is do we allow file overwrites?

I don't think we should. POSIX rename() has very overloaded semantics. We should keep it simple.
My proposal is to fail rename whenever the target file (or directory) exists.

Does HDFS need to be consistent with LocalFileSystem in this is another question. I think
it is OK that they are inconsistent, especially if LocalFileSystem itself may not be consistent
with the rename results on different platforms.
Here is what Java documentation says about its rename:
http://java.sun.com/javase/6/docs/api/java/io/File.html#renameTo%28java.io.File%29
"Many aspects of the behavior of this method are inherently platform-dependent: The rename
operation might not be able to move a file from one filesystem to another, it might not be
atomic, and it might not succeed if a file with the destination abstract pathname already
exists."

> Make contracts of LocalFileSystem and DistributedFileSystem consistent
> ----------------------------------------------------------------------
>
>                 Key: HDFS-303
>                 URL: https://issues.apache.org/jira/browse/HDFS-303
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Tom White
>         Attachments: hadoop-4114.patch, HDFS-303-common-test-case.patch
>
>
> There are a number of edge cases that the two file system implementations handle differently.
In particular:
> * When trying to make a directory under an existing file, HDFS throws an IOException
while LocalFileSystem doesn't.
> * The FileSytem#listStatus(Path) method returns null for a non-existent file on HDFS,
while LocalFileSytem returns an empty FileStatus array.
> * When trying to rename a non-existent path, LocalFileSystem throws an IOException, while
HDFS returns false.
> * When renaming a file or directory to a non-existent directory (e.g. /a/b to /c/d, where
/c doesn't exist) LocalFileSystem succeeds (returns true) while HDFS fails (false).
> * When renaming a file (or directory) as an existing file (or directory) LocalFileSystem
succeeds (returns true) while HDFS fails (false).
> We should document the expected behaviour for these cases in FileSystem's javadoc, and
make sure all implementations conform to it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message