hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Le Dem (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8842) local file system behavior of mv into an empty directory is inconsistent with HDFS
Date Tue, 25 Sep 2012 20:30:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463185#comment-13463185
] 

Julien Le Dem commented on HADOOP-8842:
---------------------------------------

this comes from the behavior of java.io.File.renameTo(File)
to see it:
{noformat}
    System.out.println("move to empty dir");
    File tmp = File.createTempFile("tmp_", "_dir");
    tmp.delete();
    tmp.mkdirs();
    File a = new File(tmp, "a");
    File c = new File(tmp, "b/c");
    a.mkdirs();
    new FileWriter(new File(a, "part")).close();
    c.mkdirs();
    System.out.println(a.renameTo(c)); // true
    System.out.println(Arrays.toString(c.listFiles())); // b/c/part

    System.out.println("move to non empty dir");
    File a2 = new File(tmp, "a2");
    File c2 = new File(tmp, "b2/c2");
    a2.mkdirs();
    new FileWriter(new File(a2, "part")).close();
    c2.mkdirs();
    new FileWriter(new File(c2, "dummy")).close();
    System.out.println(a2.renameTo(c2)); // false
{noformat}

Which RawLocalFileSystem is using:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup
{noformat}
249 public boolean rename(Path src, Path dst) throws IOException {
250   if (pathToFile(src).renameTo(pathToFile(dst))) {
251     return true;
252   }
253   return FileUtil.copy(this, src, this, dst, true, getConf());
254 }
{noformat}

I would suggest something like:
{noformat}
public boolean rename(Path src, Path dst) throws IOException {
  File srcFile = pathToFile(src);
  File dstFile = pathToFile(dst);
  if (dstFile.exists() && dstFile.isDirectory()) {
    dstFile = new File(destFile, srcFile.getName());
  }
  if (srcFile.renameTo(dstFile)) {
    return true;
  }
  return FileUtil.copy(this, src, this, new Path(dstFile.toURI()), true, getConf());
}
{noformat}
Or possibly move this logic up in the mv command to pass the final path as dst instead of
just what was passed to the mv command
                
> local file system behavior of mv into an empty directory is inconsistent with HDFS
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-8842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8842
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Julien Le Dem
>
> moving into an empty directory replaces the directory instead.
> See output of attached script to reproduce :
> repro.sh
> {noformat}
> rm -rf local_fs_bug
> mkdir local_fs_bug
> hdfs -rmr local_fs_bug
> hdfs -mkdir local_fs_bug
> echo ">>> HDFS: normal behavior"
> touch part-0000
> hdfs -mkdir local_fs_bug/a
> hdfs -copyFromLocal part-0000 local_fs_bug/a
> hdfs -mkdir local_fs_bug/b
> hdfs -mkdir local_fs_bug/b/c
> echo "content of a: 1 part"
> hdfs -ls local_fs_bug/a
> echo "content of b/c: empty"
> hdfs -ls local_fs_bug/b/c
> echo "mv a b/c"
> hdfs -mv local_fs_bug/a local_fs_bug/b/c
> echo "resulting content of b/c"
> hdfs -ls local_fs_bug/b/c
> echo "a is moved inside of c"
> echo
> echo ">>> local fs: bug"
> mkdir -p local_fs_bug/a
> touch local_fs_bug/a/part-0000
> mkdir -p local_fs_bug/b/c
> echo "content of a: 1 part"
> hdfs -fs local -ls local_fs_bug/a
> echo "content of b/c: empty"
> hdfs -fs local -ls local_fs_bug/b/c
> echo "mv a b/c"
> hdfs -fs local -mv local_fs_bug/a local_fs_bug/b/c
> echo "resulting content of b/c"
> hdfs -fs local -ls local_fs_bug/b/c
> echo "bug: a replaces c"
> echo
> echo ">>> but it works if the destination is not empty"
> mkdir local_fs_bug/a2
> touch local_fs_bug/a2/part-0000
> mkdir -p local_fs_bug/b2/c2
> touch local_fs_bug/b2/c2/dummy
> echo "content of a2: 1 part"
> hdfs -fs local -ls local_fs_bug/a2
> echo "content of b2/c2: 1 dummy file"
> hdfs -fs local -ls local_fs_bug/b2/c2
> echo "mv a2 b2/c2"
> hdfs -fs local -mv local_fs_bug/a2 local_fs_bug/b2/c2
> echo "resulting content of b/c"
> hdfs -fs local -ls local_fs_bug/b2/c2
> echo "a2 is moved inside of c2"
> {noformat}
> Output:
> {noformat}
> >>> HDFS: normal behavior
> content of a: 1 part
> Found 1 items
> -rw-r--r--   3 julien g          0 2012-09-25 17:16 /user/julien/local_fs_bug/a/part-0000
> content of b/c: empty
> mv a b/c
> resulting content of b/c
> Found 1 items
> drwxr-xr-x   - julien g          0 2012-09-25 17:16 /user/julien/local_fs_bug/b/c/a
> a is moved inside of c
> >>> local fs: bug
> content of a: 1 part
> 12/09/25 17:16:34 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> Found 1 items
> -rw-r--r--   1 julien g          0 2012-09-25 17:16 /home/julien/local_fs_bug/a/part-0000
> content of b/c: empty
> 12/09/25 17:16:34 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> mv a b/c
> 12/09/25 17:16:35 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> resulting content of b/c
> 12/09/25 17:16:35 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> Found 1 items
> -rw-r--r--   1 julien g          0 2012-09-25 17:16 /home/julien/local_fs_bug/b/c/part-0000
> bug: a replaces c
> >>> but it works if the destination is not empty
> content of a2: 1 part
> 12/09/25 17:16:36 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> Found 1 items
> -rw-r--r--   1 julien g          0 2012-09-25 17:16 /home/julien/local_fs_bug/a2/part-0000
> content of b2/c2: 1 dummy file
> 12/09/25 17:16:37 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> Found 1 items
> -rw-r--r--   1 julien g          0 2012-09-25 17:16 /home/julien/local_fs_bug/b2/c2/dummy
> mv a2 b2/c2
> 12/09/25 17:16:37 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> resulting content of b/c
> 12/09/25 17:16:38 WARN fs.FileSystem: "local" is a deprecated filesystem name. Use "file:///"
instead.
> Found 2 items
> drwxr-xr-x   - julien g       4096 2012-09-25 17:16 /home/julien/local_fs_bug/b2/c2/a2
> -rw-r--r--   1 julien g          0 2012-09-25 17:16 /home/julien/local_fs_bug/b2/c2/dummy
> a2 is moved inside of c2
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message