hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] ayushtkn commented on a change in pull request #2305: HDFS-15578: Fix the rename issues with fallback fs enabled
Date Wed, 16 Sep 2020 03:20:13 GMT

ayushtkn commented on a change in pull request #2305:
URL: https://github.com/apache/hadoop/pull/2305#discussion_r489137266



##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestViewDistributedFileSystemWithMountLinks.java
##########
@@ -61,4 +64,55 @@ public void testCreateOnRoot() throws Exception {
   public void testMountLinkWithNonExistentLink() throws Exception {
     testMountLinkWithNonExistentLink(false);
   }
+
+  @Test
+  public void testRenameOnInternalDirWithFallback() throws Exception {
+    Configuration conf = getConf();
+    URI defaultFSURI =
+        URI.create(conf.get(CommonConfigurationKeys.FS_DEFAULT_NAME_KEY));
+    final Path hdfsTargetPath1 = new Path(defaultFSURI + "/HDFSUser");
+    final Path hdfsTargetPath2 = new Path(defaultFSURI + "/NewHDFSUser/next");
+    ViewFsTestSetup.addMountLinksToConf(defaultFSURI.getAuthority(),
+        new String[] {"/HDFSUser", "/NewHDFSUser/next"},
+        new String[] {hdfsTargetPath1.toUri().toString(),
+            hdfsTargetPath2.toUri().toString()}, conf);
+    //Making sure parent dir structure as mount points available in fallback.
+    try (DistributedFileSystem dfs = new DistributedFileSystem()) {
+      dfs.initialize(defaultFSURI, conf);
+      dfs.mkdirs(hdfsTargetPath1);
+      dfs.mkdirs(hdfsTargetPath2);
+    }
+
+    try (FileSystem fs = FileSystem.get(conf)) {
+      Path src = new Path("/newFileOnRoot");
+      Path dst = new Path("/newFileOnRoot1");
+      fs.create(src).close();
+      verifyRename(fs, src, dst);
+
+      src = new Path("/newFileOnRoot1");
+      dst = new Path("/NewHDFSUser/newFileOnRoot");
+      fs.mkdirs(dst.getParent());
+      verifyRename(fs, src, dst);
+
+      src = new Path("/NewHDFSUser/newFileOnRoot");
+      dst = new Path("/NewHDFSUser/newFileOnRoot1");
+      verifyRename(fs, src, dst);
+
+      src = new Path("/NewHDFSUser/newFileOnRoot1");
+      dst = new Path("/newFileOnRoot");
+      verifyRename(fs, src, dst);
+
+      src = new Path("/HDFSUser/newFileOnRoot1");
+      dst = new Path("/HDFSUser/newFileOnRoot");
+      fs.create(src).close();
+      verifyRename(fs, src, dst);
+    }
+  }
+
+  private void verifyRename(FileSystem fs, Path src, Path dst)
+      throws IOException {
+    fs.rename(src, dst);
+    Assert.assertFalse(fs.exists(src));
+    Assert.assertTrue(fs.exists(dst));
+  }

Review comment:
       I tried a couple of cases of rename. Can you give a check once.
   ``` Test
     public void testRenameOnInternalDirWithFallback() throws Exception {
       Configuration conf = getConf();
       URI defaultFSURI =
           URI.create(conf.get(CommonConfigurationKeys.FS_DEFAULT_NAME_KEY));
       final Path hdfsTargetPath1 = new Path(defaultFSURI + "/HDFSUser");
       final Path hdfsTargetPath2 = new Path(defaultFSURI + "/dstNewHDFSUser"
           + "/next");
       ViewFsTestSetup.addMountLinksToConf(defaultFSURI.getAuthority(),
           new String[] {"/HDFSUser", "/NewHDFSUser/next/next1"},
           new String[] {hdfsTargetPath1.toUri().toString(),
               hdfsTargetPath2.toUri().toString()}, conf);
       //Making sure parent dir structure as mount points available in fallback.
       try (DistributedFileSystem dfs = new DistributedFileSystem()) {
         dfs.initialize(defaultFSURI, conf);
         dfs.mkdirs(hdfsTargetPath1);
         dfs.mkdirs(hdfsTargetPath2);
       }
   
       try (FileSystem fs = FileSystem.get(conf)) {
         // Case : 1
         Path src = new Path("/newFileOnRoot");
         Path dst = new Path("/NewHDFSUser/next");
         fs.create(src).close();
         verifyRename(fs, src, dst); // Fails. Shouldn't it move to
         // /NewHDFSUser/next/newFileOnRoot ?
   
          src = new Path("/newFileOnRoot");
          dst = new Path("/NewHDFSUser/next/file");
         verifyRename(fs, src, dst); // Fails. Guess since the parent structure
         // isn't there at fallback?```
   
   I couldn't check more, but
   CASE 1: if the destination is a directory, shouldn't it move src inside it?
   CASE:2 Seems due to parent structure isn't there in fallback?
   
   I didn't try with ViewFs, maybe something similar there as well




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message