hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] fengnanli commented on a change in pull request #2266: [RBF] HDFS-15554 Force router check file existence in destinations before adding/updating mount points
Date Fri, 04 Sep 2020 05:44:21 GMT

fengnanli commented on a change in pull request #2266:
URL: https://github.com/apache/hadoop/pull/2266#discussion_r483399376



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java
##########
@@ -562,11 +595,35 @@ public GetDestinationResponse getDestination(
       LOG.error("Cannot get location for {}: {}",
           src, ioe.getMessage());
     }
-    if (nsIds.isEmpty() && !locations.isEmpty()) {
-      String nsId = locations.get(0).getNameserviceId();
-      nsIds.add(nsId);
+    return nsIds;
+  }
+
+  /**
+   * Verify the file exists in destination nameservices to avoid dangling
+   * mount points.
+   *
+   * @param entry the new mount points added, could be from add or update.
+   * @return destination nameservices where the file doesn't exist.
+   * @throws IOException
+   */
+  private List<String> verifyFileInDestinations(MountTable entry)

Review comment:
       @goiri Uploaded an early version of trying to fix all tests. This is pretty tedious
work so before I spend more time on this, let me know your thoughts.
   There are mainly two types of tests when dealing with mount table:
   1. Use mock RouterRpcServer and so on, this way no downstream namenode calls are made.
I put the mock as well, see the change for TestRouterAdmin.java
   2. Use real downstream namenode interaction, see TestRouterMountTable.java. I created the
paths before calling mount points change.
   
   I kept thinking a much easier way is to add a Router server side config to turn this on
and the default is on. In the tests I can just turn the config off explicitly and this way
I don't need to deal with individual tests.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message