hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HDDS-1879) Support multiple excluded scopes when choosing datanodes in NetworkTopology
Date Fri, 30 Aug 2019 17:38:00 GMT

     [ https://issues.apache.org/jira/browse/HDDS-1879?focusedWorklogId=304437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-304437
]

ASF GitHub Bot logged work on HDDS-1879:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 30/Aug/19 17:37
            Start Date: 30/Aug/19 17:37
    Worklog Time Spent: 10m 
      Work Description: xiaoyuyao commented on pull request #1194: HDDS-1879.  Support multiple
excluded scopes when choosing datanodes in NetworkTopology
URL: https://github.com/apache/hadoop/pull/1194#discussion_r319611525
 
 

 ##########
 File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopologyImpl.java
 ##########
 @@ -509,40 +516,48 @@ private Node chooseNodeInternal(String scope, int leafIndex,
       ancestorGen = 0;
     }
 
-    // check overlap of excludedScope and finalScope
-    if (excludedScope != null) {
-      // excludeScope covers finalScope
-      if (finalScope.startsWith(excludedScope)) {
-        return null;
-      }
-      // excludeScope and finalScope share nothing
-      if (!excludedScope.startsWith(finalScope)) {
-        excludedScope = null;
+    // check overlap of excludedScopes and finalScope
+    List<String> mutableExcludedScopes = null;
+    if (excludedScopes != null && !excludedScopes.isEmpty()) {
+      mutableExcludedScopes = new ArrayList<>();
+      for (String s: excludedScopes) {
+        // excludeScope covers finalScope
+        if (finalScope.startsWith(s)) {
+          return null;
+        }
+        // excludeScope and finalScope share nothing case
+        if (s.startsWith(finalScope)) {
+          if (!mutableExcludedScopes.stream().anyMatch(
+              e -> s.startsWith(e))) {
+            mutableExcludedScopes.add(s);
+          }
+        }
       }
     }
 
     // clone excludedNodes before remove duplicate in it
     Collection<Node> mutableExNodes = null;
+
+    // Remove duplicate in excludedNodes
     if (excludedNodes != null) {
-      // Remove duplicate in excludedNodes
       mutableExNodes =
           excludedNodes.stream().distinct().collect(Collectors.toList());
     }
 
-    // remove duplicate in mutableExNodes and excludedScope, given ancestorGen
-    excludedScope = NetUtils.removeDuplicate(this, mutableExNodes,
-        excludedScope, ancestorGen);
+    // remove duplicate in mutableExNodes and mutableExcludedScopes
+    NetUtils.removeDuplicate(this, mutableExNodes, mutableExcludedScopes,
+        ancestorGen);
 
     // calculate available node count
     Node scopeNode = getNode(finalScope);
     int availableNodes = getAvailableNodesCount(
-        scopeNode.getNetworkFullPath(), excludedScope, mutableExNodes,
+        scopeNode.getNetworkFullPath(), mutableExcludedScopes, mutableExNodes,
         ancestorGen);
 
     if (availableNodes <= 0) {
       LOG.warn("No available node in (scope=\"{}\" excludedScope=\"{}\" " +
 
 Review comment:
   NIT: excludeScopes, we may want to expand the list instead of print the addr of the list
in the LOG.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 304437)
    Time Spent: 2h  (was: 1h 50m)

> Support multiple excluded scopes when choosing datanodes in NetworkTopology
> ---------------------------------------------------------------------------
>
>                 Key: HDDS-1879
>                 URL: https://issues.apache.org/jira/browse/HDDS-1879
>             Project: Hadoop Distributed Data Store
>          Issue Type: Sub-task
>            Reporter: Sammi Chen
>            Assignee: Sammi Chen
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message