storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [storm] dandsager1 commented on a change in pull request #3128: STORM-3509 Improved RAS scheduling
Date Wed, 02 Oct 2019 21:37:54 GMT
dandsager1 commented on a change in pull request #3128: STORM-3509 Improved RAS scheduling
URL: https://github.com/apache/storm/pull/3128#discussion_r330783420
 
 

 ##########
 File path: storm-server/src/main/java/org/apache/storm/scheduler/resource/ResourceAwareScheduler.java
 ##########
 @@ -247,13 +234,169 @@ private void scheduleTopology(TopologyDetails td, Cluster cluster,
final User to
                 }
             } catch (Exception ex) {
                 markFailedTopology(topologySubmitter, cluster, td,
-                                   "Internal Error - Exception thrown when scheduling. Please
check logs for details", ex);
+                        "Internal Error - Exception thrown when scheduling. Please check
logs for details", ex);
                 return;
             }
         }
         markFailedTopology(topologySubmitter, cluster, td, "Failed to schedule within " +
maxSchedulingAttempts + " attempts");
     }
 
+    /*
+     * Class for tracking resources for scheduling a topology.
+     *
+     * Ideally we would simply track NormalizedResources, but shared topology memory complicates
things.
+     * Topologies with shared memory may use more than the SharedMemoryLowerBound, and topologyRequiredResources
+     * ignores shared memory.
+     *
+     * Resources are tracked in two ways:
+     * 1) AvailableResources. Track cluster available resources and required topology resources.
+     * 2) RemainingRequiredResources. Start with required topology resources, and deduct
for partially scheduled and evicted topologies.
+     */
+    private class TopologySchedulingResources {
+        ResourceMetrics metrics;
 
 Review comment:
   `metrics` variable is used in NormalizedResourceOffer.remove(). It tracks when resources
become negative. During scheduling, negative resources indicate a scheduling failure; otherwise
resources should not become negative. I don't want to use Cluster metrics because (1) it causes
testSchedulingNegativeResources to fail and (2) erroneous logging of (scheduling only) negative
resources.
   
   Maybe pass null metrics and change code to handle handle that?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message