From dev-return-60849-archive-asf-public=cust-asf.ponee.io@storm.apache.org Thu Oct 3 15:12:14 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4182518065B for ; Thu, 3 Oct 2019 17:12:14 +0200 (CEST) Received: (qmail 71472 invoked by uid 500); 3 Oct 2019 15:12:13 -0000 Mailing-List: contact dev-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@storm.apache.org Delivered-To: mailing list dev@storm.apache.org Received: (qmail 71461 invoked by uid 99); 3 Oct 2019 15:12:13 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Oct 2019 15:12:13 +0000 From: GitBox To: dev@storm.apache.org Subject: [GitHub] [storm] dandsager1 commented on a change in pull request #3128: STORM-3509 Improved RAS scheduling Message-ID: <157011553328.2276.13914386176788673382.gitbox@gitbox.apache.org> Date: Thu, 03 Oct 2019 15:12:13 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit dandsager1 commented on a change in pull request #3128: STORM-3509 Improved RAS scheduling URL: https://github.com/apache/storm/pull/3128#discussion_r331096436 ########## File path: storm-server/src/main/java/org/apache/storm/scheduler/resource/ResourceAwareScheduler.java ########## @@ -247,13 +234,169 @@ private void scheduleTopology(TopologyDetails td, Cluster cluster, final User to } } catch (Exception ex) { markFailedTopology(topologySubmitter, cluster, td, - "Internal Error - Exception thrown when scheduling. Please check logs for details", ex); + "Internal Error - Exception thrown when scheduling. Please check logs for details", ex); return; } } markFailedTopology(topologySubmitter, cluster, td, "Failed to schedule within " + maxSchedulingAttempts + " attempts"); } + /* + * Class for tracking resources for scheduling a topology. + * + * Ideally we would simply track NormalizedResources, but shared topology memory complicates things. + * Topologies with shared memory may use more than the SharedMemoryLowerBound, and topologyRequiredResources + * ignores shared memory. + * + * Resources are tracked in two ways: + * 1) AvailableResources. Track cluster available resources and required topology resources. + * 2) RemainingRequiredResources. Start with required topology resources, and deduct for partially scheduled and evicted topologies. + */ + private class TopologySchedulingResources { + ResourceMetrics metrics; + boolean remainingResourcesSet; + + NormalizedResourceOffer clusterAvailableResources; + NormalizedResourceRequest topologyRequiredResources; + NormalizedResourceRequest topologyScheduledResources; + + double clusterAvailableMemory; + double topologyRequiredNonSharedMemory; + double topologySharedMemoryLowerBound; + + NormalizedResourceOffer remainingRequiredTopologyResources; + double remainingRequiredTopologyMemory; + double topologyScheduledMemory; + + TopologySchedulingResources(Cluster cluster, TopologyDetails td) { + remainingResourcesSet = false; + metrics = new ResourceMetrics(new StormMetricsRegistry()); + + // available resources + clusterAvailableResources = cluster.getNonBlacklistedClusterAvailableResources(Collections.emptyList()); Review comment: returns lower bound since blacklisted supervisors do not contribute ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services