From yarn-issues-return-153739-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Sep 21 16:53:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A876D180656 for ; Fri, 21 Sep 2018 16:53:04 +0200 (CEST) Received: (qmail 711 invoked by uid 500); 21 Sep 2018 14:53:03 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 700 invoked by uid 99); 21 Sep 2018 14:53:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Sep 2018 14:53:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 3794E1A1497 for ; Fri, 21 Sep 2018 14:53:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id BfZBc-S36kfI for ; Fri, 21 Sep 2018 14:53:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 147865F612 for ; Fri, 21 Sep 2018 14:53:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 499B0E261D for ; Fri, 21 Sep 2018 14:53:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id DB7AA23FA8 for ; Fri, 21 Sep 2018 14:53:00 +0000 (UTC) Date: Fri, 21 Sep 2018 14:53:00 +0000 (UTC) From: "Jason Lowe (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (YARN-8804) resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-8804: ----------------------------- Target Version/s: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2, 2.8.6 Thanks for updating the patch! This is a good performance improvement. However I still think having the allocation directly track the amount relevant to an allocation blocked by queue limits would be cleaner. It would remove the need to do RTTI on child queues. But that's a much bigger change, and I'm OK with this approach for now. Patch does not apply to trunk and needs to be rebased. After doing so, please move the JIRA to Patch Available so Jenkins can comment on it. > resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues > --------------------------------------------------------------------------------------------------- > > Key: YARN-8804 > URL: https://issues.apache.org/jira/browse/YARN-8804 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler > Affects Versions: 3.2.0 > Reporter: Tao Yang > Assignee: Tao Yang > Priority: Critical > Attachments: YARN-8804.001.patch, YARN-8804.002.patch > > > This problem is due to YARN-4280, parent queue will deduct child queue's headroom when the child queue reached its resource limit and the skipped type is QUEUE_LIMIT, the resource limits of deepest parent queue will be correctly calculated, but for non-deepest parent queue, its headroom may be much more than the sum of reached-limit child queues' headroom, so that the resource limit of non-deepest parent may be much less than its true value and block the allocation for later queues. > To reproduce this problem with UT: > (1) Cluster has two nodes whose node resource both are <10GB, 10core> and 3-level queues as below, among them max-capacity of "c1" is 10 and others are all 100, so that max-capacity of queue "c1" is <2GB, 2core> > {noformat} > Root > / | \ > a b c > 10 20 70 > | \ > c1 c2 > 10(max=10) 90 > {noformat} > (2) Submit app1 to queue "c1" and launch am1(resource=<1GB, 1 core>) on nm1 > (3) Submit app2 to queue "b" and launch am2(resource=<1GB, 1 core>) on nm1 > (4) app1 and app2 both ask one <2GB, 1core> containers. > (5) nm1 do 1 heartbeat > Now queue "c" has lower capacity percentage than queue "b", the allocation sequence will be "a" -> "c" -> "b", > queue "c1" has reached queue limit so that requests of app1 should be pending, > headroom of queue "c1" is <1GB, 1core> (=max-capacity - used), > headroom of queue "c" is <18GB, 18core> (=max-capacity - used), > after allocation for queue "c", resource limit of queue "b" will be wrongly calculated as <2GB, 2core>, > headroom of queue "b" will be <1GB, 1core> (=resource-limit - used) > so that scheduler won't allocate one container for app2 on nm1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org