Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C31111762D for ; Fri, 4 Sep 2015 19:17:46 +0000 (UTC) Received: (qmail 25975 invoked by uid 500); 4 Sep 2015 19:17:46 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 25931 invoked by uid 500); 4 Sep 2015 19:17:46 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 25920 invoked by uid 99); 4 Sep 2015 19:17:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Sep 2015 19:17:46 +0000 Date: Fri, 4 Sep 2015 19:17:46 +0000 (UTC) From: "Hudson (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-4105) Capacity Scheduler headroom for DRF is wrong MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731273#comment-14731273 ] Hudson commented on YARN-4105: ------------------------------ FAILURE: Integrated in Hadoop-Hdfs-trunk #2273 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2273/]) YARN-4105. Capacity Scheduler headroom for DRF is wrong. Contributed by Chang Li (jlowe: rev 6eaca2e3634a88dc55689e8960352d6248c424d9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/CHANGES.txt > Capacity Scheduler headroom for DRF is wrong > -------------------------------------------- > > Key: YARN-4105 > URL: https://issues.apache.org/jira/browse/YARN-4105 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler > Affects Versions: 2.6.0 > Reporter: Chang Li > Assignee: Chang Li > Fix For: 2.7.2 > > Attachments: YARN-4105.2.patch, YARN-4105.3.patch, YARN-4105.4.patch, YARN-4105.patch > > > relate to the problem discussed in YARN-1857. But the min method is flawed when we are using DRC. Have run into a real scenario in production where queueCapacity: , qconsumed: , consumed: limit: . headRoom calculation returns 88064 where there is only 1536 left in the queue because DRC effectively compare by vcores. It then caused deadlock because RMcontainer allocator thought there is still space for mapper and won't preempt a reducer in a full queue to schedule a mapper. Propose fix with componentwiseMin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)