From yarn-issues-return-29213-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 11 11:29:02 2014 Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A0C511178B for ; Wed, 11 Jun 2014 11:29:02 +0000 (UTC) Received: (qmail 42370 invoked by uid 500); 11 Jun 2014 11:29:02 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 42325 invoked by uid 500); 11 Jun 2014 11:29:02 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 42315 invoked by uid 99); 11 Jun 2014 11:29:02 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Jun 2014 11:29:02 +0000 Date: Wed, 11 Jun 2014 11:29:02 +0000 (UTC) From: "Andrey Stepachev (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2073) Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027644#comment-14027644 ] Andrey Stepachev commented on YARN-2073: ---------------------------------------- I read code and don't cleanly understand why inequality uses Available resources instead of Allocated? With current code preemption performed only when cluster utilised under 20percent. Suppose we have occupied 8GB of 10GB. And default preemption threshold == 0.8f. >From description I thought that preemption should occur, but it doesn't. Code below gives us: {code} 0.8 < availableMb / maxMem 0.8 < (10-8) / 10 0.8 < 0.2 false {code} {code} return (preemptionUtilizationThreshold < Math.max( (float) rootMetrics.getAvailableMB() / clusterResource.getMemory(), (float) rootMetrics.getAvailableVirtualCores() / clusterResource.getVirtualCores())); } {code} Is that was an intention for this parameter (and it should be documented for non native speakers to be more clear, say: preemption will not performed if utilisation is above this parameter) or here was a mistake? > Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free > ------------------------------------------------------------------------------------------------ > > Key: YARN-2073 > URL: https://issues.apache.org/jira/browse/YARN-2073 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler > Affects Versions: 2.4.0 > Reporter: Karthik Kambatla > Assignee: Karthik Kambatla > Priority: Critical > Fix For: 2.5.0 > > Attachments: yarn-2073-0.patch, yarn-2073-1.patch, yarn-2073-2.patch, yarn-2073-3.patch, yarn-2073-4.patch > > > Preemption should kick in only when the currently available slots don't match the request. -- This message was sent by Atlassian JIRA (v6.2#6252)