From yarn-issues-return-76820-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 29 23:24:50 2015 Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50716181C0 for ; Tue, 29 Dec 2015 23:24:50 +0000 (UTC) Received: (qmail 80523 invoked by uid 500); 29 Dec 2015 23:24:50 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 80481 invoked by uid 500); 29 Dec 2015 23:24:50 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 80433 invoked by uid 99); 29 Dec 2015 23:24:50 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Dec 2015 23:24:50 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id B74DF2C1F70 for ; Tue, 29 Dec 2015 23:24:49 +0000 (UTC) Date: Tue, 29 Dec 2015 23:24:49 +0000 (UTC) From: "Wangda Tan (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (YARN-4257) Move scheduler validateConf method to AbstractYarnScheduler and make it protected MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074422#comment-15074422 ] Wangda Tan edited comment on YARN-4257 at 12/29/15 11:24 PM: ------------------------------------------------------------- Thanks [~kasha]/[~rhaase], 0 resource makes sense to me if container is just a pack resources and could be delegated to other processes. (Llama or YARN-1488). After YARN-1488, we can relax 0 resource limits for all schedulers. I agree that we can keep validateConf as-is, each scheduler keeps own criteria about 0 resources. From my side, I still want CS checks 0 resource, since 0 mem container will be killed immediately and 0 vcore container doesn't sound correct in semantic (now we have to launch a process for each container, a process cannot use 0 vcores). Agree? was (Author: leftnoteasy): Thanks [~kasha]/[~rhaase], 0 resource makes sense to me if container is just a pack resources and could be delegated to other processes. (Llama or YARN-1408). After YARN-1408, we can relax 0 resource limits for all schedulers. I agree that we can keep validateConf as-is, each scheduler keeps own criteria about 0 resources. From my side, I still want CS checks 0 resource, since 0 mem container will be killed immediately and 0 vcore container doesn't sound correct in semantic (now we have to launch a process for each container, a process cannot use 0 vcores). Agree? > Move scheduler validateConf method to AbstractYarnScheduler and make it protected > --------------------------------------------------------------------------------- > > Key: YARN-4257 > URL: https://issues.apache.org/jira/browse/YARN-4257 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler > Reporter: Swapnil Daingade > Assignee: Rich Haase > Labels: easyfix > Attachments: YARN-4257.patch > > > Currently FairScheduler, CapacityScheduler and FifoScheduler each have a method private void validateConf(Configuration conf). > All three methods validate the minimum and maximum scheduler allocations for cpu and memory (with minor difference). FairScheduler supports 0 as minimum allocation for cpu and memory, while CapacityScheduler and FifoScheduler do not. We can move this code to AbstractYarnScheduler (avoids code duplication) and make it protected for individual schedulers to override. > Why do we care about a minimum allocation of 0 for cpu and memory? > We contribute to a project called Apache Myriad that run yarn on mesos. Myriad supports a feature call fine grained scaling (fgs). In fgs, a NM is launched with zero capacity (0 cpu and 0 mem). When a yarn container is to be run on the NM, a mesos offer for that node is accepted and the NM capacity is dynamically scaled up to match the accepted mesos offer. On completion of the yarn container, resources are returned back to Mesos and the NM capacity is scaled down back to zero (cpu & mem). > In ResourceTrackerService.registerNodeManager, yarn checks if the NM capacity is at-least as much as yarn.scheduler.minimum-allocation-mb and yarn.scheduler.minimum-allocation-vcores. These values can be set to 0 in yarn-site.xml (so a zero capacity NM is possible). However, the validateConf methods in CapacityScheduler and FifoScheduler do not allow for 0 values for these properties (The FairScheduler one does allow for 0). This behaviour should be consistent or at-least be override able. -- This message was sent by Atlassian JIRA (v6.3.4#6332)