From yarn-issues-return-134658-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Thu Jan 11 18:19:04 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 425E1180789 for ; Thu, 11 Jan 2018 18:19:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 31655160C20; Thu, 11 Jan 2018 17:19:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7B7EF160C42 for ; Thu, 11 Jan 2018 18:19:03 +0100 (CET) Received: (qmail 87597 invoked by uid 500); 11 Jan 2018 17:19:02 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 87586 invoked by uid 99); 11 Jan 2018 17:19:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Jan 2018 17:19:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2BFD9C0977 for ; Thu, 11 Jan 2018 17:19:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.911 X-Spam-Level: X-Spam-Status: No, score=-99.911 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id DZbw0vAw7WxY for ; Thu, 11 Jan 2018 17:19:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 1373C5F6C8 for ; Thu, 11 Jan 2018 17:19:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 92926E0F6E for ; Thu, 11 Jan 2018 17:19:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 4EF2B255CB for ; Thu, 11 Jan 2018 17:19:00 +0000 (UTC) Date: Thu, 11 Jan 2018 17:19:00 +0000 (UTC) From: "Manikandan R (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322572#comment-16322572 ] Manikandan R edited comment on YARN-7159 at 1/11/18 5:18 PM: ------------------------------------------------------------- [~sunilg] Fixed those 2 junit failures. Though {{TestFairSchedulerConfiguration}} changes has fixed the junit issue, requires you to check in detail because of following flow: {code} assertEquals(customResourceInformation(20000L, ""), - calculator.normalize(customResource(10001L, ""), min, max, increment) + calculator.normalize(customResource(19999L, ""), min, max, increment) .getResourceInformation(A_CUSTOM_RESOURCE)); {code} In the above junit test case, {{increment}} has been derived from {{FairSchedulerConfiguration#getIncrementAllocation}} which returns 10k as o/p. {code} conf.set(YarnConfiguration.RESOURCE_TYPES + ".a-custom-resource" + FairSchedulerConfiguration.INCREMENT_ALLOCATION, "10");{code}. Here, value has been configured as 10 with "" units, but {{ResourceUtils#createResourceTypesArray}} triggered from {{FairSchedulerConfiguration#getIncrementAllocation}} returns array of RI's containing A_CUSTOM_RESOURCE with value as 10k, as it simply copies the object and modifies only the value. Is this correct behaviour? Based on our earlier discussion, conversion should happen if there is units difference and source unit is higher than the unit at RM/server side. Because of this, old code in {{DominantResourceCalculator#normalize}} was working properly for the above junit test case. was (Author: manirajv06@gmail.com): [~sunilg] Fixed those 2 junit failures. Though {{TestFairSchedulerConfiguration}} changes has fixed the junit issue, requires you to check in detail because of following flow: {code} assertEquals(customResourceInformation(20000L, ""), - calculator.normalize(customResource(10001L, ""), min, max, increment) + calculator.normalize(customResource(19999L, ""), min, max, increment) .getResourceInformation(A_CUSTOM_RESOURCE)); {code} In the above junit test case, {{increment}} has been derived from {{FairSchedulerConfiguration#getIncrementAllocation}} which returns 10k as o/p. {code} conf.set(YarnConfiguration.RESOURCE_TYPES + ".a-custom-resource" + FairSchedulerConfiguration.INCREMENT_ALLOCATION, "10");{code}. Here, value has been configured as 10 with "" units, but {{ResourceUtils#createResourceTypesArray}} triggered from {{FairSchedulerConfiguration#getIncrementAllocation}} returns array of RI's containing A_CUSTOM_RESOURCE with value as 10k, as it simply copies the object and modifies only the value. Is this correct behaviour? Based on our earlier discussion, conversion should happen if there is units difference and source unit is higher than the unit at RM/server side. Because of this, old code DominantResourceCalculator#normalize was working properly for the above junit test case. > Normalize unit of resource objects in RM and avoid to do unit conversion in critical path > ----------------------------------------------------------------------------------------- > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager > Reporter: Wangda Tan > Assignee: Manikandan R > Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, YARN-7159.016.patch, YARN-7159.017.patch, YARN-7159.018.patch, YARN-7159.019.patch, YARN-7159.020.patch, YARN-7159.021.patch > > > Currently resource conversion could happen in critical code path when different unit is specified by client. This could impact performance and throughput of RM a lot. We should do unit normalization when resource passed to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org