Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 4 Aug 2017 08:14:00 +0000 (UTC)
From: "Sunil G (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13085833.1499657170000.92111.1501834440259@Atlassian.JIRA>
In-Reply-To: <JIRA.13085833.1499657170000@Atlassian.JIRA>
References: <JIRA.13085833.1499657170000@Atlassian.JIRA> <JIRA.13085833.1499657170893@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (YARN-6788) Improve performance of resource
 profile branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 04 Aug 2017 08:14:07 -0000


    [ https://issues.apache.org/jira/browse/YARN-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114043#comment-16114043 ] 

Sunil G edited comment on YARN-6788 at 8/4/17 8:13 AM:
-------------------------------------------------------

Thanks [~templedf]

Quick clarification for few points:

bq.the findbugs warning is worth fixing
Findbugs warnings which are pointed by Jenkins is the one which is existing at YARN-3926. I have fixed as part of this patch, hence there are no findbugs warning shown with patch (Please refer *Patch Compile Tests* part in jenkins)

bq.don't forget to move TestResourceUtils into yarn-api
There were two fundamental reasons why it is kept in yarn-common
# ResourceUtils open files like {{resource-types.xml}} or any other files using {{ConfigurationProvider}}. Default ConfigurationProvider class is {{org.apache.hadoop.yarn.LocalConfigurationProvider}}. But this compiles with yarn-common package. Due to this, we can't compile TestResourceUtils when it is in yarn-api since this package is first built as per hadoop-yarn pom (yarn-common is built post yarn-api) 
# A bunch of sample resource files are also added as {{testResources}} in yarn-common pom.xml. I can point to same dir from yarn-api pom or needed to copy/duplicate these resources. This is something which we can do (point lookup to yarn-common resources for junit tests)

I think point 1 is little tricky and we can leave this test file in yarn-common for now. I could add a comment and detail in this file for reference.

bq.checkstyle issue in ResourceUtils s
I could handle this in next patch. I guess I will wait for your comment for above point before sharing next patch.


was (Author: sunilg):
Thanks [~templedf]

Quick clarification for few points:

bq.the findbugs warning is worth fixing
Current findbugs warnings pointed by Jenkins is the one which is existing at YARN-3926. I have fixed as part of this patch, hence there are no findbugs warning shown with patch (Please refer *Patch Compile Tests* part in jenkins)

bq.don't forget to move TestResourceUtils into yarn-api
There were two fundamental reason why its kept in yarn-common
# ResourceUtils open files such {{resource-types.xml}} or any other files using {{ConfigurationProvider}}. Default ConfigurationProvider class is {{org.apache.hadoop.yarn.LocalConfigurationProvider}}. But this compiles with yarn-common package. Due to this, we can't compile TestResourceUtils when its in yarn-api since this package is first build as per hadoop-yarn pom (yarn-common is built post yarn-api) 
# A bunch of sample resource files are added as {{testResources}} in yarn-common pom.xml. I can hard point to same dir from yarn-api or need to copy/duplicate these resource. This something doing (point lookup to yarn-common resources for junit tests)

I think point 1 is little tricky and we can leave this file in yarn-common for now. I could add a comment and detail in this file for reference.

bq.checkstyle issue in ResourceUtils s
I could handle this in next patch. I guess I will wait for your comment for above point before sharing next patch.

> Improve performance of resource profile branch
> ----------------------------------------------
>
>                 Key: YARN-6788
>                 URL: https://issues.apache.org/jira/browse/YARN-6788
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Sunil G
>            Assignee: Sunil G
>            Priority: Blocker
>         Attachments: YARN-6788-YARN-3926.001.patch, YARN-6788-YARN-3926.002.patch, YARN-6788-YARN-3926.003.patch, YARN-6788-YARN-3926.004.patch, YARN-6788-YARN-3926.005.patch, YARN-6788-YARN-3926.006.patch, YARN-6788-YARN-3926.007.patch, YARN-6788-YARN-3926.008.patch, YARN-6788-YARN-3926.009.patch, YARN-6788-YARN-3926.010.patch, YARN-6788-YARN-3926.011.patch, YARN-6788-YARN-3926.012.patch, YARN-6788-YARN-3926.013.patch, YARN-6788-YARN-3926.014.patch, YARN-6788-YARN-3926.015.patch, YARN-6788-YARN-3926.016.patch, YARN-6788-YARN-3926.017.patch, YARN-6788-YARN-3926.018.patch, YARN-6788-YARN-3926.019.patch, YARN-6788-YARN-3926.020.patch, YARN-6788-YARN-3926.021.patch, YARN-6788-YARN-3926.022.patch, YARN-6788-YARN-3926.022.patch
>
>
> Currently we could see a 15% performance delta with this branch. 
> Few performance improvements to improve the same.
> Also this patch will handle [comments|https://issues.apache.org/jira/browse/YARN-6761?focusedCommentId=16075418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16075418] from [~leftnoteasy].


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org