hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6788) Improve performance of resource profile branch
Date Fri, 04 Aug 2017 08:14:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114043#comment-16114043
] 

Sunil G edited comment on YARN-6788 at 8/4/17 8:13 AM:
-------------------------------------------------------

Thanks [~templedf]

Quick clarification for few points:

bq.the findbugs warning is worth fixing
Findbugs warnings which are pointed by Jenkins is the one which is existing at YARN-3926.
I have fixed as part of this patch, hence there are no findbugs warning shown with patch (Please
refer *Patch Compile Tests* part in jenkins)

bq.don't forget to move TestResourceUtils into yarn-api
There were two fundamental reasons why it is kept in yarn-common
# ResourceUtils open files like {{resource-types.xml}} or any other files using {{ConfigurationProvider}}.
Default ConfigurationProvider class is {{org.apache.hadoop.yarn.LocalConfigurationProvider}}.
But this compiles with yarn-common package. Due to this, we can't compile TestResourceUtils
when it is in yarn-api since this package is first built as per hadoop-yarn pom (yarn-common
is built post yarn-api) 
# A bunch of sample resource files are also added as {{testResources}} in yarn-common pom.xml.
I can point to same dir from yarn-api pom or needed to copy/duplicate these resources. This
is something which we can do (point lookup to yarn-common resources for junit tests)

I think point 1 is little tricky and we can leave this test file in yarn-common for now. I
could add a comment and detail in this file for reference.

bq.checkstyle issue in ResourceUtils s
I could handle this in next patch. I guess I will wait for your comment for above point before
sharing next patch.


was (Author: sunilg):
Thanks [~templedf]

Quick clarification for few points:

bq.the findbugs warning is worth fixing
Current findbugs warnings pointed by Jenkins is the one which is existing at YARN-3926. I
have fixed as part of this patch, hence there are no findbugs warning shown with patch (Please
refer *Patch Compile Tests* part in jenkins)

bq.don't forget to move TestResourceUtils into yarn-api
There were two fundamental reason why its kept in yarn-common
# ResourceUtils open files such {{resource-types.xml}} or any other files using {{ConfigurationProvider}}.
Default ConfigurationProvider class is {{org.apache.hadoop.yarn.LocalConfigurationProvider}}.
But this compiles with yarn-common package. Due to this, we can't compile TestResourceUtils
when its in yarn-api since this package is first build as per hadoop-yarn pom (yarn-common
is built post yarn-api) 
# A bunch of sample resource files are added as {{testResources}} in yarn-common pom.xml.
I can hard point to same dir from yarn-api or need to copy/duplicate these resource. This
something doing (point lookup to yarn-common resources for junit tests)

I think point 1 is little tricky and we can leave this file in yarn-common for now. I could
add a comment and detail in this file for reference.

bq.checkstyle issue in ResourceUtils s
I could handle this in next patch. I guess I will wait for your comment for above point before
sharing next patch.

> Improve performance of resource profile branch
> ----------------------------------------------
>
>                 Key: YARN-6788
>                 URL: https://issues.apache.org/jira/browse/YARN-6788
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Sunil G
>            Assignee: Sunil G
>            Priority: Blocker
>         Attachments: YARN-6788-YARN-3926.001.patch, YARN-6788-YARN-3926.002.patch, YARN-6788-YARN-3926.003.patch,
YARN-6788-YARN-3926.004.patch, YARN-6788-YARN-3926.005.patch, YARN-6788-YARN-3926.006.patch,
YARN-6788-YARN-3926.007.patch, YARN-6788-YARN-3926.008.patch, YARN-6788-YARN-3926.009.patch,
YARN-6788-YARN-3926.010.patch, YARN-6788-YARN-3926.011.patch, YARN-6788-YARN-3926.012.patch,
YARN-6788-YARN-3926.013.patch, YARN-6788-YARN-3926.014.patch, YARN-6788-YARN-3926.015.patch,
YARN-6788-YARN-3926.016.patch, YARN-6788-YARN-3926.017.patch, YARN-6788-YARN-3926.018.patch,
YARN-6788-YARN-3926.019.patch, YARN-6788-YARN-3926.020.patch, YARN-6788-YARN-3926.021.patch,
YARN-6788-YARN-3926.022.patch, YARN-6788-YARN-3926.022.patch
>
>
> Currently we could see a 15% performance delta with this branch. 
> Few performance improvements to improve the same.
> Also this patch will handle [comments|https://issues.apache.org/jira/browse/YARN-6761?focusedCommentId=16075418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16075418]
from [~leftnoteasy].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message