hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikandan R (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
Date Thu, 11 Jan 2018 17:20:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322572#comment-16322572
] 

Manikandan R edited comment on YARN-7159 at 1/11/18 5:19 PM:
-------------------------------------------------------------

[~sunilg] Fixed those 2 junit failures. Though {{TestFairSchedulerConfiguration}} changes
has fixed the junit issue, requires you to check in detail because of following flow:

{code}
       assertEquals(customResourceInformation(20000L, ""),
-          calculator.normalize(customResource(10001L, ""), min, max, increment)
+          calculator.normalize(customResource(19999L, ""), min, max, increment)
             .getResourceInformation(A_CUSTOM_RESOURCE));
{code}

In the above junit test case, {{increment}} has been derived from {{FairSchedulerConfiguration#getIncrementAllocation}}
which returns 10k as o/p. {code}      conf.set(YarnConfiguration.RESOURCE_TYPES + ".a-custom-resource"
+
          FairSchedulerConfiguration.INCREMENT_ALLOCATION, "10");{code}. Here, value has been
configured as 10 with "" units, but {{ResourceUtils#createResourceTypesArray}} triggered from
{{FairSchedulerConfiguration#getIncrementAllocation}} returns array of RI's containing A_CUSTOM_RESOURCE
with value as 10k, as it simply copies the object and modifies only the value. Is this correct
behaviour? Based on our earlier discussion, conversion should happen if there is units difference
and source unit is higher than the unit at RM/server side. Because of this, old code in {{DominantResourceCalculator#normalize}}
was working properly for the above junit test case {{customResource(10001L, "")}}


was (Author: manirajv06@gmail.com):
[~sunilg] Fixed those 2 junit failures. Though {{TestFairSchedulerConfiguration}} changes
has fixed the junit issue, requires you to check in detail because of following flow:

{code}
       assertEquals(customResourceInformation(20000L, ""),
-          calculator.normalize(customResource(10001L, ""), min, max, increment)
+          calculator.normalize(customResource(19999L, ""), min, max, increment)
             .getResourceInformation(A_CUSTOM_RESOURCE));
{code}

In the above junit test case, {{increment}} has been derived from {{FairSchedulerConfiguration#getIncrementAllocation}}
which returns 10k as o/p. {code}      conf.set(YarnConfiguration.RESOURCE_TYPES + ".a-custom-resource"
+
          FairSchedulerConfiguration.INCREMENT_ALLOCATION, "10");{code}. Here, value has been
configured as 10 with "" units, but {{ResourceUtils#createResourceTypesArray}} triggered from
{{FairSchedulerConfiguration#getIncrementAllocation}} returns array of RI's containing A_CUSTOM_RESOURCE
with value as 10k, as it simply copies the object and modifies only the value. Is this correct
behaviour? Based on our earlier discussion, conversion should happen if there is units difference
and source unit is higher than the unit at RM/server side. Because of this, old code in {{DominantResourceCalculator#normalize}}
was working properly for the above junit test case.

> Normalize unit of resource objects in RM and avoid to do unit conversion in critical
path
> -----------------------------------------------------------------------------------------
>
>                 Key: YARN-7159
>                 URL: https://issues.apache.org/jira/browse/YARN-7159
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Manikandan R
>            Priority: Critical
>         Attachments: YARN-7159.001.patch, YARN-7159.002.patch, YARN-7159.003.patch, YARN-7159.004.patch,
YARN-7159.005.patch, YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, YARN-7159.009.patch,
YARN-7159.010.patch, YARN-7159.011.patch, YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch,
YARN-7159.016.patch, YARN-7159.017.patch, YARN-7159.018.patch, YARN-7159.019.patch, YARN-7159.020.patch,
YARN-7159.021.patch
>
>
> Currently resource conversion could happen in critical code path when different unit
is specified by client. This could impact performance and throughput of RM a lot. We should
do unit normalization when resource passed to RM and avoid expensive unit conversion every
time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message