hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikandan R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
Date Thu, 11 Jan 2018 17:18:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322572#comment-16322572

Manikandan R commented on YARN-7159:

[~sunilg] Fixed those 2 junit failures. Though {{TestFairSchedulerConfiguration}} changes
has fixed the junit issue, requires you to check in detail because of following flow:

       assertEquals(customResourceInformation(20000L, ""),
-          calculator.normalize(customResource(10001L, ""), min, max, increment)
+          calculator.normalize(customResource(19999L, ""), min, max, increment)

In the above junit test case, {{increment}} has been derived from {{FairSchedulerConfiguration#getIncrementAllocation}}
which returns 10k as o/p. {code}      conf.set(YarnConfiguration.RESOURCE_TYPES + ".a-custom-resource"
          FairSchedulerConfiguration.INCREMENT_ALLOCATION, "10");{code}. Here, value has been
configured as 10 with "" units, but {{ResourceUtils#createResourceTypesArray}} triggered from
{{FairSchedulerConfiguration#getIncrementAllocation}} returns array of RI's containing A_CUSTOM_RESOURCE
with value as 10k, as it simply copies the object and modifies only the value. Is this correct
behaviour? Based on our earlier discussion, conversion should happen if there is units difference
and source unit is higher than the unit at RM/server side. Because of this, old code DominantResourceCalculator#normalize
was working properly for the above junit test case.

> Normalize unit of resource objects in RM and avoid to do unit conversion in critical
> -----------------------------------------------------------------------------------------
>                 Key: YARN-7159
>                 URL: https://issues.apache.org/jira/browse/YARN-7159
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Manikandan R
>            Priority: Critical
>         Attachments: YARN-7159.001.patch, YARN-7159.002.patch, YARN-7159.003.patch, YARN-7159.004.patch,
YARN-7159.005.patch, YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, YARN-7159.009.patch,
YARN-7159.010.patch, YARN-7159.011.patch, YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch,
YARN-7159.016.patch, YARN-7159.017.patch, YARN-7159.018.patch, YARN-7159.019.patch, YARN-7159.020.patch,
> Currently resource conversion could happen in critical code path when different unit
is specified by client. This could impact performance and throughput of RM a lot. We should
do unit normalization when resource passed to RM and avoid expensive unit conversion every

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message