hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
Date Tue, 05 Sep 2017 20:35:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154255#comment-16154255
] 

Jason Lowe commented on YARN-7136:
----------------------------------

bq. Do you think is it fine according to the perf report for 3+ resource types?

I don't think we could deploy it with 3+ resource types without addressing that performance
drop, but I could see it being fine for smaller clusters.  I'm OK with the 3+ performance
being fixed later, but please much sooner rather than later.

Thanks for updating the patch!  Comments:

The toString method now avoids iterating over the memory resource in the loop by starting
at index 2, but it still has the string compares for MEMORY_URI and "Mi" units which is now
unnecessary.

As I mentioned in the last review, I think we should just unroll the loop in calculateSharesForTwoMandatoryResouces.
 Since there are only two resources, we can leverage that fact to know that if dom == 0 then
sub == 1 and if dom == 1 then sub == 0.  Unrolling the loop then leads to something like this
which will have less compares overall than the original loop:
{code}
  private int calculateSharesForTwoMandatoryResources(
      ResourceInformation[] clusterRes, Resource first, Resource second,
      double[] firstShares, double[] secondShares) {
    ResourceInformation[] firstRes = first.getResources();
    ResourceInformation[] secondRes = second.getResources();
    firstShares[0] = calculateShare(clusterRes[0], firstRes[0]);
    secondShares[0] = calculateShare(clusterRes[0], secondRes[0]);
    firstShares[1] = calculateShare(clusterRes[1], firstRes[1]);
    secondShares[1] = calculateShare(clusterRes[1], secondRes[1]);

    int firstDom = 0;
    int firstSub = 1;
    if (firstShares[1] > firstShares[0]) {
      firstDom = 1;
      firstSub = 0;
    }
    int secondDom = 0;
    int secondSub = 1;
    if (secondShares[1] > secondShares[0]) {
      secondDom = 1;
      secondSub = 0;
    }

    if (firstShares[firstDom] > secondShares[secondDom]) {
      return 1;
    } else if (firstShares[firstDom] < secondShares[secondDom]) {
      return -1;
    } else if (firstShares[firstSub] > secondShares[secondSub]) {
      return 1;
    } else if (firstShares[firstSub] < secondShares[secondSub]) {
      return -1;
    } else {
      return 0;
    }
  }
{code}


> Additional Performance Improvement for Resource Profile Feature
> ---------------------------------------------------------------
>
>                 Key: YARN-7136
>                 URL: https://issues.apache.org/jira/browse/YARN-7136
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Critical
>         Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, YARN-7136.YARN-3926.002.patch,
YARN-7136.YARN-3926.003.patch, YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch,
YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch, YARN-7136.YARN-3926.008.patch
>
>
> This JIRA is plan to add following misc perf improvements:
> 1) Use final int in Resources/ResourceCalculator to cache #known-resource-types. (Significant
improvement).
> 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length every time.
(Significant improvement).
> 3) Avoid setUnit validation (which is a HashSet lookup) when initialize default Memory/VCores
ResourceInformation (Significant improvement).
> 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some improvement).
> 5) Removed readOnlyResources in BaseResource. (Minor improvement).
> 6) Removed enum: MandatoryResources, use final integer instead. (Minor improvement).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message