hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wilfred Spiegelenburg (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8059) Resource type is ignored when FS decide to preempt
Date Fri, 19 Oct 2018 01:33:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656136#comment-16656136

Wilfred Spiegelenburg commented on YARN-8059:

Tank you [~snemeth]

Jenkins has not run but I assume that the tests are fixed. While applying the patch I got
6 trailing white space warnings, they should be fixed.

* Why are we treating the virtual cores and memory specific in the {{subtractFromNonNegative}}?
In {{subtractFrom}} we do not do this so why are we not mirroring it here.
* In FairScheduler you use {{clusterUtilization}} which is not the utilisation of the cluster
but the size of the cluster for that resource, please rename because that is really confusing
* In {{TestFairSchedulerPreemption}} please check the indents and line breaks (preemptHalfResourcesCustomResource,
takeAllResourcesOfCustomResource, TestFairSchedulerPreemption)

For the tests:
testGetMaxUtilizationOfAllocatedResourcesWithAllocatedResource: when I run it the memory and
the custom ""custom_res_1" are both fully allocated. That means this test also passes if I
am not using custom resources. Please fix that, you explicitly want to test the custom resources
based on the check in the code.
This also makes me wonder if some of the other tests are triggering of memory or custom resource
as the dominant resource.  A number of tests use the same {{takeAllResourcesOfCustomResource}}
which seems to be the problem because of the allocation request build.

Last nit: remove the empty line in the list of yarn server imports in {{TestFairSchedulerPreemption}}

> Resource type is ignored when FS decide to preempt
> --------------------------------------------------
>                 Key: YARN-8059
>                 URL: https://issues.apache.org/jira/browse/YARN-8059
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: Yufei Gu
>            Assignee: Szilard Nemeth
>            Priority: Major
>         Attachments: YARN-8059.001.patch, YARN-8059.002.patch, YARN-8059.003.patch, YARN-8059.004.patch
> Method Fairscheduler#shouldAttemptPreemption doesn't consider resources other than vcore
and memory. We may need to rethink it in the resource type scenario. cc [~miklos.szegedi@cloudera.com],
[~wilfreds] and [~snemeth].
> {code}
>     if (context.isPreemptionEnabled()) {
>       return (context.getPreemptionUtilizationThreshold() < Math.max(
>           (float) rootMetrics.getAllocatedMB() /
>               getClusterResource().getMemorySize(),
>           (float) rootMetrics.getAllocatedVirtualCores() /
>               getClusterResource().getVirtualCores()));
>     }
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message