hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szilard Nemeth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8059) Resource type is ignored when FS decide to preempt
Date Fri, 19 Oct 2018 13:41:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656820#comment-16656820

Szilard Nemeth commented on YARN-8059:

Hi [~wilfreds]!
Thanks for your comments!
1. If you meant we need to zero the values for memory / cores within the same loop, I did
the fix for that in {{subtractFromNonNegative}}
2. Fixed the name of the variable, it was indeed confusing.
3. Fixed the indents and line breaks.

1. You are right, I debugged a bit and also had the memory fully utilizied. It was caused
by the {{RM_SCHEDULER_INCREMENT_ALLOCATION_MB}} scheduler config, I decreased its value to
100 so I could request 400mb * <the no. of containers> so that it haven't reached the
cluster's capacity.
This way, the test still passes but only because the custom resource utilization is reached
the cluster capacity of that resource.

2. I don't really get this. Do you mean to use resource requests with smaller amounts of memory
/ vcores to have the custom resource as the dominant?
What is the problem with the allocation request build? 

Uploaded a new patch that fixes most of your comments.


> Resource type is ignored when FS decide to preempt
> --------------------------------------------------
>                 Key: YARN-8059
>                 URL: https://issues.apache.org/jira/browse/YARN-8059
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: Yufei Gu
>            Assignee: Szilard Nemeth
>            Priority: Major
>         Attachments: YARN-8059.001.patch, YARN-8059.002.patch, YARN-8059.003.patch, YARN-8059.004.patch
> Method Fairscheduler#shouldAttemptPreemption doesn't consider resources other than vcore
and memory. We may need to rethink it in the resource type scenario. cc [~miklos.szegedi@cloudera.com],
[~wilfreds] and [~snemeth].
> {code}
>     if (context.isPreemptionEnabled()) {
>       return (context.getPreemptionUtilizationThreshold() < Math.max(
>           (float) rootMetrics.getAllocatedMB() /
>               getClusterResource().getMemorySize(),
>           (float) rootMetrics.getAllocatedVirtualCores() /
>               getClusterResource().getVirtualCores()));
>     }
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message