hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
Date Mon, 25 Apr 2016 22:35:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257178#comment-15257178

Wangda Tan commented on YARN-4390:


Thanks for review:

bq. Looks like the approach is to loop over almost all containers several times on the cluster
for every preemption cycle (3 secs by default), to see whether some containers can be preempted
to make room for the reserved container on the same node. Will this cause too much overhead
in a large cluster where we have a large amount of containers ? A unit test may test the time
cost for this mega loop.

Will run benchmark test for this.

bq. Does this equal to node.getUnallocatedResource?
Yes it is, I cannot remember why I chose not to use it, will update.

bq. Insn't FifoCandidatesSelector the first selector and the selectedCandidates is empty ?
No, it could be second selector as well.
    if (selectCandidatesForResevedContainers) {
          new ReservedContainerCandidatesSelector(this));

    // initialize candidates preemption selection policies
        new FifoCandidatesSelector(this));

> Do surgical preemption based on reserved container in CapacityScheduler
> -----------------------------------------------------------------------
>                 Key: YARN-4390
>                 URL: https://issues.apache.org/jira/browse/YARN-4390
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>    Affects Versions: 3.0.0, 2.8.0, 2.7.3
>            Reporter: Eric Payne
>            Assignee: Wangda Tan
>         Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf, YARN-4390.1.patch,
YARN-4390.2.patch, YARN-4390.3.branch-2.patch, YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch
> There are multiple reasons why preemption could unnecessarily preempt containers. One
is that an app could be requesting a large container (say 8-GB), and the preemption monitor
could conceivably preempt multiple containers (say 8, 1-GB containers) in order to fill the
large container request. These smaller containers would then be rejected by the requesting
AM and potentially given right back to the preempted app.

This message was sent by Atlassian JIRA

View raw message