hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
Date Sat, 11 Mar 2017 01:53:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905999#comment-15905999
] 

ASF GitHub Bot commented on YARN-5829:
--------------------------------------

GitHub user szegedim opened a pull request:

    https://github.com/apache/hadoop/pull/201

    YARN-5829 Reserve containers for preemptors

    YARN-5829 Reserve containers for preemptors.
    The change does the following.
    1. When a container is selected for preemption, it stores the preemptor next to the container.
    2. When a container is finally completed it creates a reservation for the preemptor application.
    3. On the next node update we check first reservations, then preemption reservations.
They are satisfied before any other request, even if those have higher priorities. This is
used tho avoid stealing preempted containers.
    4. It also disables standard reservations because I noticed that the preempted applications
reserve nodes for themselves stealing back the preempted resources.
    5. If a preemptor is deleted before the preemption or after the preemption, the assignment
won't happen at node update.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/szegedim/hadoop YARN-5829

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/hadoop/pull/201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #201
    
----
commit 07acc98178ea88bdd1621dc39e2aa28354be7f63
Author: Miklos Szegedi <miklos.szegedi@cloudera.com>
Date:   2017-03-10T04:50:08Z

    YARN-5829 Initial suggestion. Track preempted containers and create reservation, when
they free up

commit 7009aac04f0eb9301868b50c6e0c1720db3b92e2
Author: Miklos Szegedi <miklos.szegedi@cloudera.com>
Date:   2017-03-11T01:00:28Z

    YARN-5829 Added unit test, fixed concurrent exception bug with multiple preempted applications.

commit bbaa3a5980d1e25d2788c5f80539dfcdff500184
Author: Miklos Szegedi <miklos.szegedi@cloudera.com>
Date:   2017-03-11T01:42:15Z

    YARN-5829 Make sure deleted applications do not satisfy reservations.

----


> FS preemption should reserve a node before considering containers on it for preemption
> --------------------------------------------------------------------------------------
>
>                 Key: YARN-5829
>                 URL: https://issues.apache.org/jira/browse/YARN-5829
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>            Reporter: Karthik Kambatla
>            Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts identified containers.
If this node is not reserved for a specific application, any other application could be allocated
resources on this node. 
> Reserving the node for the starved application before preempting containers would help
avoid this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message