hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7019) Ability for applications to notify YARN about container reuse
Date Wed, 16 Aug 2017 06:53:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128415#comment-16128415

Sunil G commented on YARN-7019:

In ideal cases, lowest priority containers are selected for preemption from an application
which in turn is selected based on (fifo + priority / fair ordering policy). These selected
containers of an application may just about to finish or have some critical data etc. So given
a container selection policy, it will be easier to manage these container selections as well
in long run.
As suggested here, along with container priority, *score* which is derived from reused/checkpointed
could give a better candidate selection. In YARN-3784, we were trying to send preemption timeout
to AMs, so could AM take an immediate action about checkpointing OR even send a feedback to
RM with alternative container to preempt instead of selected one?

> Ability for applications to notify YARN about container reuse
> -------------------------------------------------------------
>                 Key: YARN-7019
>                 URL: https://issues.apache.org/jira/browse/YARN-7019
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Jason Lowe
> During preemption calculations YARN can try to reduce the amount of work lost by considering
how long a container has been running.  However when an application framework like Tez reuses
a container across multiple tasks it changes the work lost calculation since the container
has essentially checkpointed between task assignments.  It would be nice if applications could
inform YARN when a container has been reused/checkpointed and therefore is a better candidate
for preemption wrt. lost work than other, younger containers.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message