hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7019) Ability for applications to notify YARN about container reuse
Date Tue, 15 Aug 2017 22:44:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127996#comment-16127996
] 

Arun Suresh edited comment on YARN-7019 at 8/15/17 10:43 PM:
-------------------------------------------------------------

[~jlowe], given that Container reuse, espescially Tez's implementation of it, is pretty much
NM agnostic (it is essentially a Tez AM - Tez container protocol). I was wondering if - instead
of notifying the RM of the how many times a container has been re-used, maybe a more general
way to solve this might be to introduce a *preempt-ability* score for a container. Initially,
all containers of the AM are equally preemptible, but once the AM has say 're-used' a container
certain number of times or perhaps decided to use the container for some best effort task,
it can lower the preemptability score of the Container at the RM in the next allocate call.
Thoughts ?


was (Author: asuresh):
[~jlowe], given that Container reuse, espescially Tez's implementation of it, is pretty much
NM agnostic. I was wondering if - instead of notifying the RM of the how many times a container
has been re-used, maybe a more general way to solve this might be to introduce a *preempt-ability*
score for a container. Initially, all containers of the AM are equally preemptible, but once
the AM has say 're-used' a container certain number of times or perhaps decided to use the
container for some best effort task, it can lower the preemptability score of the Container
at the RM in the next allocate call. Thoughts ?

> Ability for applications to notify YARN about container reuse
> -------------------------------------------------------------
>
>                 Key: YARN-7019
>                 URL: https://issues.apache.org/jira/browse/YARN-7019
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Jason Lowe
>
> During preemption calculations YARN can try to reduce the amount of work lost by considering
how long a container has been running.  However when an application framework like Tez reuses
a container across multiple tasks it changes the work lost calculation since the container
has essentially checkpointed between task assignments.  It would be nice if applications could
inform YARN when a container has been reused/checkpointed and therefore is a better candidate
for preemption wrt. lost work than other, younger containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message