hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"
Date Tue, 20 Jan 2015 21:27:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284446#comment-14284446

Jian Fang commented on YARN-1039:

The duration concept comes with a good intention, but what I really am afraid of is that it
could introduce a huge complex to YARN if it is not designed properly. 

First, there are so many moving parts under the hook for the estimation, for example, the
time of a 30 node cluster may be significantly different from the one of a 300 node cluster.
Getting into the measurement and estimation business is very much like walking into benchmark
comparison business,  which is very hard in reality. 

Secondly, the duration probably relies on hadoop customers to provide a proper value for it
if YARN is not smart enough to derive the value by itself, which could be impractical for
many customers. Remember that many hadoop users are not even developers. Many of them rely
on high level components such as pig and hive to run hadoop jobs. They probably don't know
or care about the estimation. 

As a result, at least, the duration should only be an enhancement if the value is provided.
YARN should still work properly without such a value.

> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be used by
a scheduler that would know not to host the service on a transient (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived containers
on the same node

This message was sent by Atlassian JIRA

View raw message