hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"
Date Tue, 24 Jun 2014 08:43:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041874#comment-14041874

Zhijie Shen commented on YARN-1039:

bq. I see. I'd assume that the "service" flag would imply long-lived, but maybe they could
be separated.

Just think it out loudly. Please correct me if I'm wrong or missing something.

"service" and "long-lived" overlap to some extent when describing an application, as we usually
think a service is going to run for a long time. However, IMHO, "service" should not be the
necessity for "long-lived". Theoretically, a MR job can be big enough to run for a long time
as well. We may want to differ the application with "service" from others by some of the applications'
native characteristics. For example, "progress" is not going to make sense to the applications
that are labeled "service", while we still want it for a MR job even if it runs for days,
don't we? Moreover, "service" sounds the application-level only property, and we won't mark
a single container as a "service".

On the other hand, "long-lived" is used to mark an application that is supposed to run for
long time. However, it can only indicate the application is likely to run for a long time,
but can not guarantee it will actually. I'm wondering if we really need to mark an application
"long-lived" when submission. Is it feasible to justify whether an application is "long-lived"
by how much time it has already spent in the cluster, and the "long-lived" application is
going to be handled properly in implicit way? For example, when we come to AM retry opportunities
(one issue for "long-lived" application), we can choose to refresh the "quota" given the application
is working well for a while. We don't need to rely on "long-lived" label. The reason that
I can think of why we must has this label upfront is that some special treatments for the
"long-lived" application should start from the beginning.

> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>            Priority: Minor
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be used by
a scheduler that would know not to host the service on a transient (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived containers
on the same node

This message was sent by Atlassian JIRA

View raw message