hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6593) [API] Introduce Placement Constraint object
Date Thu, 20 Jul 2017 19:17:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095217#comment-16095217

Chris Douglas commented on YARN-6593:

bq. I found it is very important otherwise we cannot support complex placement request which
need to be updated.
Do you have specific use cases in mind? If there's a restricted form we could use to support
them (e.g., adjusting cardinality, as in your example), that would be easier for us to support
and for users to reason about. Since we don't have applications that use placement constraints
yet, it may be difficult for us to predict where they need to change during execution (if
at all).

bq. Regarding to semantics, I prefer to apply to all containers placed subsequently, this
is also the closest behavior of existing YARN. We just need to verify updated placement request
is still valid, probably we don't need to restricted to some parameters.
I don't have a clear definition of validity across placement requests, particularly preserving
it across a sequence of updates to the constraints. We could support relaxations of existing
constraints, probably. Still, updates also require the LRA scheduler to maintain lineage for
all its internal structures. A likely implementation will convert users' expressions to some
normal form, combine those with admin constraints, forecast future allocations, inject requests
into the scheduler, etc. Even if we could offer well-defined semantics for updates, the implementation
and maintenance cost could outweigh the marginal benefit to users. If the workarounds (like
submitting a new application or a new set of constraints) are easier to understand, that's
probably what users will prefer, anyway.

Placement constraint updates also compound the {{ResourceRequest}} problem you cited in YARN-6594.
Which epoch of the placement constraints applied to a container returned by the RM, and for
which RR? If a user's application isn't getting containers, how is that debugged? If someone
wants to reason about a group of constraints for a production cluster while applications change
clauses programmatically at runtime, then that analysis goes from difficult to intractable.

You guys are implementing it, but I'd push this to future work.

> [API] Introduce Placement Constraint object
> -------------------------------------------
>                 Key: YARN-6593
>                 URL: https://issues.apache.org/jira/browse/YARN-6593
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Konstantinos Karanasos
>            Assignee: Konstantinos Karanasos
>             Fix For: 3.0.0-alpha3
>         Attachments: YARN-6593.001.patch, YARN-6593.002.patch, YARN-6593.003.patch, YARN-6593.004.patch
> This JIRA introduces an object for defining placement constraints.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message