hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7745) Allow DistributedShell to take a placement specification for containers it wants to launch
Date Fri, 12 Jan 2018 23:41:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324741#comment-16324741
] 

Arun Suresh edited comment on YARN-7745 at 1/12/18 11:40 PM:
-------------------------------------------------------------

Formal specification of the *placementspec*:

{noformat}
PlacementSpec  => "" | KeyVal;PlacementSpec
KeyVal         => SourceTag=Constraint
SourceTag      => String
Constraint     => NumContainers,ConstraintType,Scope,TargetTag | NumContainers,ConstraintType,Scope,TargetTag,MinCard,MaxCard
NumContainers  => int (number of containers)
ConstraintType => "in" | "notIn" | "cardinality"
Scope          => "NODE"|"RACK"
TargetTag      => String (Target Tag)
MinCard        => int (min cardinality - needed if ConstraintType == cardinality)
MaxCard        => int (max cardinality - needed if ConstraintType == cardinality)
{noformat}

Event though the PlacementConstraint API has support for a Set of source and target expressions,
to avoid complicating the distributed shell, we will support a single source and target expression
per constraint.
{code}
.. -placement_spec foo=3,notIn,NODE,foo;bar=5;in,RACK,blur;blur,6,cardinality,NODE,blur,0,3
{code}
would encode 3 constraints:
# foo : 3 containers with antiaffinity to each other at the node scope (no more than 1 container
per node)
# bar : 5 containers with affinity to a rack on which containers with tag "blur" are running
# blur : 6 containers with cardinality constraint, such that no more than 3 containers are
scheduled on a node. 

NOTE:
* since the placement spec contains the number of containers, it will override the *-num_containers*
option.
* placement specification works only for GUARANTEED containers for the time being.


was (Author: asuresh):
Formal specification of the *placementspec*:

{noformat}
PlacementSpec  => "" | KeyVal;PlacementSpec
KeyVal         => SourceTag=Constraint
SourceTag      => String
Constraint     => NumContainers,ConstraintType,Scope,TargetTag | NumContainers,ConstraintType,Scope,TargetTag,MinCard,MaxCard
NumContainers  => int (number of containers)
ConstraintType => "in" | "notIn" | "cardinality"
Scope          => "NODE"|"RACK"
TargetTag      => String (Target Tag)
MinCard        => int (min cardinality - needed if ConstraintType == cardinality)
MaxCard        => int (max cardinality - needed if ConstraintType == cardinality)
{noformat}

Event though the PlacementConstraint API has support for a Set of source and target expressions,
to avoid complicating the distributed shell, we will support a single source and target expression
per constraint.
{code}
.. -placement_spec foo=3,notIn,NODE,foo;bar=5;in,RACK,blur;blur,6,cardinality,NODE,blur,0,3
{code}
would encode 3 constraints:
# foo : 3 containers with antiaffinity to each other at the node scope (no more than 1 container
per node)
# bar : 5 containers with affinity to a rack on which containers with tag "blur" are running
# blur : 6 containers with cardinality constraint, such that no more than 3 containers are
scheduled on a node. 



> Allow DistributedShell to take a placement specification for containers it wants to launch
> ------------------------------------------------------------------------------------------
>
>                 Key: YARN-7745
>                 URL: https://issues.apache.org/jira/browse/YARN-7745
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>
> This is add a '-placement_spec' option to the distributed shell client. Where the user
can specify a string-ified specification for how it wants containers to be placed.
> For eg:
> {noformat}
> $ yarn org.apache.hadoop.yarn.applications.distributedshell.Client –jar \
> $YARN_DS/hadoop-yarn-applications-distributedshell-$YARN_VERSION.jar \
>  -shell_command sleep -shell_args 10 -placement_spec <placementspecification>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message