hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Suresh <asur...@apache.org>
Subject [DISCUSS] Merge YARN-6592 to trunk
Date Thu, 25 Jan 2018 23:46:39 GMT
Hello yarn-dev@

We feel that the YARN-6592 dev branch mostly in shape to be merged into
trunk. This branch adds support for placing containers in YARN using rich
placement constraints. For example, this can be used by applications to
co-locate containers on a node or rack (*affinity *constraint), spread
containers across nodes or racks (*anti-affinity* constraint), or even
specify the maximum number of containers on a node/rack (*cardinality *
constraint).

We have integrated this feature into the Distributed-Shell application for
feature testing. We have performed end-to-end testing on moderately-sized
clusters to verify that constraints work fine. Performance tests have been
done via both SLS tests and Capacity Scheduler performance unit tests, and
no regression was found. We have opened a JIRA to track Jenkins acceptance
of the aggregated patch [2]. Documentation is in the process of being
completed [3]. You can also check our design document for more details [4].

Config flags are needed to enable this feature and it should not have any
effect on YARN when turned off. Once merged, we plan to work on further
improvements, which can be tracked in the umbrella YARN-7812 [5].

Kindly do take a look at the branch and raise issues/concerns that need to
be addressed before the merge.

Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for
their contributions to this effort, as well as Subru, Chris, Carlo, and
Vinod for their inputs and discussions.

Cheers
-Arun


[1] https://issues.apache.org/jira/browse/YARN-6592
[2] https://issues.apache.org/jira/browse/YARN-7792
[3] https://issues.apache.org/jira/browse/YARN-7780
[4]
https://issues.apache.org/jira/secure/attachment/12867869/YARN-6592-Rich-Placement-Constraints-Design-V1.pdf
[5] https://issues.apache.org/jira/browse/YARN-7812

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message