hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weiwei Yang <cheersy...@hotmail.com>
Subject Re: [DISCUSS] Merge YARN-6592 to trunk
Date Fri, 26 Jan 2018 02:53:55 GMT
+1, thanks for getting to this milestone Arun.
I’ve done some basic validations on a 4 nodes cluster, with some general affinity/anti-affinty/cardinality
constraints, it worked. I’ve also reviewed the doc, it’s in good shape and very illustrative.

Thanks.

--
Weiwei

On 26 Jan 2018, 10:44 AM +0800, Sunil G <sunilg@apache.org>, wrote:
+1.

Thanks Arun.

I did manual testing for check affinity and anti-affinity features with placement allocator.
Also checked SLS to see any performance regression, and there are not much difference as Arun
mentioned.

Thanks all the folks for working on this. Kudos!

- Sunil


On Fri, Jan 26, 2018 at 5:16 AM Arun Suresh <asuresh@apache.org<mailto:asuresh@apache.org>>
wrote:
Hello yarn-dev@

We feel that the YARN-6592 dev branch mostly in shape to be merged into trunk. This branch
adds support for placing containers in YARN using rich placement constraints. For example,
this can be used by applications to co-locate containers on a node or rack (affinity constraint),
spread containers across nodes or racks (anti-affinity constraint), or even specify the maximum
number of containers on a node/rack (cardinality constraint).

We have integrated this feature into the Distributed-Shell application for feature testing.
We have performed end-to-end testing on moderately-sized clusters to verify that constraints
work fine. Performance tests have been done via both SLS tests and Capacity Scheduler performance
unit tests, and no regression was found. We have opened a JIRA to track Jenkins acceptance
of the aggregated patch [2]. Documentation is in the process of being completed [3]. You can
also check our design document for more details [4].

Config flags are needed to enable this feature and it should not have any effect on YARN when
turned off. Once merged, we plan to work on further improvements, which can be tracked in
the umbrella YARN-7812 [5].

Kindly do take a look at the branch and raise issues/concerns that need to be addressed before
the merge.

Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for their contributions
to this effort, as well as Subru, Chris, Carlo, and Vinod for their inputs and discussions.

Cheers
-Arun


[1] https://issues.apache.org/jira/browse/YARN-6592
[2] https://issues.apache.org/jira/browse/YARN-7792
[3] https://issues.apache.org/jira/browse/YARN-7780
[4] https://issues.apache.org/jira/secure/attachment/12867869/YARN-6592-Rich-Placement-Constraints-Design-V1.pdf
[5] https://issues.apache.org/jira/browse/YARN-7812

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message