hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlo Aldo Curino <carlo.cur...@gmail.com>
Subject Re: [DISCUSS] Merge YARN-6592 to trunk
Date Fri, 26 Jan 2018 09:34:56 GMT
Also I think this can help us close the gap (and surpass) Kubernetes for
complex services (at least for resource management)... It would be awesome
to have a compatibility layer so folks can run Kubernetes natives apps on a
yarn cluster.




On Jan 26, 2018 1:32 AM, "Carlo Aldo Curino" <carlo.curino@gmail.com> wrote:

+1. I didn't runs tests, but I like the design, and speaking with ops teams
that operate large clusters I hear this is a feature they think is going to
help a lot, so I am very supportive of this effort.

On Jan 25, 2018 7:08 PM, "Konstantinos Karanasos" <kkaranasos@gmail.com>
wrote:

> Thanks for starting the thread Arun, +1 from me too.
>
> Konstantinos
>
> On Thu, Jan 25, 2018 at 18:54 Weiwei Yang <cheersyang@hotmail.com> wrote:
>
>> +1, thanks for getting to this milestone Arun.
>> I’ve done some basic validations on a 4 nodes cluster, with some general
>> affinity/anti-affinty/cardinality constraints, it worked. I’ve also
>> reviewed the doc, it’s in good shape and very illustrative.
>>
>> Thanks.
>>
>> --
>> Weiwei
>>
>> On 26 Jan 2018, 10:44 AM +0800, Sunil G <sunilg@apache.org>, wrote:
>> +1.
>>
>> Thanks Arun.
>>
>> I did manual testing for check affinity and anti-affinity features with
>> placement allocator. Also checked SLS to see any performance regression,
>> and there are not much difference as Arun mentioned.
>>
>> Thanks all the folks for working on this. Kudos!
>>
>> - Sunil
>>
>>
>> On Fri, Jan 26, 2018 at 5:16 AM Arun Suresh <asuresh@apache.org<mailto:
>> asuresh@apache.org>> wrote:
>> Hello yarn-dev@
>>
>> We feel that the YARN-6592 dev branch mostly in shape to be merged into
>> trunk. This branch adds support for placing containers in YARN using rich
>> placement constraints. For example, this can be used by applications to
>> co-locate containers on a node or rack (affinity constraint), spread
>> containers across nodes or racks (anti-affinity constraint), or even
>> specify the maximum number of containers on a node/rack (cardinality
>> constraint).
>>
>> We have integrated this feature into the Distributed-Shell application
>> for feature testing. We have performed end-to-end testing on
>> moderately-sized clusters to verify that constraints work fine. Performance
>> tests have been done via both SLS tests and Capacity Scheduler performance
>> unit tests, and no regression was found. We have opened a JIRA to track
>> Jenkins acceptance of the aggregated patch [2]. Documentation is in the
>> process of being completed [3]. You can also check our design document for
>> more details [4].
>>
>> Config flags are needed to enable this feature and it should not have any
>> effect on YARN when turned off. Once merged, we plan to work on further
>> improvements, which can be tracked in the umbrella YARN-7812 [5].
>>
>> Kindly do take a look at the branch and raise issues/concerns that need
>> to be addressed before the merge.
>>
>> Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for
>> their contributions to this effort, as well as Subru, Chris, Carlo, and
>> Vinod for their inputs and discussions.
>>
>> Cheers
>> -Arun
>>
>>
>> [1] https://issues.apache.org/jira/browse/YARN-6592
>> [2] https://issues.apache.org/jira/browse/YARN-7792
>> [3] https://issues.apache.org/jira/browse/YARN-7780
>> [4] https://issues.apache.org/jira/secure/attachment/12867869/
>> YARN-6592-Rich-Placement-Constraints-Design-V1.pdf
>> [5] https://issues.apache.org/jira/browse/YARN-7812
>>
>> --
> Konstantinos
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message