hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com>
Subject Re: Capacity scheduler puts all containers on one box
Date Sat, 22 Mar 2014 06:35:57 GMT
Hey Guys,

@Vinod: We aren't overriding the default, so we must be using -1 as the
setting.

@Sandy: We aren't specifying any racks/hosts when sending the resource
requests. +1 regarding introducing a similar limit in capacity scheduler.

Any recommended work-arounds in the mean time? Our utilization of the grid
is very low because we're having to force high memory requests for the
containers in order to guarantee a maximum number of containers on a
single node (e.g. Set container memory MB set to 17GB to disallow more
than 2 containers from being assigned to any one 48GB node).

Cheers,
Chris

On 3/21/14 11:30 PM, "Sandy Ryza" <sandy.ryza@cloudera.com> wrote:

>yarn.scheduler.capacity.node-locality-delay will help if the app is
>requesting containers at particular locations, but won't help spread
>things
>out evenly otherwise.
>
>The Fair Scheduler attempts an even spread.  By default, it only schedules
>a single container each time it considers a node.  Decoupling scheduling
>from node heartbeats (YARN-1010) makes it so that a high node heartbeat
>interval doesn't result in this being slow.  Now that the Capacity
>Scheduler has similar capabilities (YARN-1512), it might make sense to
>introduce a similar limit?
>
>-Sandy
>
>
>On Fri, Mar 21, 2014 at 4:42 PM, Vinod Kumar Vavilapalli
><vinodkv@apache.org
>> wrote:
>
>> What's the value for yarn.scheduler.capacity.node-locality-delay? It is
>>-1
>> by default in 2.2.
>>
>> We fixed the default to be a reasonable 40 (nodes in a rack) in 2.3.0
>>that
>> should spread containers a bit.
>>
>> Thanks,
>> +Vinod
>>
>> On Mar 21, 2014, at 12:48 PM, Chris Riccomini <criccomini@linkedin.com>
>> wrote:
>>
>> > Hey Guys,
>> >
>> > We're running YARN 2.2 with the capacity scheduler. Each NM is running
>> with 40G of memory capacity. When we request a series containers with
>>2G of
>> memory from a single AM, we see the RM assigning them entirely to one NM
>> until that NM is full, and then moving on to the next, and so on.
>> Essentially, we have a grid with 20 nodes, and two are completely full,
>>and
>> the rest are completely empty. This is problematic because our
>>containers
>> use disk heavily, and are completely saturating the disks on the two
>>nodes,
>> which slows all of the containers down on these NMs.
>> >
>> >  1.  Is this expected behavior of the capacity scheduler? What about
>>the
>> fifo scheduler?
>> >  2.  Is the recommended work around just to increase memory allocation
>> per-container as a proxy for the disk capacity that's required? Given
>>that
>> there's no disk-level isolation, and no disk-level resource, I don't see
>> another way around this.
>> >
>> > Cheers,
>> > Chris
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or
>>entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the
>>reader
>> of this message is not the intended recipient, you are hereby notified
>>that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>>immediately
>> and delete it from your system. Thank You.
>>


Mime
View raw message