hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Murthy <...@hortonworks.com>
Subject Re: Scheduler Questions
Date Sat, 26 Nov 2011 22:20:41 GMT
> On Sat, Nov 26, 2011 at 3:56 AM, Patrick Wendell <pwendell@gmail.com> wrote:
>
>> Hey All,
>>
>> Two questions about the MR2 scheduler code, for anyone more familiar.
>>
>> - The return type of allocate() suggests that the AM will get some
>> instantaneous allocation of resources. Reading through the two current
>> schedulers, however, it looks like containers are only allocated after Node
>> heartbeats, and that allocate() will rarely return any useful allocation.
>> Is that correct or am I missing something?

Typically applications submit all resource requests up-front and then
get allocations in subsequent calls to allocate...

>>
>> - The SchedulerNode class maintains state for both a single "reserved"
>> container and several "allocated" containers. As far as I can tell only the
>> capacity scheduler uses the former, but it's not clear to me what the
>> reserved container is for. Is this to enforce hard limits on allocation
>> reserved for particular users?

Reserved containers are partial containers reserved while the
scheduler awaits for a larger resource request to be fulfilled. For
e.g. if a 8G container is required and only 4G is available,
schedulers can 'reserve' the 4G while waiting for the rest.

Reservation prevents starvation of larger resource requests, but
should be done carefully since it's essentially leaving resources
under-utilized.

The CS 'charges' upfront for the whole reservation (8G in the e.g.
above) to ensure reservations are 'expensive' - larger the requested
container, larger is the cost for reserving them (upfront).

hth,
Arun

Mime
View raw message