hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: Container size configuration
Date Wed, 19 Jun 2013 15:31:56 GMT
Sorry I am a bit behind on some of the changes that have happened to the
scheduler.  And I guess I am also behind on what has happened to the MR
AM.  I just looked at the MR AM code and it goes only off of priority when
assigning containers.  It also does a sanity check that the memory
allocated is large enough to meet the needs of the given task.

So I am happy to say I was wrong on this one :)

Sorry I wasted your time, and thanks for the help.

--Bobby

On 6/18/13 12:59 PM, "Alejandro Abdelnur" <tucu@cloudera.com> wrote:

>Bobby,
>
>With MAPREDUCE-5310 we removed normalization of resource request on the
>MRAM side. This was done because the normalization is an implementation
>detail of the RM scheduler.
>
>IMO, if this is a problem for the MRAM as you suggest, then we should fix
>the MRAM logic.
>
>Note this may happen only the MR job specifies memory requirements for its
>tasks that do not much with its normalize value.
>
>Thanks.
>
>
>
>On Tue, Jun 18, 2013 at 10:45 AM, Robert Evans <evans@yahoo-inc.com>
>wrote:
>
>> Even returning an over sized container can be very confusing for an
>> application.  The MR AM will not handle it correctly.  If it sees a
>> container returned that does not match exactly the priority and size it
>> expects, I believe that container is thrown away.  We had deadlocks in
>>the
>> past where it somehow used a reducer container for a mapper and then
>>never
>> updated the reducer count to request a new one.  It is best for now to
>>not
>> mix the two, and we need to lock down/fix the semantics of what happens
>>in
>> those situations for a scheduler.
>>
>> --Bobby
>>
>> On 6/18/13 12:13 AM, "Bikas Saha" <bikas@hortonworks.com> wrote:
>>
>> >I think the API allows different size requests at the same priority.
>>The
>> >implementation of the scheduler drops the size information and uses the
>> >last value set. We should probably at least change it to use the
>>largest
>> >value used so that users don't get containers that are too small for
>>them.
>> >YARN-847 tracks this.
>> >
>> >Bikas
>> >
>> >-----Original Message-----
>> >From: Robert Evans [mailto:evans@yahoo-inc.com]
>> >Sent: Friday, June 14, 2013 7:09 AM
>> >To: yarn-dev@hadoop.apache.org
>> >Subject: Re: Container size configuration
>> >
>> >Is this specifically for YARN?  If so, yes you can do this, MR does
>>this
>> >for Maps vs Reduces.  The API right now requires that the different
>>sized
>> >containers have a different priority.
>> >
>> >
>> 
>>http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-yarn/hadoop-yarn-site/W
>>r
>> >i
>> >tingYarnApplications.html
>> >
>> >Shows how to make a resource request. It also shows how to make a
>> >AllocateRequest.  If you put in multiple ResourceRequests into the
>> >AllocateRequest it will allocate both of them.  But remember that that
>>the
>> >priority needs to be different, and the priority determines the order
>>in
>> >which the containers will be allocated to your application.
>> >
>> >--Bobby
>> >
>> >On 6/13/13 10:41 AM, "Yuzhang Han" <yuzhanghan1982@gmail.com> wrote:
>> >
>> >>Hi,
>> >>
>> >>I am wondering if I can allocate different size of containers to the
>> >>tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 =
>>Task2
>> >>= 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks.
>> >>
>> >>Yuzhang
>>
>>
>
>
>-- 
>Alejandro


Mime
View raw message