hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: Container size configuration
Date Thu, 13 Jun 2013 16:59:47 GMT
You mean different memory sizes for different reducers?  If you're willing
to mess with the MR Application Master code, you should be to do it there.
 Check out RMContainerAllocator.java and RMContainerRequestor.java.

-Sandy


On Thu, Jun 13, 2013 at 9:56 AM, Yuzhang Han <yuzhanghan1982@gmail.com>wrote:

>  Thank you Sandy.
>
> I am trying to do to some experiments with MapReduce on YARN, where some
> reducers gain larger processing tasks than the other reducers. I want to
> see that if I allocate more memory to the larger-task reducers, how the
> performance would be improved.
>
> Anyway, if I want to manually configure memory size for a particular
> reducer, how can I do it? In ApplicationMaster? Or this would not work at
> all?
>
> Thank you.
>
>
>
>
>
> On 6/13/2013 12:47 PM, Sandy Ryza wrote:
>
> Hi Yuzhang,
>
>  Moving this question to the Hadoop user list.
>
>  Are you using MapReduce or writing your own YARN application?  In
> MapReduce, all maps must request the same amount of memory and all reduces
> must request the same amount of memory.  It would be trivial to do this in
> your own YARN application.
>
>  -Sandy
>
>
> On Thu, Jun 13, 2013 at 8:41 AM, Yuzhang Han <yuzhanghan1982@gmail.com>wrote:
>
>> Hi,
>>
>> I am wondering if I can allocate different size of containers to the
>> tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = Task2 =
>> 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks.
>>
>> Yuzhang
>>
>
>
>

Mime
View raw message