hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: I've probably hit some system limits
Date Wed, 12 Aug 2009 22:23:59 GMT
On Wed, Aug 12, 2009 at 2:14 PM, Mayuran Yogarajah <
mayuran.yogarajah@casalemedia.com> wrote:

> Hello,
>
> Amandeep Khurana wrote:
>
>> So you are running 16 map tasks per node? Plus 2 reducers?
>>
>>
> Thats correct.
>
>> I think that's high. With 6gb RAM, you should be looking at around 2
>> map tasks plus 1 reducer...
>> I have 9 nodes with quad core + 8gb RAM and I run 2M+1R on each node..
>>
>>
>>
> I thought the number of maps should be set to 1/2 - 2 * number of cpus,
> thats why
> we set it so high.  Right now I've set:
> mapred.tasktracker.map.tasks.maximum = 16
> mapred.tasktracker.reduce.tasks.maximum = 16
>

Its 2*number of nodes
Moreover, its not only the CPU's, but also the RAM that matters.. Plus I/O..
Now, I'm not sure if you are I/O bound on this job or not, but thats also a
consideration.

Reduce the number to 2+1 and see how it goes. Once things work stably,
increase the mapper by 2 and see.. You'll have to try a few times before
you'll find out the optimal number for your setup.



>
> So the max mappers/reducers is 96/96.
>
>  How much heap size have you given your hadoop instance?
>>
>> Also, is there a lot of processing going on in the mappers and reducers?
>>
>>
>>
> Yes these are pretty intensive jobs.
>
> thanks,
> M
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message