hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ed <hadoopn...@gmail.com>
Subject Re: Set number Reducer per machines.
Date Tue, 05 Oct 2010 15:50:10 GMT
Hi Pramod,

How much memory does each node in your cluster have?

What type of processors do those nodes have? (dual core, quad core, dual
quad core? etc..)

In what step are you seeing the heap space error (mapper or reducer?)

It's quite possible that you're mapper or reducer code could be improved to
reduce heap space usage.

~Ed

On Tue, Oct 5, 2010 at 10:05 AM, Marcos Medrado Rubinelli <
marcosm@buscape-inc.com> wrote:

> You can set the mapred.tasktracker.map.tasks.maximum and
> mapred.tasktracker.reduce.tasks.maximum properties in your mapred-site.xml
> file, but you may also want to check your current mapred.child.java.opts and
> mapred.child.ulimit values to make sure they aren't overriding the 4GB you
> set globally.
>
> Cheers,
> Marcos
>
>  Hi,
>>
>> I am trying to run a job on my hadoop cluster, where I get consistently
>> get
>> heap space error.
>>
>> I increased the heap-space to 4 GB in hadoop-env.sh and reboot the
>> cluster.
>> However, I still get the heap space error.
>>
>>
>> One of things, I want to try is to reduce the number of map / reduce
>> process
>> per machine. Currently each machine can have 2 maps and 2 reduce process
>> running.
>>
>>
>> I want to configure the hadoop to run 1 map and 1 reduce per machine to
>> give
>> more heap space per process.
>>
>> How can I configure the number of maps and number of reducer per node ?
>>
>>
>> thanks in advance,
>> -- Pramod
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message