hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitaliy Semochkin <vitaliy...@gmail.com>
Subject Re: how to set diffent VM parameters for mappers and reducers?
Date Tue, 05 Oct 2010 13:24:10 GMT
I'm using apache hadoop-0.20.2 - the recent version i found in maven
central repo.

Regards,
Vitaliy S

On Tue, Oct 5, 2010 at 5:02 PM, Michael Segel <michael_segel@hotmail.com> wrote:
>
> Hi,
>
> You don't say which version of Hadoop you are using.
> Going from memory, I believe in the CDH3 release from Cloudera, there are some specific
OPTs you can set in hadoop-env.sh.
>
> HTH
>
> -Mike
>
>
>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>> Subject: how to set diffent VM parameters for mappers and reducers?
>> From: vitaliy.se@gmail.com
>> To: common-user@hadoop.apache.org
>>
>> Hello,
>>
>>
>> I have mappers that do not need much ram but combiners and reducers need a lot.
>> Is it possible to set different VM parameters for mappers and reducers?
>>
>>
>>
>> PS Often I face interesting problem, on same set of data I
>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>> but it happens not all the time.
>> What could be cause of such behavior?
>> My personal opinion is that I have
>>
>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>> it should.
>>
>> Thanks in Advance,
>> Vitaliy S
>

Mime
View raw message