hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitaliy Semochkin <vitaliy...@gmail.com>
Subject Re: how to set diffent VM parameters for mappers and reducers?
Date Thu, 07 Oct 2010 19:50:32 GMT
Hi,

I tried using  mapred.map.child.java.opts and mapred.reduce.child.java.opts
but looks like  hadoop-0.20.2  ingnores it.

On which version have you seen it working?

Regards,
Vitaliy S

On Tue, Oct 5, 2010 at 5:14 PM, Alejandro Abdelnur <tucu@cloudera.com> wrote:
> The following 2 properties should work:
>
> mapred.map.child.java.opts
> mapred.reduce.child.java.opts
>
> Alejandro
>
>
> On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <michael_segel@hotmail.com> wrote:
>>
>> Hi,
>>
>> You don't say which version of Hadoop you are using.
>> Going from memory, I believe in the CDH3 release from Cloudera, there are some specific
OPTs you can set in hadoop-env.sh.
>>
>> HTH
>>
>> -Mike
>>
>>
>>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>>> Subject: how to set diffent VM parameters for mappers and reducers?
>>> From: vitaliy.se@gmail.com
>>> To: common-user@hadoop.apache.org
>>>
>>> Hello,
>>>
>>>
>>> I have mappers that do not need much ram but combiners and reducers need a lot.
>>> Is it possible to set different VM parameters for mappers and reducers?
>>>
>>>
>>>
>>> PS Often I face interesting problem, on same set of data I
>>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>>> but it happens not all the time.
>>> What could be cause of such behavior?
>>> My personal opinion is that I have
>>>
>>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>>> it should.
>>>
>>> Thanks in Advance,
>>> Vitaliy S
>>
>

Mime
View raw message