hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitaliy Semochkin <vitaliy...@gmail.com>
Subject Re: how to set diffent VM parameters for mappers and reducers?
Date Fri, 08 Oct 2010 09:56:45 GMT
>You can set mapred.child.java.opts in mapred-site.xml
it affects both Mappers and Reducers, while I need to specify jvm
options for mappers and reducers separately.

Regards,
Vitaliy S

On Tue, Oct 5, 2010 at 5:14 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> You can set mapred.child.java.opts in mapred-site.xml
>
> BTW, combiner can been run both in map side and reduce side
>
>
>
>
> On Tue, Oct 5, 2010 at 8:59 PM, Vitaliy Semochkin <vitaliy.se@gmail.com> wrote:
>> Hello,
>>
>>
>> I have mappers that do not need much ram but combiners and reducers need a lot.
>> Is it possible to set different VM parameters for mappers and reducers?
>>
>>
>>
>> PS Often I face interesting problem, on same set of data I
>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>> but it happens not all the time.
>> What could be cause of such behavior?
>> My personal opinion is that I have
>>
>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>> it should.
>>
>> Thanks in Advance,
>> Vitaliy S
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Mime
View raw message