hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitaliy Semochkin <vitaliy...@gmail.com>
Subject Re: how to set diffent VM parameters for mappers and reducers?
Date Fri, 08 Oct 2010 09:58:21 GMT
I use new API and  hadoop-0.20.2

On which distribution have you seen it working?

Thanks in Advance,
Vitaliy S

On Fri, Oct 8, 2010 at 3:08 AM, Alejandro Abdelnur <tucu@cloudera.com> wrote:
> Never used those myself, always used the global one, but I knew they are
> there.
>
> Which Hadoop API are you using, the old one or the new one?
>
> Alejandro
>
> On Fri, Oct 8, 2010 at 3:50 AM, Vitaliy Semochkin <vitaliy.se@gmail.com>wrote:
>
>> Hi,
>>
>> I tried using  mapred.map.child.java.opts and mapred.reduce.child.java.opts
>> but looks like  hadoop-0.20.2  ingnores it.
>>
>> On which version have you seen it working?
>>
>> Regards,
>> Vitaliy S
>>
>> On Tue, Oct 5, 2010 at 5:14 PM, Alejandro Abdelnur <tucu@cloudera.com>
>> wrote:
>> > The following 2 properties should work:
>> >
>> > mapred.map.child.java.opts
>> > mapred.reduce.child.java.opts
>> >
>> > Alejandro
>> >
>> >
>> > On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <michael_segel@hotmail.com>
>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> You don't say which version of Hadoop you are using.
>> >> Going from memory, I believe in the CDH3 release from Cloudera, there
>> are some specific OPTs you can set in hadoop-env.sh.
>> >>
>> >> HTH
>> >>
>> >> -Mike
>> >>
>> >>
>> >>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>> >>> Subject: how to set diffent VM parameters for mappers and reducers?
>> >>> From: vitaliy.se@gmail.com
>> >>> To: common-user@hadoop.apache.org
>> >>>
>> >>> Hello,
>> >>>
>> >>>
>> >>> I have mappers that do not need much ram but combiners and reducers
>> need a lot.
>> >>> Is it possible to set different VM parameters for mappers and reducers?
>> >>>
>> >>>
>> >>>
>> >>> PS Often I face interesting problem, on same set of data I
>> >>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>> >>> but it happens not all the time.
>> >>> What could be cause of such behavior?
>> >>> My personal opinion is that I have
>> >>>
>> >>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>> >>> it should.
>> >>>
>> >>> Thanks in Advance,
>> >>> Vitaliy S
>> >>
>> >
>>
>

Mime
View raw message