hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bejoy Ks <bejoy.had...@gmail.com>
Subject Re: Unable to set the heap size on Amazon elastic mapreduce
Date Thu, 05 Apr 2012 17:44:53 GMT
Hey Shrish
        If you are looking to do it on a job level, you can achive the same
hadoop jar ........ -D mapred.map.java.opts=-Xmx1024m

Also it is not recommended to change this on a per job level, normally in
production environments this value is set in mapred-site.xml and is marked
final, so that it can't be overridden at per job level. Overriding at job
level can lead to out of memory issues, if not handled wisely at job level.


Bejoy KS

On Thu, Apr 5, 2012 at 10:12 PM, kasi subrahmanyam

> Hi Shirsh,
> You need to increase the heap size of the JVM for the changes you made for
> mapred.child to take effect
> These changes need to be made in the files of the hadoop configuration
> folder.
> In hadoop-env.sh increase the heap size to may be 2000.(I think this
> property by default will be commented.)
> Then in the mapred-default.xml change( in the src folder) overwrite the
> other properties like mapred.map.child.java.opts.
> I hope this helps.
> On Thu, Apr 5, 2012 at 1:26 PM, shrish bajpai <random_unkown@hotmail.co.uk
> > wrote:
>>   Hi,
>> I have tried the following combinations of bootstrap actions to increase
>> the heap size of my job but none of them seem to work:
>> --mapred-key-value mapred.child.java.opts=-Xmx1024m
>> --mapred-key-value mapred.child.ulimit=unlimited
>> --mapred-key-value mapred.map.child.java.opts=-Xmx1024m
>> --mapred-key-value mapred.map.child.ulimit=unlimited
>> -m mapred.map.child.java.opts=-Xmx1024m
>> -m mapred.map.child.ulimit=unlimited
>> -m mapred.child.java.opts=-Xmx1024m
>> -m mapred.child.ulimit=unlimited
>> What is the right syntax?
>> Thanks
>> Shrish

View raw message