Hi Andy,
 
This is a client side property that you can set in hadoop-env.sh (/etc/hadoop/conf). The hadoop script (/bin/hadoop)
itself, which is a bash shell script, contains code like the following.
 
if [ "$HADOOP_HEAPSIZE" != "" ]; then
  #echo "run with heapsize $HADOOP_HEAPSIZE"
  JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
  #echo $JAVA_HEAP_MAX
fi
 
Use hadoop-env.sh.
 
Best,
Bjorn


On Fri, Apr 18, 2014 at 7:20 PM, Andy Srine <andy.srine@gmail.com> wrote:
I guess then the question is, how do I set the (system?) property and what property exactly is this? 

"export HADOOP_HEAPSIZE=4096" - Works from the environment.

And none of the following seem to work from the command line.
-Xmx4G

-Dmapred.child.java.opts=-Xmx4G

-DHADOOP_HEAPSIZE=4096

-DHADOOP_OPTS=-Xmx4G


I tried this from Java code as well and no luck.

System.setProperty("HADOOP_HEAPSIZE", "4096");


What am I missing?


Thanks,

Andy




On Fri, Apr 18, 2014 at 11:28 AM, Shahab Yunus <shahab.yunus@gmail.com> wrote:
You can pass hadoop conf properties through the -D option. Have you seen this?

This is not for system properties. The assumption is that you want to specify hadoop conf property through the command line.

Regards,
Shahab


On Fri, Apr 18, 2014 at 2:12 PM, Andy Srine <andy.srine@gmail.com> wrote:
Thanks for the tip. May be it's a different setting I need. Let me explain the problem. 

I get an heap error on the command line - "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space". It woks when I set "export HADOOP_HEAPSIZE=4096" in my env. But how can I set it via the hadoop command line (hadoop jar ...)? Next choice would be to do it programmatically in Java using hadoop conf. I can't really change the XML files across the cluster. 

Regards,
Andy


On Thu, Apr 17, 2014 at 6:39 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
Do you want add "-Xmx4g" to your MR tasks?  if so, just add it as "mapred.child.java.opts" in the mapred-site.xml


On Fri, Apr 18, 2014 at 9:35 AM, Andy Srine <andy.srine@gmail.com> wrote:
Quick question. How would I pass the following JVM option to the Hadoop command line?

"-Xmx4G"

hadoop jar <jar file> <class_name> <args>


Thanks,
Andy





--
Thanks,
Andy





--
Thanks,
Andy