hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brendan W." <bw8...@gmail.com>
Subject Memory tuning for map-reduce jobs
Date Tue, 01 Mar 2011 16:35:45 GMT
 I haven't been able to answer this question in the documentation...

I want to up the memory allocation for the launched reduce tasks. I can see
no way to do this in the code (via a jobConf setting, for instance). So
instead I've gone to each node in my 10-node cluster and edited the
mapred-site.xml file to add the property

mapreduce.reduce.java.opts, with value -Xmx 2048M, then stopped and
restarted the cluster.

But now I have a few questions about this:

1. Now when I run a job, the Job Configuration lists two simultaneous
properties: mapreduce.reduce.java.opts, and mapred.child.java.opts, with
values 2048M and 200m, respectively. Which of these is taking priority? Is
the first overriding the second for reduce jobs (I hope)?
2. Is this child JVM allocation being carved out of HADOOP_HEAPSIZE? Or is
it separate?
3. All the documentation says that these child memory settings can be set
"at the job level" rather than at the cluster level. But I haven't been able
to see how. As I mentioned above, only way I've seen to do it is to edit ten
different mapred-site.xml files, stop, and then restart my cluster, which
certainly doesn't seem like job-level configuration. Am I missing a way to
do this via jobConf?

Thanks very much for any help.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message