hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Hadoop heap and other memory settings
Date Thu, 10 Mar 2011 03:58:44 GMT

On Wed, Mar 9, 2011 at 11:17 PM, Mark <static.void.dev@gmail.com> wrote:
> HADOOP_HEAPSIZE=8192 (in hadoop-env.sh)

HADOOP_HEAPSIZE affects _all_ the Hadoop daemons, and all of them
would be started with the specified value as their heap size. Your
JT/NN/TT/DN/SNN would be started with 8192M as their heapsize (which
might not be what you're really looking for).

It does not affect the launched Task JVMs, which is what you're
looking to change.

> mapred.child.java.opts=8192 (in mapred-site.xml)

This affects all launched Task JVMs (Mappers and Reducers together),
and is a per-job specified setting (i.e. set by the JobConf/Job or -D
parameters in streaming, etc.). If you want your mapred-site.xml to be
the final authority on that property, set it as <final>true</final> so
nothing else can override it. By default, it is set to 200M.

Also, know that it is applied _per_ launched JVM, so set it sensibly
(based on the total configured slots per tasktracker/total memory
available for use in the machine/etc.).

Harsh J

View raw message