hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: Heap Size question.
Date Wed, 01 Jun 2011 12:01:47 GMT
The values show you the maximum heap size and currently used heap of the job tracker, not running
jobs. Furthermore, the HADOOP_HEAPSIZE setting only sets the maximum heap for the daemons,
not the tasks in your job.

If you're getting OOMEs, you should add a setting to your mapred-site.xml file that looks
like this:

<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx1g</value>
</property>

-Joey

On Jun 1, 2011, at 7:35, Ken Williams <zoo9000@hotmail.com> wrote:

> 
> 
>  Hi All,
> 
>  I'm a bit confused about the values displayed on the 'jobtracker.jsp' page.
>  In particular, there's a section called 'Cluster Summary'.
> 
>  I'm running a small 4-machine Hadoop cluster, and when I point a web-browser 
>  at my master machine (http://master:50030/jobtracker.jsp) it displays,
> 
>               Cluster Summary (Heap Size is 15.5 MB / 1.89 GB)
> 
>   What exactly do these figures mean ?
> 
>   I know that the second figure (1.89 GB) is determined by the value of 
>   the HADOOP_HEAPSIZE variable set in  'conf/hadoop-env.sh'. What I'm not 
>   sure about is exactly what it means, or where the first value (15.5 MB) is determined
>   or what it means.
> 
>   I'm guessing the 1.89 GB is the amount of heap-memory allocated to Hadoop
>   on each machine in the cluster. (Correct ?)
> 
>   I have no idea what the 15.5 MB means or where it comes from. It never changes,
>   not even when a job is running, and I can't find any explanation in the documentation.

> 
>   This page, https://issues.apache.org/jira/browse/HADOOP-4435, seems to suggest
>   that the 15.5 MB should be the amount of heap memory currently in use but since
>   this value never changes - not even when a job is running and I refresh the page 
>   - I'm not convinced this is working.   
> 
>   I'm asking this question because I have a Mahout job which slowly comes to halt with
>   a lot of 'OutOfMemoryError: Java heap space' errors, before it is 'Killed'.
> 
>   I'm using Hadoop 0.20.2 and the latest Mahout snapshot version.
> 
>   Thanks for any help.
> 
>           Ken
> 
>                         

Mime
View raw message