hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Phulari <rphul...@yahoo-inc.com>
Subject Re: new to hadoop
Date Tue, 04 May 2010 23:12:04 GMT
You can configure (conf/hadoop-env.sh) configuration files on each node to specify -Xmx values.
You can use conf/mapred-site.xml to configure default mappers and reducers running on a node.

  <description>The default number of map tasks per job.
  Ignored when mapred.job.tracker is "local".

  <description>The default number of reduce tasks per job. Typically set to 99%
  of the cluster's reduce capacity, so that if a node fails the reduces can
  still be executed in a single wave.
  Ignored when mapred.job.tracker is "local".


On 5/4/10 3:54 PM, "jamborta" <jamborta@gmail.com> wrote:


I am tring to set up a small hadoop cluster with 6 machines. the problem I
have now is that if I set the memory allocated to a task low (e.g -Xmx512m)
the application does not run, if I set it higher some machines in the
cluster only have not got too much memory (1 or 2GB) and when the
computation gets intensive hadoop create so many jobs and send them to these
weaker machines, which brings the whole cluster down.
my question is whether it is possible to specify -Xmx for each machine in
the cluster and specify how many task can run on a machine. or what is the
optimal setting in this situation?

thanks for your help


View this message in context: http://old.nabble.com/new-to-hadoop-tp28454028p28454028.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message