hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Jambor <jambo...@googlemail.com>
Subject Re: new to hadoop
Date Tue, 04 May 2010 23:33:04 GMT
thank you. so what would be the optimal setting for mapred.map.tasks and 
mapred.reduce.tasks, say, on a dual-core machine?

Tom

On 05/05/2010 00:12, Ravi Phulari wrote:
> You can configure (conf/hadoop-env.sh) configuration files on each 
> node to specify --Xmx values.
> You can use conf/mapred-site.xml to configure default mappers and 
> reducers running on a node.
>
> <property>
> <name>mapred.map.tasks</name>
> <value>2</value>
> <description>The default number of map tasks per job.
>   Ignored when mapred.job.tracker is "local".
> </description>
> </property>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> <description>The default number of reduce tasks per job. Typically set 
> to 99%
>   of the cluster's reduce capacity, so that if a node fails the 
> reduces can
>   still be executed in a single wave.
>   Ignored when mapred.job.tracker is "local".
> </description>
> </property>
>
>
> -
> Ravi
>
> On 5/4/10 3:54 PM, "jamborta" <jamborta@gmail.com> wrote:
>
>
>
>     Hi,
>
>     I am tring to set up a small hadoop cluster with 6 machines. the
>     problem I
>     have now is that if I set the memory allocated to a task low (e.g
>     -Xmx512m)
>     the application does not run, if I set it higher some machines in the
>     cluster only have not got too much memory (1 or 2GB) and when the
>     computation gets intensive hadoop create so many jobs and send
>     them to these
>     weaker machines, which brings the whole cluster down.
>     my question is whether it is possible to specify -Xmx for each
>     machine in
>     the cluster and specify how many task can run on a machine. or
>     what is the
>     optimal setting in this situation?
>
>     thanks for your help
>
>     Tom
>
>     --
>     View this message in context:
>     http://old.nabble.com/new-to-hadoop-tp28454028p28454028.html
>     Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
>
> Ravi
> -- 
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message