hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Jambor <jambo...@googlemail.com>
Subject Re: new to hadoop
Date Tue, 04 May 2010 23:45:13 GMT
great. thank you. I'll set it up that way.

Tom

On 05/05/2010 00:37, Ravi Phulari wrote:
> How much RAM ?
> With 6-8GB RAM you can go for 4 mappers and 2 reducers (this is my 
> personal guess).
>
> -
> Ravi
>
> On 5/4/10 4:33 PM, "Tamas Jambor" <jamborta@googlemail.com> wrote:
>
>     thank you. so what would be the optimal setting for
>     mapred.map.tasks and mapred.reduce.tasks, say, on a dual-core machine?
>
>     Tom
>
>     On 05/05/2010 00:12, Ravi Phulari wrote:
>
>         Re: new to hadoop You can configure (conf/hadoop-env.sh)
>         configuration files on each node to specify --Xmx values.
>         You can use conf/mapred-site.xml to configure default mappers
>         and reducers running on a node.
>
>         <property>
>         <name>mapred.map.tasks</name>
>         <value>2</value>
>         <description>The default number of map tasks per job.
>           Ignored when mapred.job.tracker is "local".
>         </description>
>         </property>
>
>         <property>
>         <name>mapred.reduce.tasks</name>
>         <value>1</value>
>         <description>The default number of reduce tasks per job.
>         Typically set to 99%
>           of the cluster's reduce capacity, so that if a node fails
>         the reduces can
>           still be executed in a single wave.
>           Ignored when mapred.job.tracker is "local".
>         </description>
>         </property>
>
>
>         -
>         Ravi
>
>         On 5/4/10 3:54 PM, "jamborta" <jamborta@gmail.com> wrote:
>
>
>
>
>             Hi,
>
>             I am tring to set up a small hadoop cluster with 6
>             machines. the problem I
>             have now is that if I set the memory allocated to a task
>             low (e.g -Xmx512m)
>             the application does not run, if I set it higher some
>             machines in the
>             cluster only have not got too much memory (1 or 2GB) and
>             when the
>             computation gets intensive hadoop create so many jobs and
>             send them to these
>             weaker machines, which brings the whole cluster down.
>             my question is whether it is possible to specify -Xmx for
>             each machine in
>             the cluster and specify how many task can run on a
>             machine. or what is the
>             optimal setting in this situation?
>
>             thanks for your help
>
>             Tom
>
>             --
>             View this message in context:
>             http://old.nabble.com/new-to-hadoop-tp28454028p28454028.html
>             Sent from the Hadoop core-user mailing list archive at
>             Nabble.com.
>
>
>
>
>         Ravi
>
>
> Ravi
> -- 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message