hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Murthy <...@hortonworks.com>
Subject Re: Enable Hyperthreading? / Typical M/R ratios and rules of thumb
Date Mon, 14 Nov 2011 04:34:42 GMT
How much RAM do you have?

A good rule of thumb is to use 1-1.5G for maps and 2G per reduce
(vmem). Ensure your OS has at least 2G of memory.

Thus, with 24G and dual quad cores you should be at 8-10m/2r. Scale up
if you have more memory.

Also, ensure you turn on memory monitoring so that a rouge task won't
overrun a node and cause your TT/DN to swap... that would very bad for
the cluster.

hth,
Arun

Sent from my iPhone

On Nov 13, 2011, at 6:57 PM, Tom Hall <thattommyhall@gmail.com> wrote:

> Hi All,
>
> We have been using dual socket quad core machines for a while and have
> been running with 8 mappers 2 reducers. The rule of thumb we heard was
> slightly oversubscribe the number of cores but at Hadoop World several
> people said other things.
> Some new machines we are moving to have 2 socket 6 core machines with
> hyperthreading (24 'cores' if you count each hyperthreaded one as 2)
>
> What are people doing for M/R ratios and number of M+R per machine at
> the moment?
>
> Thanks,
> Tom

Mime
View raw message