hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Hall <thattommyh...@gmail.com>
Subject Re: Enable Hyperthreading? / Typical M/R ratios and rules of thumb
Date Wed, 16 Nov 2011 12:27:27 GMT
Thanks Arun,

On Mon, Nov 14, 2011 at 4:34 AM, Arun Murthy <acm@hortonworks.com> wrote:
> How much RAM do you have?
>
> A good rule of thumb is to use 1-1.5G for maps and 2G per reduce
> (vmem). Ensure your OS has at least 2G of memory.
>
> Thus, with 24G and dual quad cores you should be at 8-10m/2r. Scale up
> if you have more memory.

Would you say RAM was the main factor? We currently have 1G heap per mapper.
We had heard multiples of 1 disk / 2 core / 4G were good with slightly
more slots for (mappers + reducers) than cores. Would you agree?
Can you speak to how we should use hyperthreading, can I treat them as
separate cores? (I know in virtualisation that the recommendation is
to disable it but for some other workloads you get 2x performance
improvement)


Tom

Mime
View raw message