hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Hall <thattommyh...@gmail.com>
Subject Re: Enable Hyperthreading? / Typical M/R ratios and rules of thumb
Date Wed, 16 Nov 2011 12:27:27 GMT
Thanks Arun,

On Mon, Nov 14, 2011 at 4:34 AM, Arun Murthy <acm@hortonworks.com> wrote:
> How much RAM do you have?
> A good rule of thumb is to use 1-1.5G for maps and 2G per reduce
> (vmem). Ensure your OS has at least 2G of memory.
> Thus, with 24G and dual quad cores you should be at 8-10m/2r. Scale up
> if you have more memory.

Would you say RAM was the main factor? We currently have 1G heap per mapper.
We had heard multiples of 1 disk / 2 core / 4G were good with slightly
more slots for (mappers + reducers) than cores. Would you agree?
Can you speak to how we should use hyperthreading, can I treat them as
separate cores? (I know in virtualisation that the recommendation is
to disable it but for some other workloads you get 2x performance


View raw message