hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MARCOS MEDRADO RUBINELLI <marc...@buscapecompany.com>
Subject Re: Adjusting tasktracker heap size?
Date Wed, 17 Apr 2013 13:11:28 GMT
Thank you for the replies. Thankfully, this cluster works with a fairly regular load, so it
shouldn't be too hard to fine-tune.


On 17-04-2013 09:23, Bejoy Ks wrote:
Hi Marcos,

You need to consider the slots based on the available memory

Available Memory = Total RAM - (Memory for OS + Memory for Hadoop Daemons like DN,TT + Memory
for other servicess if any running in that node)

Now you need to consider the generic MR jobs planned on your cluster. Say if your tasks need
1G of JVM to run gracefully, then

Possible number of slots = Available Memory / JVM size of each task

Now divide the slots between mappers and reducers.

On Mon, Apr 15, 2013 at 11:38 PM, Amal G Jose <amalgjos@gmail.com<mailto:amalgjos@gmail.com>>
It depends on the type of job that is frequently submitting.
RAM size of the machine.
Heap size of tasktracker= (mapslots+reduceslots)*jvm size
We can adjust this according to our requirement to fine tune our cluster.
This is my thought.

On Mon, Apr 15, 2013 at 4:40 PM, MARCOS MEDRADO RUBINELLI <marcosm@buscapecompany.com<mailto:marcosm@buscapecompany.com>>

I am currently tuning a cluster, and I haven't found much information on
what factors to consider while adjusting the heap size of tasktrackers.
Is it a direct multiple of the number of map+reduce slots? Is there
anything else I should consider?

Thank you,

View raw message