hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bejoy.had...@gmail.com
Subject Re: Hadoop Installation Mappers setting
Date Thu, 23 May 2013 19:15:49 GMT
When you take a mapreduce tasks, you need CPU cycles to do the processing, not just memory.

So ideally based on the processor type(hyperthreaded or not) compute the available cores.
Then may be compute as, one core for each task slot.

Regards 
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Jitendra Yadav <jeetuyadav200890@gmail.com>
Date: Fri, 24 May 2013 00:26:29 
To: <user@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Re: Hadoop Installation Mappers setting

Hi,

Thanks for your clarification.

I have one more question.

How does cores factor influence slots calculation?

Thanks~

On 5/23/13, Amal G Jose <amalgjos@gmail.com> wrote:
> I am explaining it more.
> If your machine have 8 GB of memory.
> After reserving to Operating system and all other processes except
> tasktracker, you have 4 GB remaining(assume).
> The remaining process running is tasktracker.
> If the child jvm size is 200 MB,
> Then you can define a maximum slots of 4*1024 MB/ 200 MB
> Which is approximately 20.
> You can divide the slots into mapper and reducer slots as per your
> requirement.
> This is just an example that I explained based on my knowledge.
>
>
>
> On Thu, May 23, 2013 at 7:48 PM, <bejoy.hadoop@gmail.com> wrote:
>
>> **
>> Hi
>>
>> I assume the question is on how many slots.
>>
>> It dependents on
>> - the child/task jvm size and the available memory.
>> - available number of cores
>>
>>
>>
>> Your available memory for tasks is total memory - memory used for OS and
>> other services running on your box.
>>
>> Other services include non hadoop services as well as hadoop daemons.
>>
>> Divide the available memory with child jvm size and that would get the
>> max
>> num of slots.
>>
>> Also check whether sufficient number of cores are available as well.
>>
>>
>> Regards
>> Bejoy KS
>>
>> Sent from remote device, Please excuse typos
>> ------------------------------
>> *From: * Jitendra Yadav <jeetuyadav200890@gmail.com>
>> *Date: *Thu, 23 May 2013 18:10:38 +0530
>> *To: *<user@hadoop.apache.org>
>> *ReplyTo: * user@hadoop.apache.org
>> *Subject: *Hadoop Installation Mappers setting
>>
>> Hi,
>>
>> While installing hadoop cluster, how we can calculate the exact number of
>> mappers value.
>>
>>
>> Thanks~
>>
>
Mime
View raw message