hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SACHINGUPTA <sac...@datametica.com>
Subject Re: number of mappers allowed in a container in hadoop2
Date Wed, 15 Oct 2014 13:24:28 GMT
I have one more doubt i was reading this

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html

there is one property as

mapreduce.map.memory.mb 	= 2*1024 MB
mapreduce.reduce.memory.mb 	= 2 * 2 = 4*1024 MB


what are these properties mapreduce.map.memory.mb and 
mapreduce.reduce.memory.mb

On Wednesday 15 October 2014 06:17 PM, Shahab Yunus wrote:
> It cannot run more mappers (tasks) in parallel than the underlying 
> cores available. Just like it cannot run multiple mappers in parallel 
> if each mapper's (task's) memory requirements are greater than 
> allocated and available container size configured on each node.
>
> The links that I provided earlier...see the following section in that 
> one:
> Section:"Configuring YARN"
>
> Also this: 
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
> Section "1. YARN Concurrency (aka “What Happened to Slots?”)"
>
> This should help in putting things in perspective regarding how 
> resource allocation for each task, container and resources available 
> on the node relate to each other.
>
> Regards,
> Shahab
>
> On Wed, Oct 15, 2014 at 8:18 AM, SACHINGUPTA <sachin@datametica.com 
> <mailto:sachin@datametica.com>> wrote:
>
>     but Shahab if i have only 4 core machine then how yarn can run
>     more then 4 mappers in parallel
>     On Wednesday 15 October 2014 05:45 PM, Shahab Yunus wrote:
>>     It depends on memory settings as well, that how much you want to
>>     assign resources to each container. Then yarn will run as many
>>     mappers in parallel as possible.
>>
>>     See this:
>>     http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
>>     http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html
>>
>>     Regards,
>>     Shahab
>>
>>     On Wed, Oct 15, 2014 at 8:09 AM, SACHINGUPTA
>>     <sachin@datametica.com <mailto:sachin@datametica.com>> wrote:
>>
>>         Hi guys
>>
>>         I have situation in which i have machine with 4 processor and
>>         i have 5 containers so does it mean i can have only 4 mappers
>>         running parallely at a time
>>
>>         and number of mappers is not dependent on the number of
>>         containers in a machine then what is the use of container concept
>>
>>         sorry if i have asked anything obvious.
>>
>>         -- 
>>         Thanks
>>         Sachin Gupta
>>
>>
>
>     -- 
>     Thanks
>     Sachin Gupta
>
>

-- 
Thanks
Sachin Gupta


Mime
View raw message