hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: number of mappers allowed in a container in hadoop2
Date Wed, 15 Oct 2014 13:36:16 GMT
Explanation here.

http://stackoverflow.com/questions/24070557/what-is-the-relation-between-mapreduce-map-memory-mb-and-mapred-map-child-jav
https://support.pivotal.io/hc/en-us/articles/201462036-Mapreduce-YARN-Memory-Parameters
http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/ClusterSetup.html
(scroll towards the end.)

Regards,
Shahab

On Wed, Oct 15, 2014 at 9:24 AM, SACHINGUPTA <sachin@datametica.com> wrote:

>  I have one more doubt i was reading this
>
>
> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html
>
> there is one property as
>
>   mapreduce.map.memory.mb  = 2*1024 MB
> mapreduce.reduce.memory.mb           = 2 * 2 = 4*1024 MB
> what are these properties mapreduce.map.memory.mb and
> mapreduce.reduce.memory.mb
>
> On Wednesday 15 October 2014 06:17 PM, Shahab Yunus wrote:
>
> It cannot run more mappers (tasks) in parallel than the underlying cores
> available. Just like it cannot run multiple mappers in parallel if each
> mapper's (task's) memory requirements are greater than allocated and
> available container size configured on each node.
>
>  The links that I provided earlier...see the following section in that
> one:
> Section:"Configuring YARN"
>
>  Also this:
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
> Section "1. YARN Concurrency (aka “What Happened to Slots?”)"
>
>  This should help in putting things in perspective regarding how resource
> allocation for each task, container and resources available on the node
> relate to each other.
>
>  Regards,
> Shahab
>
> On Wed, Oct 15, 2014 at 8:18 AM, SACHINGUPTA <sachin@datametica.com>
> wrote:
>
>>  but Shahab if i have only 4 core machine then how yarn can run more then
>> 4 mappers in parallel
>> On Wednesday 15 October 2014 05:45 PM, Shahab Yunus wrote:
>>
>> It depends on memory settings as well, that how much you want to assign
>> resources to each container. Then yarn will run as many mappers in parallel
>> as possible.
>>
>>  See this:
>> http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
>>
>> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html
>>
>>  Regards,
>> Shahab
>>
>> On Wed, Oct 15, 2014 at 8:09 AM, SACHINGUPTA <sachin@datametica.com>
>> wrote:
>>
>>> Hi guys
>>>
>>> I have situation in which i have machine with 4 processor and i have 5
>>> containers so does it mean i can have only 4 mappers running parallely at a
>>> time
>>>
>>> and number of mappers is not dependent on the number of containers in a
>>> machine then what is the use of container concept
>>>
>>> sorry if i have asked anything obvious.
>>>
>>> --
>>> Thanks
>>> Sachin Gupta
>>>
>>>
>>
>> --
>> Thanks
>> Sachin Gupta
>>
>>
>
> --
> Thanks
> Sachin Gupta
>
>

Mime
View raw message