hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kawa <kawa.a...@gmail.com>
Subject Re: YARN creates only 1 container
Date Wed, 02 Jul 2014 22:27:37 GMT
You also might want to increase values for mapreduce.{map,reduce}.memory.mb
to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
-Xmx1024m).

mapreduce.{map,reduce}.memory.mb is logical size of the container and it
should be larger than mapreduce.{map,reduce}.java.opts that specifies the
heap for JVM running the actual task.


2014-06-27 5:43 GMT+02:00 Wangda Tan <wheeleast@gmail.com>:

> It should be yarn-site.xml not yarn.xml.
> yarn.xml will not be added to $CLASSPATH
>
> Thanks,
> Wangda
>
>
> On Wed, May 28, 2014 at 8:56 AM, hari <haribaha@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>>  yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb  = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <haribaha@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Mime
View raw message