hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wheele...@gmail.com>
Subject Re: YARN creates only 1 container
Date Fri, 27 Jun 2014 03:43:06 GMT
It should be yarn-site.xml not yarn.xml.
yarn.xml will not be added to $CLASSPATH

Thanks,
Wangda

On Wed, May 28, 2014 at 8:56 AM, hari <haribaha@gmail.com> wrote:

> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
>  yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb  = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <haribaha@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>

Mime
View raw message