hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil Govind <sunil.gov...@gmail.com>
Subject Re: maximum-am-resource-percent is insufficient to start a single application
Date Wed, 15 Jun 2016 06:23:51 GMT
Hi Philip

Higher maximum-am-resource-percent value (0~1) will help to allocate more
resource for your ApplicationMaster container of a yarn application (MR
Jobs here), but also depend on the capacity configured for the queue. You
have mentioned that there is only default queue here, so that wont be a
problem. Few questions:
    - How much is your total cluster resource size and how much of cluster
resource is used now ?
    - Is there any other application were running in cluster and whether it
was taking full cluster resource.? This is a possibility since you now gave
whole queue's capacity for AM containers.
    - Do you have multiple users in your cluster who runs applications
other that this hive job? If so,
yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent will have
impact on AM resource usage limit. I think you can double check this.


- Sunil

On Wed, Jun 15, 2016 at 8:47 AM Phillip Wu <phillip.wu@unsw.edu.au> wrote:

> Hi,
>
>
>
> I'm new to Hadoop and Hive.
>
>
>
> I'm using Hadoop 2.6.4 (binary I got from internet) & Hive 2.0.1 (binary I
> got from internet).
>
> I can create a database and table in hive.
>
>
>
> However when I try to insert a record into a previously created table I
> get:
>
> "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
> maximum-am-resource-percent is insufficient to start a single application
> in queue"
>
>
>
> yarn-site.xml
>
> <property>
>
>       <name>yarn.resourcemanager.scheduler.class</name>
>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
>
> </property>
>
>
>
> capacity-scheduler.xml
>
> <property>
>
>     <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>
>     <value>1.0</value>
>
>     <description>
>
>       Maximum percent of resources in the cluster which can be used to run
>
>       application masters i.e. controls number of concurrent running
>
>       applications.
>
>     </description>
>
>   </property>
>
>
>
> According to the documentation this means I have allocated 100% to my one
> and only default scheduler queue.
>
> [
> https://hadoop.apache.org/docs/r2.6.4/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
> ]
>
> "yarn.scheduler.capacity.maximum-am-resource-percent /
> yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent
>
> Maximum percent of resources in the cluster which can be used to run
> application masters - controls number of concurrent active applications.
>
> Limits on each queue are directly proportional to their queue capacities
> and user limits.
>
> Specified as a float - ie 0.5 = 50%. Default is 10%. This can be set for
> all queues with yarn.scheduler.capacity.maximum-am-resource-percent and can
> also be overridden on a per queue basis by setting
>
> yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent"
>
>
>
> Can someone tell me how to fix this?
>

Mime
View raw message