hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Murthy <...@hortonworks.com>
Subject Re: In Yarn how to increase the number of concurrent applications for a queue
Date Mon, 08 Sep 2014 10:57:33 GMT
How many nodes do you have in your cluster?

Also, could you share the CapacityScheduler initialization logs for each
queue, such as:

2014-08-14 15:14:23,835 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Initialized queue: unfunded: capacity=0.5, absoluteCapacity=0.5,
usedResources=<memory:0, vCores:0>, usedCapacity=0.0,
absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-08-14 15:14:23,840 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Initializing default
capacity = 0.5 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.5 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 5000 [= configuredMaximumSystemApplicationsPerQueue or
(int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 5000 [= (int)(maxApplications * (userLimit /
100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
(userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.87506104 [= (float)(maximumAllocationMemory -
minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = SUBMIT_APPLICATIONS: ADMINISTER_QUEUE:  [= configuredAcls ]
nodeLocalityDelay = 0


Then, look at values for maxActiveAppsUsingAbsCap &
maxActiveApplicationsPerUser. That should help debugging.

thanks,
Arun


On Sun, Sep 7, 2014 at 9:37 AM, Anfernee Xu <anfernee.xu@gmail.com> wrote:

> Hi,
>
> I'rm running my cluster at Hadoop 2.2.0,  and use CapacityScheduler. And
> all my jobs are uberized and running among 2 queues, one queue takes
> majority of capacity(90%), another take 10%. What I found is for small
> queue, only one job is running for a given time, I tried twisting below
> properties, but no luck so far, could you guys share some light on this?
>
>  <property>
>     <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>     <value>1.0</value>
>     <description>
>       Maximum percent of resources in the cluster which can be used to run
>       application masters i.e. controls number of concurrent running
>       applications.
>     </description>
>   </property>
>
>
>  <property>
>     <name>yarn.scheduler.capacity.root.queues</name>
>     <value>default,small</value>
>     <description>
>       The queues at the this level (root is the root queue).
>     </description>
>   </property>
>
>  <property>
>
> <name>yarn.scheduler.capacity.root.small.maximum-am-resource-percent</name>
>     <value>1.0</value>
>   </property>
>
>
>  <property>
>     <name>yarn.scheduler.capacity.root.small.user-limit</name>
>     <value>1</value>
>   </property>
>
>  <property>
>     <name>yarn.scheduler.capacity.root.default.capacity</name>
>     <value>88</value>
>     <description>Default queue target capacity.</description>
>   </property>
>
>
>   <property>
>     <name>yarn.scheduler.capacity.root.small.capacity</name>
>     <value>12</value>
>     <description>Default queue target capacity.</description>
>   </property>
>
>  <property>
>     <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
>     <value>88</value>
>     <description>
>       The maximum capacity of the default queue.
>     </description>
>   </property>
>
>   <property>
>     <name>yarn.scheduler.capacity.root.small.maximum-capacity</name>
>     <value>12</value>
>     <description>Maximum queue capacity.</description>
>   </property>
>
>
> Thanks
>
> --
> --Anfernee
>



-- 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message