hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (Naga)" <garlanaganarasi...@huawei.com>
Subject RE: Yarn with CapacityScheduler will only schedule two applications - open to consultants today
Date Thu, 03 Mar 2016 04:21:20 GMT
Hi Marcin,

we're seeing is that even though we have 12 machines in our yarn cluster, yarn will only schedule
two applications.
If this is the problem you are facing with the below configuration then its mostly YARN-3216<https://issues.apache.org/jira/browse/YARN-3216>
itself.
And i think its not yet out in any released hadoop version yet (not sure about HDP) also not
aware of the version  2.7.1.2.3 (may be HDP version number)
IIUC you would have tried submitting in 2 diff partitions "data" and "yarn" hence 2 if not
for a given partition only one would have run (if the am resource is less than the mimium
size CS allows atleast 1 am to run)

Given that you have already identified the issue, what more are you expecting ?

Regards,
+ Naga
________________________________
From: Marcin Tustin [mtustin@handybook.com]
Sent: Wednesday, March 02, 2016 20:09
To: user@hadoop.apache.org
Subject: Yarn with CapacityScheduler will only schedule two applications - open to consultants
today

Hi All,

We're hitting this issue. If you're a consultant with capacity today (2 March 2016 EST in
New York) please feel free to contact me on or off list.

In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks distribution. It's
possible we're hitting this bug: https://issues.apache.org/jira/browse/YARN-3216

The behaviour we're seeing is that even though we have 12 machines in our yarn cluster, yarn
will only schedule two applications. The breakdown of machines is:
10 machines with the data label
2 with the yarn label

We have three queues: interactive, noninteractive, and default.
We're expecting to split capacity in the data label 20%/80% and default 100% of yarn label.

We have the following capacity scheduler config (Key=value format taken from ambari):

yarn.scheduler.capacity.maximum-am-resource-percent=100
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default-node-label-expression=data
yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=100
yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
yarn.scheduler.capacity.root.interactive.capacity=10
yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
yarn.scheduler.capacity.root.interactive.maximum-applications=2000
yarn.scheduler.capacity.root.interactive.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
yarn.scheduler.capacity.root.interactive.state=RUNNING
yarn.scheduler.capacity.root.interactive.user-limit-factor=100
yarn.scheduler.capacity.root.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
yarn.scheduler.capacity.root.noninteractive.capacity=40
yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.noninteractive.state=RUNNING
yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
yarn.scheduler.capacity.root.user-limit-factor=40


Also yarn.resourcemanager.scheduler.class is org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.

Any suggestions gratefully received.

Marcin

Want to work at Handy? Check out our culture deck and open roles<http://www.handy.com/careers>
Latest news<http://www.handy.com/press> at Handy
Handy just raised $50m<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/>
led by Fidelity

[http://marketing-email-assets.handybook.com/smalllogo.png]

Mime
View raw message