hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Tustin <mtus...@handybook.com>
Subject Yarn with CapacityScheduler will only schedule two applications - open to consultants today
Date Wed, 02 Mar 2016 14:39:28 GMT
Hi All,

We're hitting this issue. If you're a consultant with capacity today (2
March 2016 EST in New York) please feel free to contact me on or off list.

In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks
distribution. It's possible we're hitting this bug:
https://issues.apache.org/jira/browse/YARN-3216

The behaviour we're seeing is that even though we have 12 machines in our
yarn cluster, yarn will only schedule two applications. The breakdown of
machines is:
10 machines with the data label
2 with the yarn label

We have three queues: interactive, noninteractive, and default.
We're expecting to split capacity in the data label 20%/80% and default
100% of yarn label.

We have the following capacity scheduler config (Key=value format taken
from ambari):

yarn.scheduler.capacity.maximum-am-resource-percent=100
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default-node-label-expression=data
yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=50
yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=100
yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
yarn.scheduler.capacity.root.interactive.capacity=10
yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
yarn.scheduler.capacity.root.interactive.maximum-applications=2000
yarn.scheduler.capacity.root.interactive.maximum-capacity=100
yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
yarn.scheduler.capacity.root.interactive.state=RUNNING
yarn.scheduler.capacity.root.interactive.user-limit-factor=100
yarn.scheduler.capacity.root.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
yarn.scheduler.capacity.root.noninteractive.capacity=40
yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.noninteractive.state=RUNNING
yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
yarn.scheduler.capacity.root.user-limit-factor=40


Also yarn.resourcemanager.scheduler.class is
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.

Any suggestions gratefully received.

Marcin

-- 
Want to work at Handy? Check out our culture deck and open roles 
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m 
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/>
led 
by Fidelity


Mime
View raw message