hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Tustin <mtus...@handybook.com>
Subject Re: Yarn with CapacityScheduler will only schedule two applications - open to consultants today
Date Thu, 03 Mar 2016 04:33:39 GMT
Hi Naga,

You're quite right. Our fix was balance out the resources in each label
more, in order to have the bug bite us less. Beyond that I would direct any
readers to your text and my text below.

Marcin

On Wednesday, March 2, 2016, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:

> Hi Marcin,
>
> *we're seeing is that even though we have 12 machines in our yarn cluster,
> yarn will only schedule two applications. *
> If this is the problem you are facing with the below configuration then
> its mostly YARN-3216 <https://issues.apache.org/jira/browse/YARN-3216>
>  itself.
> And i think its not yet out in any released hadoop version yet (not sure
> about HDP) also not aware of the version  2.7.1.2.3 (may be HDP version
> number)
> IIUC you would have tried submitting in 2 diff partitions "data" and
> "yarn" hence 2 if not for a given partition only one would have run (if the
> am resource is less than the mimium size CS allows atleast 1 am to run)
>
> Given that you have already identified the issue, what more are you
> expecting ?
>
> Regards,
> + Naga
> ------------------------------
> *From:* Marcin Tustin [mtustin@handybook.com
> <javascript:_e(%7B%7D,'cvml','mtustin@handybook.com');>]
> *Sent:* Wednesday, March 02, 2016 20:09
> *To:* user@hadoop.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hadoop.apache.org');>
> *Subject:* Yarn with CapacityScheduler will only schedule two
> applications - open to consultants today
>
> Hi All,
>
> We're hitting this issue. If you're a consultant with capacity today (2
> March 2016 EST in New York) please feel free to contact me on or off list.
>
> In terms of stack, we're using Yarn 2.7.1.2.3 from the latest Hortonworks
> distribution. It's possible we're hitting this bug:
> https://issues.apache.org/jira/browse/YARN-3216
>
> The behaviour we're seeing is that even though we have 12 machines in our
> yarn cluster, yarn will only schedule two applications. The breakdown of
> machines is:
> 10 machines with the data label
> 2 with the yarn label
>
> We have three queues: interactive, noninteractive, and default.
> We're expecting to split capacity in the data label 20%/80% and default
> 100% of yarn label.
>
> We have the following capacity scheduler config (Key=value format taken
> from ambari):
>
> yarn.scheduler.capacity.maximum-am-resource-percent=100
> yarn.scheduler.capacity.maximum-applications=10000
> yarn.scheduler.capacity.node-locality-delay=40
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.root.accessible-node-labels=*
> yarn.scheduler.capacity.root.accessible-node-labels.data.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default-node-label-expression=data
> yarn.scheduler.capacity.root.default.accessible-node-labels=yarn
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.capacity=100
>
> yarn.scheduler.capacity.root.default.accessible-node-labels.yarn.maximum-capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=50
> yarn.scheduler.capacity.root.default.default-node-label-expression=yarn
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.default.maximum-capacity=100
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
>
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=100
> yarn.scheduler.capacity.root.interactive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.capacity=20
>
> yarn.scheduler.capacity.root.interactive.accessible-node-labels.data.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.acl_administer_queue=*
> yarn.scheduler.capacity.root.interactive.acl_submit_applications=*
> yarn.scheduler.capacity.root.interactive.capacity=10
> yarn.scheduler.capacity.root.interactive.maximum-am-resource-percent=50
> yarn.scheduler.capacity.root.interactive.maximum-applications=2000
> yarn.scheduler.capacity.root.interactive.maximum-capacity=100
> yarn.scheduler.capacity.root.interactive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.interactive.ordering-policy=fifo
> yarn.scheduler.capacity.root.interactive.state=RUNNING
> yarn.scheduler.capacity.root.interactive.user-limit-factor=100
> yarn.scheduler.capacity.root.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels=data
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.capacity=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-am-resource-percent=80
>
> yarn.scheduler.capacity.root.noninteractive.accessible-node-labels.data.maximum-capacity=80
> yarn.scheduler.capacity.root.noninteractive.acl_submit_applications=*
> yarn.scheduler.capacity.root.noninteractive.capacity=40
>
> yarn.scheduler.capacity.root.noninteractive.default-node-label-expression=data
> yarn.scheduler.capacity.root.noninteractive.maximum-am-resource-percent=80
> yarn.scheduler.capacity.root.noninteractive.maximum-applications=8000
> yarn.scheduler.capacity.root.noninteractive.maximum-capacity=100
> yarn.scheduler.capacity.root.noninteractive.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.noninteractive.ordering-policy=fair
>
> yarn.scheduler.capacity.root.noninteractive.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.noninteractive.state=RUNNING
> yarn.scheduler.capacity.root.noninteractive.user-limit-factor=100
> yarn.scheduler.capacity.root.queues=default,interactive,noninteractive
> yarn.scheduler.capacity.root.user-limit-factor=40
>
>
> Also yarn.resourcemanager.scheduler.class is
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.
>
> Any suggestions gratefully received.
>
> Marcin
>
> Want to work at Handy? Check out our culture deck and open roles
> <http://www.handy.com/careers>
> Latest news <http://www.handy.com/press> at Handy
> Handy just raised $50m
> <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/>
led
> by Fidelity
>
>

-- 
Want to work at Handy? Check out our culture deck and open roles 
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m 
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/>
led 
by Fidelity


Mime
View raw message