hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil Govind <sunil.gov...@gmail.com>
Subject Re: Max Application Master Resources with Queue Elasticity
Date Tue, 07 Feb 2017 05:25:31 GMT
Hello Benson

I could help to explain a little bit here.

maximum-am-resource-percent could be configured per-queue level (from next
release, it could be configure per node-label level as well). By default
10% is default, and hence 10% of queue's capacity could be used for running
AM resources. However due to elasticity, a queue could have resources above
its configured capacity. In that case, "Max Application Master Resources"
will be considering queue's max limit.

To answer your question, Yes. Ideally this resources is available for
running AM. However there could many other reasons by which this resource
may not be available for AM. To list a few, assume QueueA has 50% capacity
and 100% as its max-capacity. AM resource percentage is 20%. Cluster
resource is 20GB.
- Assume QueueB has taken over 15GB. And one app is running in QueueA with
1GB as AM resource. As per calculation 4GB could go to AM resource.
However, we need to wait till some resource are freed from QueueB or use
preemption.
- User limit. If user-limit-factor is <=1, then you may not be able to get
more resources for elasticity.

If you tune all params as per your scenario, and if there are enough
resources in cluster, you could avail this resource for AM.

Thanks
Sunil

On Tue, Feb 7, 2017 at 9:20 AM Benson Qiu <benson.qiu@salesforce.com> wrote:

> Hi,
>
> I noticed that "Max Application Master Resources" on the ResourceManager
> UI (/cluster/scheduler) takes into account queue elasticity.
>
> AMResourceLimit and userAMResourceLimit on the ResourceManager API
> (/ws/v1/cluster/scheduler) also takes into account queue elasticity.
>
> Are these AM resources always guaranteed? If a queue cannot grow because
> all of the other queues in the cluster are fully utilized, does the queue
> still have "Max Application Master Resources" available for AM containers?
>
> Thanks,
> Benson
>

Mime
View raw message