hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabio <anyte...@gmail.com>
Subject Questions about Capacity scheduler behavior
Date Mon, 03 Nov 2014 14:30:43 GMT
Hi guys, I'm posting this in the user mailing list since I got no reply 
in the Yarn-dev. I have to model as well as possible the capacity 
scheduler behavior and I have some questions, hope someone can help me 
with this. In the following I will consider all containers to be equal 
for simplicity:

1) In case of multiple queues having the same level of assigned 
resources, what's the policy to decide which comes first in the resource 

2) Let's consider this configuration:
We have a cluster hosting a total of 40 containers. We have 3 queues: A 
is configured to get 39% of cluster capacity, B also gets 39% and C gets 
22%. The number of containers is going to be 15.6, 15.6 and 8.8 for A, B 
an C. Since we can't split a container, how does the Capacity scheduler 
round these values in a real case? Who gets the two contended 
containers? I may think they are considered as extra containers, thus 
shared upon need among the three queues. Is this correct?

3) Let's say I have queues A and B. A is configured to get 20% (20 
containers) of the total cluster capacity (100 containers), B gets 80% 
(80 containers). Capacity scheduler gives available resources firstly to 
the most under-served queue.
In case A is using 10 containers and B is using 20, who is going to get 
the first available container? A is already using 50% of it's assigned 
capacity, B just 25%, but A has less containers than B... who is 
considered to be more under-served?

4) Does the previous question make sense at all? Because I have a doubt 
that when I have free containers I will just serve requests as they 
arrive, possibly over-provisioning a queue (that is: if I get a 
container request for an app in A, I will give it a container since I 
don't know that after a few milliseconds I will get a new request from 
B, or vice versa). The previous question may have sense if there was 
some sort of buffer that is filled with incoming requests, due to the 
difficulty of serving them in real time, thus making the scheduler able 
to choose the request from the most under-served queue. Is this what 

5) According to the example presented in "Apache Hadoop YARN: Moving 
beyond MapReduce and Batch Processing with Apache Hadoop 2" about the 
resource allocation with the capacity scheduler, what I understood is 
that the chance for a leaf queue to get resources above it's assigned 
capacity is always upper-limited by the fraction of cluster capacity 
assigned to its first/closer parent queue. That is: if I am a leaf queue 
A1, I can only get at most the resources dedicated to my parent A, while 
I can't get the ones from B, sibling of A, even if it doesn't have any 
running application. Actually at first I thought this over-provisioning 
was not limited, and regardless of the queue configuration a single 
application could get the whole cluster (excluding per-application 
limits). Did I misunderstood the example?

Thanks a lot


View raw message