hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shay Rojansky <r...@roji.org>
Subject Fair scheduler queue and preemption question
Date Sun, 07 Sep 2014 15:20:32 GMT
Hi everyone. I'm pretty much a Hadoop newbie and want to make sure I
understand things correctly.

I set up my Hadoop cluster with the fair schedular and 3 queues, where each
queue has the same weight as the other - the goal is for 3 users to get the
same share of the cluster.

Preemption is enabled, but I'm seeing some non-intuitive behavior. If one
user submits a job A that takes up the entire cluster, and then another
user submits job B, 33% of job A's containers are preempted and their
capacity transferred to job B. I had expected it to be 50% - since the two
*active* queues have the same weight.

What seems problematic here, is that if the two users submitted their jobs
at the same time, they would receive 50% each, right? It seems very strange
that the *stable* scheduling situation of long-running jobs would be
influenced by a race condition such as the exact submission time. Or in
other words, that the scheduling policy for allocating new/empty containers
is different from the scheduling policy for preempting already-running ones.

I do understand that this is how the fair scheduler works. I was wondering
if I'm missing something, or whether some other setup could provide my
expected behavior (perhaps with the capacity scheduler?).

Any input here would be greatly appreciated!


View raw message