hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 胡子千 <hzq0...@gmail.com>
Subject Some questions about preemption policy on yarn
Date Mon, 20 Mar 2017 09:28:59 GMT
I'm a user of Hadoop YARN and these days i'm testing Node LABEL function on
YARN 2.8.0 with capacity scheduler. I found that the preemption didn't work
on queues with label. Here is the details:
1. I set *test* label to 2 nodes in our cluster.
2. I set *test1*, *test2* queues under root which can only access *test*
label. And each queue's accessible-node-labels.test.capacity=50,
3. enable the preemption policy:  yarn.resourcemanager.
4. submit a spark task (named A) to queue *test1*, which asks for 16
executors and will use all resource of *test* partition.
5. submit a spark test (named B) to queue *test2*. I assumes that because
of the under-satisfied of test2 and over-satisfied of test1, the preemption
will happen and each queue will use 50% resource of partition test finally.
In fact, the preempting didn't happen and the task B stay in accepted
state. when task A finished, task b started to run.
6. submit same task to different queues in default partition and the
preemption happens as we expected。

I found that a patch YARN-2498 about preemption have merged to 2.8.0 and i
think with this patch YARN supports preemption on labeled partitions. So is
there any configure need to set or I made some mistake when i use this
function? Or did I misunderstand patch YARN-2498 and in fact 2.8.0 don't
support preemption on labeled partitions?

Looking forward your reply, thank you.


Best regards!

Ziqian HU 胡子千
Department of Computer Science, School of EECS, Peking University

View raw message