hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 胡子千 <hzq0...@gmail.com>
Subject Some questions about preemption policy on yarn
Date Mon, 20 Mar 2017 09:28:59 GMT
Hi,
I'm a user of Hadoop YARN and these days i'm testing Node LABEL function on
YARN 2.8.0 with capacity scheduler. I found that the preemption didn't work
on queues with label. Here is the details:
1. I set *test* label to 2 nodes in our cluster.
2. I set *test1*, *test2* queues under root which can only access *test*
label. And each queue's accessible-node-labels.test.capacity=50,
accessible-node-labels.test.maximum-capacity=100
3. enable the preemption policy:  yarn.resourcemanager.
scheduler.monitor.enable=true
4. submit a spark task (named A) to queue *test1*, which asks for 16
executors and will use all resource of *test* partition.
5. submit a spark test (named B) to queue *test2*. I assumes that because
of the under-satisfied of test2 and over-satisfied of test1, the preemption
will happen and each queue will use 50% resource of partition test finally.
In fact, the preempting didn't happen and the task B stay in accepted
state. when task A finished, task b started to run.
6. submit same task to different queues in default partition and the
preemption happens as we expected。

I found that a patch YARN-2498 about preemption have merged to 2.8.0 and i
think with this patch YARN supports preemption on labeled partitions. So is
there any configure need to set or I made some mistake when i use this
function? Or did I misunderstand patch YARN-2498 and in fact 2.8.0 don't
support preemption on labeled partitions?

Looking forward your reply, thank you.

-- 

Best regards!

Ziqian HU 胡子千
Department of Computer Science, School of EECS, Peking University

Mime
View raw message