hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miles Crawford <mil...@allenai.org>
Subject Control rate of preemption?
Date Mon, 11 Apr 2016 16:24:28 GMT
I'm using the YARN fair scheduler to allow a group of users to equally
share a cluster for running Spark jobs.

Works great, but when a large rebalance happens, Spark sometimes can't keep
up, and the job fails.

Is there any way to control the rate at which YARN preempts resources? I'd
love to limit the killing of containers to a slower pace, so Spark has a
chance to keep up.


View raw message