Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
Date: Mon, 11 Apr 2016 09:24:28 -0700
Message-ID: 
 <CAMTGeQkneeAcAQEYqZUKKk-_wG2utWcA_K1uBMDcpwaSfU6XOQ@mail.gmail.com>
Subject: Control rate of preemption?
From: Miles Crawford <milesc@allenai.org>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a113eb9b296c079053037fab7

--001a113eb9b296c079053037fab7
Content-Type: text/plain; charset=UTF-8

I'm using the YARN fair scheduler to allow a group of users to equally
share a cluster for running Spark jobs.

Works great, but when a large rebalance happens, Spark sometimes can't keep
up, and the job fails.

Is there any way to control the rate at which YARN preempts resources? I'd
love to limit the killing of containers to a slower pace, so Spark has a
chance to keep up.

Thanks,
-miles

--001a113eb9b296c079053037fab7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><br></div>I&#39;m using the YARN fair scheduler to al=
low a group of users to equally share a cluster for running Spark jobs.<div=
><br></div><div>Works great, but when a large rebalance happens, Spark some=
times can&#39;t keep up, and the job fails.</div><div><br></div><div>Is the=
re any way to control the rate at which YARN preempts resources? I&#39;d lo=
ve to limit the killing of containers to a slower pace, so Spark has a chan=
ce to keep up.</div><div><br></div><div>Thanks,</div><div>-miles</div></div=
>

--001a113eb9b296c079053037fab7--