Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: 
 <CAMTGeQkneeAcAQEYqZUKKk-_wG2utWcA_K1uBMDcpwaSfU6XOQ@mail.gmail.com>
References: 
 <CAMTGeQkneeAcAQEYqZUKKk-_wG2utWcA_K1uBMDcpwaSfU6XOQ@mail.gmail.com>
Date: Tue, 12 Apr 2016 13:58:10 -0700
Message-ID: 
 <CAMTGeQn24YznrM7+3En-ghVxywPaPJvrLghV1nVeTXi+OWvg-Q@mail.gmail.com>
Subject: Re: Control rate of preemption?
From: Miles Crawford <milesc@allenai.org>
To: user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

In looking at the code I found two undocumented config properties:

yarn.scheduler.fair.preemptionInterval
yarn.scheduler.fair.waitTimeBeforeKill

But these don't seem to enough for me, since it appears the fair
scheduler will still preempt as many containers as it would like in a
single operation.  I was hoping for something like:

yarn.scheduler.fair.maxContainersToPreemptPerInterval

So that I could smooth out the rebalance operation over a longer time...

-m

On Mon, Apr 11, 2016 at 9:24 AM, Miles Crawford <milesc@allenai.org> wrote:
>
> I'm using the YARN fair scheduler to allow a group of users to equally share
> a cluster for running Spark jobs.
>
> Works great, but when a large rebalance happens, Spark sometimes can't keep
> up, and the job fails.
>
> Is there any way to control the rate at which YARN preempts resources? I'd
> love to limit the killing of containers to a slower pace, so Spark has a
> chance to keep up.
>
> Thanks,
> -miles

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org