Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (athena.apache.org: domain of alexbaretta@gmail.com
 designates 209.85.214.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKWX9VVSNJtvugnYU21Ms=G2fq6OWEwP+QmhCZmkMV9CXpoXtA@mail.gmail.com>
References: 
 <CAJc_syLZJ2VJC7H6gxLBbpTfQtsXrt2HJvhGGhyA6VBSjW55Sw@mail.gmail.com>
 <CAAsvFP=rdV-JG0Yv950TY9979kGAZ3d4+28GLotNJHq+-mHfGg@mail.gmail.com>
 <CAJc_syKPRdrAN_bR4hvBEvrBEDm+WB0432cuo0R2-VdLg8b78g@mail.gmail.com>
 <CAKWX9VWpW_xueXQ7TcNG4FgTc6_wKUzZ=SdZeoJH6s4aH1oR7A@mail.gmail.com>
 <CAJc_syJJvmYYF+H5zFOGUcWMmLoBGM1wC9A6Mu4F1jDSt54idg@mail.gmail.com>
 <CAKWX9VVSNJtvugnYU21Ms=G2fq6OWEwP+QmhCZmkMV9CXpoXtA@mail.gmail.com>
From: Alessandro Baretta <alexbaretta@gmail.com>
Date: Sun, 11 Jan 2015 07:36:44 -0800
Message-ID: 
 <CAJc_syJWQkEzUdcXMWrAtoZt7aOKPtOGJ5ALV_CG0x1nfL=WVw@mail.gmail.com>
Subject: Re: Job priority
To: Cody Koeninger <cody@koeninger.org>
Cc: Mark Hamstra <mark@clearstorydata.com>,
	"user@spark.apache.org" <user@spark.apache.org>,
 "dev@spark.apache.org" <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a113d34306a21d5050c6229c5

--001a113d34306a21d5050c6229c5
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Cody,

While I might be able to improve the scheduling of my jobs by using a few
different pools with weights equal to, say, 1, 1e3 and 1e6, effectively
getting a small handful of priority classes. Still, this is really not
quite what I am describing. This is why my original post was on the dev
list. Let me then ask if there is any interest in having priority queue job
scheduling in Spark. This is something I might be able to pull off.

Alex

On Sun, Jan 11, 2015 at 6:21 AM, Cody Koeninger <cody@koeninger.org> wrote:

> If you set up a number of pools equal to the number of different priority
> levels you want, make the relative weights of those pools very different,
> and submit a job to the pool representing its priority, I think youll get
> behavior equivalent to a priority queue. Try it and see.
>
> If I'm misunderstandng what youre trying to do, then I don't know.
>
>
> On Sunday, January 11, 2015, Alessandro Baretta <alexbaretta@gmail.com>
> wrote:
>
>> Cody,
>>
>> Maybe I'm not getting this, but it doesn't look like this page is
>> describing a priority queue scheduling policy. What this section discuss=
es
>> is how resources are shared between queues. A weight-1000 pool will get
>> 1000 times more resources allocated to it than a priority 1 queue. Great=
,
>> but not what I want. I want to be able to define an Ordering on make my
>> tasks representing their priority, and have Spark allocate all resources=
 to
>> the job that has the highest priority.
>>
>> Alex
>>
>> On Sat, Jan 10, 2015 at 10:11 PM, Cody Koeninger <cody@koeninger.org>
>> wrote:
>>
>>>
>>> http://spark.apache.org/docs/latest/job-scheduling.html#configuring-poo=
l-properties
>>>
>>> "Setting a high weight such as 1000 also makes it possible to implement
>>> *priority* between pools=E2=80=94in essence, the weight-1000 pool will =
always
>>> get to launch tasks first whenever it has jobs active."
>>>
>>> On Sat, Jan 10, 2015 at 11:57 PM, Alessandro Baretta <
>>> alexbaretta@gmail.com> wrote:
>>>
>>>> Mark,
>>>>
>>>> Thanks, but I don't see how this documentation solves my problem. You
>>>> are referring me to documentation of fair scheduling; whereas, I am as=
king
>>>> about as unfair a scheduling policy as can be: a priority queue.
>>>>
>>>> Alex
>>>>
>>>> On Sat, Jan 10, 2015 at 5:00 PM, Mark Hamstra <mark@clearstorydata.com=
>
>>>> wrote:
>>>>
>>>>> -dev, +user
>>>>>
>>>>> http://spark.apache.org/docs/latest/job-scheduling.html
>>>>>
>>>>>
>>>>> On Sat, Jan 10, 2015 at 4:40 PM, Alessandro Baretta <
>>>>> alexbaretta@gmail.com> wrote:
>>>>>
>>>>>> Is it possible to specify a priority level for a job, such that the
>>>>>> active
>>>>>> jobs might be scheduled in order of priority?
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>

--001a113d34306a21d5050c6229c5--