aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Khutornenko <ma...@apache.org>
Subject Re: AURORA-1440 Evaluate Fenzo scheduling library
Date Tue, 29 Dec 2015 22:25:44 GMT
Some time ago I did a very quick and rough POC integrating scheduler
with Fenzo just to evaluate how it may fit into our architecture.
Surprisingly, it took just a few lines [1] to plug it in (without
constraints support of course, which is a much bigger effort). There
was also a big chunk of functionality missing that would allow us to
support task preemption within Fenzo. I spoke with Sharma back then
and he had intention to explore adding it. Not sure where it ended up
though.

I still think we should explore Fenzo integration idea as it will
unlock various rich bin packing algorithms we may never get to
implement. I have heard other Mesos frameworks (besides internal
Netflix proprietary schedulers) are actively exploring Fenzo [2]. It's
interesting to see how Fenzo could help us reduce online fragmentation
and support features like soft constraints out of the box.

That said, I don't think Fenzo should be considered as a sole
scheduling algorithm but rather as a supported alternative (at least
in the first iteration). If anything, Fenzo onboarding exercise may
help us make scheduling architecture more modular and pluggable.

I agree with Bill though that adding Fenzo into Aurora should not be a
self-goal but rather a well justified effort that clearly outlines the
benefits. I'd suggest to whoever takes on exploring this route start
with an Aurora list discussion first.

Thanks,
Maxim

[1] - https://github.com/maxim111333/incubator-aurora/blob/fenzo_proto/src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java#L210-L249
[2] - https://github.com/twosigma/Cook

On Tue, Dec 29, 2015 at 9:43 AM, Dobromir Montauk
<dmontauk@twitter.com.invalid> wrote:
> For AWS-based Aurora installations having 'tight bin packing' is critical
> for shutting down unused machines. I believe Fenzo helps with that, right?
>
> On Tue, Dec 29, 2015 at 9:31 AM, Bill Farner <wfarner@apache.org> wrote:
>
>> It would also be helpful to capture the goals we hope to achieve with the
>> integration.  We should also assess the risk of bringing it in,
>> specifically that we won't be left maintaining it (currently it's 100%
>> owned by netflix, mostly 1 developer AFAICT).
>>
>> On Tue, Dec 29, 2015 at 4:01 AM, Erb, Stephan <Stephan.Erb@blue-yonder.com
>> >
>> wrote:
>>
>> > Someone also expressed interest in Fenzo by adding it to the
>> > community-driven roadmap [1].
>> >
>> > AFAIK nobody has looked at in in detail, yet. Or at least nobody has
>> > posted about it on the mailinglist. Feel free to be that someone and
>> take a
>> > closer look at what would be necessary to leverage the power of Fenzo in
>> > Aurora :-)
>> >
>> > Without checking, I would assume that the following areas might need the
>> > most effort to address:
>> > * the preemption mechanism of Aurora to make room for priority/production
>> > jobs
>> > * the work-in-progress feature using oversubscribed resources [2]
>> >
>> > Regards,
>> > Stephan
>> >
>> > [1]
>> >
>> https://docs.google.com/document/d/1vyhTZSlEPeibQm2_7HK6JXOkydO0ZllZNQZ2O3cC4_0/edit?usp=sharing
>> > [2]
>> >
>> https://docs.google.com/document/d/1r1WCHgmPJp5wbrqSZLsgtxPNj3sULfHrSFmxp2GyPTo/edit?pref=2&pli=1#heading=h.af56b6bntcao
>> >
>> > ________________________________________
>> > From: Mauricio Garavaglia <mauriciogaravaglia@gmail.com>
>> > Sent: Tuesday, December 29, 2015 6:02 AM
>> > To: dev@aurora.apache.org
>> > Subject: AURORA-1440 Evaluate Fenzo scheduling library
>> >
>> > Hello,
>> >
>> > I found the issue AURORA-1440
>> > <https://issues.apache.org/jira/browse/AURORA-1440> about evaluating
>> fenzo
>> > to replace the scheduling algorithm. Has there been any progress on it?
>> is
>> > the intention just replace the implementation or also expose part of the
>> > configurations that fenzo allows?
>> >
>> > It would be handy to be able to select, for example, between cpu bin
>> > packing and memory bin packing. But also take advantage of the rich
>> > scheduling constraints api that it has. I assume a lot of discussion
>> would
>> > be needed regarding how to expose them though :)
>> > Thanks
>> >
>> >
>> > Mauricio
>> >
>>

Mime
View raw message