airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bharath palaksha <bharath...@gmail.com>
Subject Re: Proposed roadmap for Airflow 2.0
Date Wed, 09 Oct 2019 06:38:38 GMT
Hi,

Is there any discussion thread on adding priority to tasks and cost-based
optimization?
priority and pre-emption as an option to the user. If priority is
specified, scheduler has to schedule high priority tasks and if pre-emption
is true, it can pre-empt current running task which is of lower priority

Thanks,
Bharath


On Mon, Sep 30, 2019 at 11:19 PM James Meickle
<jmeickle@quantopian.com.invalid> wrote:

> For what I'm looking for out of a 2.0, as an operator/cluster admin
> (separate from what I'd like to see as a DAG developer), I'd love to see:
>
> - Combine breaking changes into 2.0, and do as few as possible after
> - A semver policy for 2.0 and onwards. (For instance we got bit hard by a
> breaking API change in the k8s operator)
> - Regularly scheduled releases (like: "minor every other month, major every
> other year")
> - A security backport policy
> - Pinned deps for releases
> - A way to get integration/cloud vendor operator updates out-of-tree,
> without having to pull in unrelated Airflow updates
>
> For a lot of people, Airflow is an off-the-shelf app rather than a library,
> but we don't actually ship or support it anything like most comparable
> off-the-shelf apps. It makes it much harder to support than other
> applications, unless you're a Python developer yourself.
>
> On Mon, Sep 30, 2019 at 11:18 AM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> wrote:
>
> > All those are very important and we are going to work on some of them as
> > well.
> >
> > I think if there are breaking changes, we should rather try to fit them
> in
> > 2.0 release - at least to the point that they can be base for extending
> it
> > in later versions in backwards-compatible way (maybe then we should adopt
> > SemVer officially and follow it).
> >
> > J.
> >
> >
> > On Tue, Sep 24, 2019 at 11:52 PM James Meickle
> > <jmeickle@quantopian.com.invalid> wrote:
> >
> > > My question with that is, how often do we want to do major version
> > > increments? There's a few  API breaking changes I'd love to see, but
> > > whether to propose them for 2.0 depends on what the wait until 3.0
> looks
> > > like (or whether we'll allow more minor version breakages in the
> future)
> > >
> > > On Tue, Sep 24, 2019, 11:44 Dan Davydov <ddavydov@twitter.com.invalid>
> > > wrote:
> > >
> > > > I think along with "Improve Webserver Performance" we should solve
> the
> > > > serialization and task execution isolation problems a little bit more
> > > > completely since I imagine there could be backwards compatibility
> > issues.
> > > > e.g. mapping each task JSON to a Docker image or other serialized
> > > > representation that workers would then consume. See the attached PDF,
> > > > AIP-24 is a subset of the DAG Definition Serialization work, but in
> my
> > > > opinion we should still work on DAG Isolation too. My only concern is
> > > that
> > > > the scope is too big for 2.0.
> > > >
> > > > cc @Sumit Maheshwari <smaheshwari@twitter.com> who is also looking
> at
> > > > tackling some of these problems.
> > > >
> > > > On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor <ash@apache.org>
> > > wrote:
> > > >
> > > >> I'm also in favour of py-test (and it's what I use for most of my
> > > >> development) which is why I created
> > > >> https://issues.apache.org/jira/browse/AIRFLOW-4863, but I don't
> think
> > > >> non-user-facing/impacting changes need to go on the road map.
> > > >>
> > > >> -ash
> > > >>
> > > >> > On 24 Sep 2019, at 13:53, Tomasz Urbaszek <
> > > tomasz.urbaszek@polidea.com>
> > > >> wrote:
> > > >> >
> > > >> > I am thinking about proposing migration from nosetest to pytest
> > > because
> > > >> > it's "more up to date". I have even a POC but a lot of test fails
> > due
> > > to
> > > >> > probably side effects.
> > > >> >
> > > >> > Best,
> > > >> > Tomek
> > > >> >
> > > >> > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor <ash@apache.org
> >
> > > >> wrote:
> > > >> >
> > > >> >> That formatted very badly in plain text. The list was:
> > > >> >>
> > > >> >>        • Knative Executor (AIP-25, currently draft. Being
worked
> on
> > > by
> > > >> >> Daniel Imberman )
> > > >> >>        • Improve Webserver performance (AIP-24, currently
draft.
> > > Being
> > > >> >> worked on by myself, Kaxil Naik and Zhou Fang)
> > > >> >>        • Enhanced real-time UI
> > > >> >>        • Improve Scheduler performance
> > > >> >>        • Extend/finish the API (AIP-13 is part of this,
but not
> > all)
> > > >> >>        • Production Docker image + Helm chart
> > > >> >>
> > > >> >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor <ash@apache.org>
> > wrote:
> > > >> >>>
> > > >> >>> Hi everyone,
> > > >> >>>
> > > >> >>> I'd like to start working on a concrete plan to get Airflow
2.0
> > out,
> > > >> and
> > > >> >> as a result I've started updating
> > > >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
> > > >> >>>
> > > >> >>> In addition to all the tidy up work ("spring cleaning",
finish
> > tidy
> > > up
> > > >> >> after dropping Py2 etc) I'd propose the following 6 high
level
> > items:
> > > >> >>>
> > > >> >>> Knative Executor (AIP-25, currently draft. Being worked
on by
> > Daniel
> > > >> >> Imberman )
> > > >> >>> Improve Webserver performance (AIP-24, currently draft.
Being
> > worked
> > > >> on
> > > >> >> by myself, Kaxil Naik and Zhou Fang)
> > > >> >>> Enhanced real-time UI
> > > >> >>> Improve Scheduler performance
> > > >> >>> Extend/finish the API (AIP-13 is part of this, but not
all)
> > > >> >>> Production Docker image + Helm chart
> > > >> >>> We at Astronomer are committing to work on these in roughly
this
> > > order
> > > >> >> if no one else gets to them first. I also propose that we
create
> > SIGs
> > > >> >> (Special Interest Groups) in slack with weekly/fortnightly
(every
> > 14
> > > >> days)
> > > >> >> "calls"/update sessions. We already have #sig-ui and
> > > >> #sig-dag-serialisation.
> > > >> >>>
> > > >> >>> This roadmap is also not a promise that all of these
will be
> done
> > > >> before
> > > >> >> Airflow 2.0 - we may decide later to push something back
to v2.1
> > etc.
> > > >> >>>
> > > >> >>> Does anyone disagree strongly with these priorities,
or have
> > > anything
> > > >> >> they want to add that you are willing to work on?
> > > >> >>>
> > > >> >>> Thanks,
> > > >> >>> Ash
> > > >> >>
> > > >> >>
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Tomasz Urbaszek
> > > >> > Polidea <https://www.polidea.com/> | Junior Software Engineer
> > > >> >
> > > >> > M: +48 505 628 493 <+48505628493>
> > > >> > E: tomasz.urbaszek@polidea.com <tomasz.urbaszeki@polidea.com>
> > > >> >
> > > >> > Unique Tech
> > > >> > Check out our projects! <https://www.polidea.com/our-work>
> > > >>
> > > >>
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message