airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chao-Han Tsai <milton0...@gmail.com>
Subject Re: Proposed roadmap for Airflow 2.0
Date Thu, 10 Oct 2019 01:59:56 GMT
Although Airflow has the concept of task priority like Ash mentioned, it
does not pre-empt running tasks.

On Wed, Oct 9, 2019 at 12:42 AM Ash Berlin-Taylor <ash@apache.org> wrote:

> There's already a concept called priority_weight on tasks
> http://airflow.apache.org/concepts.html?highlight=priority_weight#pools
> (the doc about it is in relation to pools, but everything is run in a pool
> of "default_pool" if not specified.)
>
> Is that what you want?
>
> On 9 October 2019 07:38:38 BST, bharath palaksha <bharathp87@gmail.com>
> wrote:
> >Hi,
> >
> >Is there any discussion thread on adding priority to tasks and
> >cost-based
> >optimization?
> >priority and pre-emption as an option to the user. If priority is
> >specified, scheduler has to schedule high priority tasks and if
> >pre-emption
> >is true, it can pre-empt current running task which is of lower
> >priority
> >
> >Thanks,
> >Bharath
> >
> >
> >On Mon, Sep 30, 2019 at 11:19 PM James Meickle
> ><jmeickle@quantopian.com.invalid> wrote:
> >
> >> For what I'm looking for out of a 2.0, as an operator/cluster admin
> >> (separate from what I'd like to see as a DAG developer), I'd love to
> >see:
> >>
> >> - Combine breaking changes into 2.0, and do as few as possible after
> >> - A semver policy for 2.0 and onwards. (For instance we got bit hard
> >by a
> >> breaking API change in the k8s operator)
> >> - Regularly scheduled releases (like: "minor every other month, major
> >every
> >> other year")
> >> - A security backport policy
> >> - Pinned deps for releases
> >> - A way to get integration/cloud vendor operator updates out-of-tree,
> >> without having to pull in unrelated Airflow updates
> >>
> >> For a lot of people, Airflow is an off-the-shelf app rather than a
> >library,
> >> but we don't actually ship or support it anything like most
> >comparable
> >> off-the-shelf apps. It makes it much harder to support than other
> >> applications, unless you're a Python developer yourself.
> >>
> >> On Mon, Sep 30, 2019 at 11:18 AM Jarek Potiuk
> ><Jarek.Potiuk@polidea.com>
> >> wrote:
> >>
> >> > All those are very important and we are going to work on some of
> >them as
> >> > well.
> >> >
> >> > I think if there are breaking changes, we should rather try to fit
> >them
> >> in
> >> > 2.0 release - at least to the point that they can be base for
> >extending
> >> it
> >> > in later versions in backwards-compatible way (maybe then we should
> >adopt
> >> > SemVer officially and follow it).
> >> >
> >> > J.
> >> >
> >> >
> >> > On Tue, Sep 24, 2019 at 11:52 PM James Meickle
> >> > <jmeickle@quantopian.com.invalid> wrote:
> >> >
> >> > > My question with that is, how often do we want to do major
> >version
> >> > > increments? There's a few  API breaking changes I'd love to see,
> >but
> >> > > whether to propose them for 2.0 depends on what the wait until
> >3.0
> >> looks
> >> > > like (or whether we'll allow more minor version breakages in the
> >> future)
> >> > >
> >> > > On Tue, Sep 24, 2019, 11:44 Dan Davydov
> ><ddavydov@twitter.com.invalid>
> >> > > wrote:
> >> > >
> >> > > > I think along with "Improve Webserver Performance" we should
> >solve
> >> the
> >> > > > serialization and task execution isolation problems a little
> >bit more
> >> > > > completely since I imagine there could be backwards
> >compatibility
> >> > issues.
> >> > > > e.g. mapping each task JSON to a Docker image or other
> >serialized
> >> > > > representation that workers would then consume. See the
> >attached PDF,
> >> > > > AIP-24 is a subset of the DAG Definition Serialization work,
> >but in
> >> my
> >> > > > opinion we should still work on DAG Isolation too. My only
> >concern is
> >> > > that
> >> > > > the scope is too big for 2.0.
> >> > > >
> >> > > > cc @Sumit Maheshwari <smaheshwari@twitter.com> who is also
> >looking
> >> at
> >> > > > tackling some of these problems.
> >> > > >
> >> > > > On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor
> ><ash@apache.org>
> >> > > wrote:
> >> > > >
> >> > > >> I'm also in favour of py-test (and it's what I use for most
of
> >my
> >> > > >> development) which is why I created
> >> > > >> https://issues.apache.org/jira/browse/AIRFLOW-4863, but I
> >don't
> >> think
> >> > > >> non-user-facing/impacting changes need to go on the road
map.
> >> > > >>
> >> > > >> -ash
> >> > > >>
> >> > > >> > On 24 Sep 2019, at 13:53, Tomasz Urbaszek <
> >> > > tomasz.urbaszek@polidea.com>
> >> > > >> wrote:
> >> > > >> >
> >> > > >> > I am thinking about proposing migration from nosetest
to
> >pytest
> >> > > because
> >> > > >> > it's "more up to date". I have even a POC but a lot
of test
> >fails
> >> > due
> >> > > to
> >> > > >> > probably side effects.
> >> > > >> >
> >> > > >> > Best,
> >> > > >> > Tomek
> >> > > >> >
> >> > > >> > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor
> ><ash@apache.org
> >> >
> >> > > >> wrote:
> >> > > >> >
> >> > > >> >> That formatted very badly in plain text. The list
was:
> >> > > >> >>
> >> > > >> >>        • Knative Executor (AIP-25, currently draft.
Being
> >worked
> >> on
> >> > > by
> >> > > >> >> Daniel Imberman )
> >> > > >> >>        • Improve Webserver performance (AIP-24,
currently
> >draft.
> >> > > Being
> >> > > >> >> worked on by myself, Kaxil Naik and Zhou Fang)
> >> > > >> >>        • Enhanced real-time UI
> >> > > >> >>        • Improve Scheduler performance
> >> > > >> >>        • Extend/finish the API (AIP-13 is part
of this, but
> >not
> >> > all)
> >> > > >> >>        • Production Docker image + Helm chart
> >> > > >> >>
> >> > > >> >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor
> ><ash@apache.org>
> >> > wrote:
> >> > > >> >>>
> >> > > >> >>> Hi everyone,
> >> > > >> >>>
> >> > > >> >>> I'd like to start working on a concrete plan
to get
> >Airflow 2.0
> >> > out,
> >> > > >> and
> >> > > >> >> as a result I've started updating
> >> > > >> >>
> >https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
> >> > > >> >>>
> >> > > >> >>> In addition to all the tidy up work ("spring
cleaning",
> >finish
> >> > tidy
> >> > > up
> >> > > >> >> after dropping Py2 etc) I'd propose the following
6 high
> >level
> >> > items:
> >> > > >> >>>
> >> > > >> >>> Knative Executor (AIP-25, currently draft. Being
worked on
> >by
> >> > Daniel
> >> > > >> >> Imberman )
> >> > > >> >>> Improve Webserver performance (AIP-24, currently
draft.
> >Being
> >> > worked
> >> > > >> on
> >> > > >> >> by myself, Kaxil Naik and Zhou Fang)
> >> > > >> >>> Enhanced real-time UI
> >> > > >> >>> Improve Scheduler performance
> >> > > >> >>> Extend/finish the API (AIP-13 is part of this,
but not
> >all)
> >> > > >> >>> Production Docker image + Helm chart
> >> > > >> >>> We at Astronomer are committing to work on these
in
> >roughly this
> >> > > order
> >> > > >> >> if no one else gets to them first. I also propose
that we
> >create
> >> > SIGs
> >> > > >> >> (Special Interest Groups) in slack with weekly/fortnightly
> >(every
> >> > 14
> >> > > >> days)
> >> > > >> >> "calls"/update sessions. We already have #sig-ui
and
> >> > > >> #sig-dag-serialisation.
> >> > > >> >>>
> >> > > >> >>> This roadmap is also not a promise that all
of these will
> >be
> >> done
> >> > > >> before
> >> > > >> >> Airflow 2.0 - we may decide later to push something
back to
> >v2.1
> >> > etc.
> >> > > >> >>>
> >> > > >> >>> Does anyone disagree strongly with these priorities,
or
> >have
> >> > > anything
> >> > > >> >> they want to add that you are willing to work on?
> >> > > >> >>>
> >> > > >> >>> Thanks,
> >> > > >> >>> Ash
> >> > > >> >>
> >> > > >> >>
> >> > > >> >
> >> > > >> > --
> >> > > >> >
> >> > > >> > Tomasz Urbaszek
> >> > > >> > Polidea <https://www.polidea.com/> | Junior Software
> >Engineer
> >> > > >> >
> >> > > >> > M: +48 505 628 493 <+48505628493>
> >> > > >> > E: tomasz.urbaszek@polidea.com
> ><tomasz.urbaszeki@polidea.com>
> >> > > >> >
> >> > > >> > Unique Tech
> >> > > >> > Check out our projects! <https://www.polidea.com/our-work>
> >> > > >>
> >> > > >>
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > Jarek Potiuk
> >> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >> >
> >> > M: +48 660 796 129 <+48660796129>
> >> > [image: Polidea] <https://www.polidea.com/>
> >> >
> >>
>


-- 

Chao-Han Tsai

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message