airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: Proposed roadmap for Airflow 2.0
Date Wed, 09 Oct 2019 07:42:03 GMT
There's already a concept called priority_weight on tasks http://airflow.apache.org/concepts.html?highlight=priority_weight#pools
(the doc about it is in relation to pools, but everything is run in a pool of "default_pool"
if not specified.)

Is that what you want?

On 9 October 2019 07:38:38 BST, bharath palaksha <bharathp87@gmail.com> wrote:
>Hi,
>
>Is there any discussion thread on adding priority to tasks and
>cost-based
>optimization?
>priority and pre-emption as an option to the user. If priority is
>specified, scheduler has to schedule high priority tasks and if
>pre-emption
>is true, it can pre-empt current running task which is of lower
>priority
>
>Thanks,
>Bharath
>
>
>On Mon, Sep 30, 2019 at 11:19 PM James Meickle
><jmeickle@quantopian.com.invalid> wrote:
>
>> For what I'm looking for out of a 2.0, as an operator/cluster admin
>> (separate from what I'd like to see as a DAG developer), I'd love to
>see:
>>
>> - Combine breaking changes into 2.0, and do as few as possible after
>> - A semver policy for 2.0 and onwards. (For instance we got bit hard
>by a
>> breaking API change in the k8s operator)
>> - Regularly scheduled releases (like: "minor every other month, major
>every
>> other year")
>> - A security backport policy
>> - Pinned deps for releases
>> - A way to get integration/cloud vendor operator updates out-of-tree,
>> without having to pull in unrelated Airflow updates
>>
>> For a lot of people, Airflow is an off-the-shelf app rather than a
>library,
>> but we don't actually ship or support it anything like most
>comparable
>> off-the-shelf apps. It makes it much harder to support than other
>> applications, unless you're a Python developer yourself.
>>
>> On Mon, Sep 30, 2019 at 11:18 AM Jarek Potiuk
><Jarek.Potiuk@polidea.com>
>> wrote:
>>
>> > All those are very important and we are going to work on some of
>them as
>> > well.
>> >
>> > I think if there are breaking changes, we should rather try to fit
>them
>> in
>> > 2.0 release - at least to the point that they can be base for
>extending
>> it
>> > in later versions in backwards-compatible way (maybe then we should
>adopt
>> > SemVer officially and follow it).
>> >
>> > J.
>> >
>> >
>> > On Tue, Sep 24, 2019 at 11:52 PM James Meickle
>> > <jmeickle@quantopian.com.invalid> wrote:
>> >
>> > > My question with that is, how often do we want to do major
>version
>> > > increments? There's a few  API breaking changes I'd love to see,
>but
>> > > whether to propose them for 2.0 depends on what the wait until
>3.0
>> looks
>> > > like (or whether we'll allow more minor version breakages in the
>> future)
>> > >
>> > > On Tue, Sep 24, 2019, 11:44 Dan Davydov
><ddavydov@twitter.com.invalid>
>> > > wrote:
>> > >
>> > > > I think along with "Improve Webserver Performance" we should
>solve
>> the
>> > > > serialization and task execution isolation problems a little
>bit more
>> > > > completely since I imagine there could be backwards
>compatibility
>> > issues.
>> > > > e.g. mapping each task JSON to a Docker image or other
>serialized
>> > > > representation that workers would then consume. See the
>attached PDF,
>> > > > AIP-24 is a subset of the DAG Definition Serialization work,
>but in
>> my
>> > > > opinion we should still work on DAG Isolation too. My only
>concern is
>> > > that
>> > > > the scope is too big for 2.0.
>> > > >
>> > > > cc @Sumit Maheshwari <smaheshwari@twitter.com> who is also
>looking
>> at
>> > > > tackling some of these problems.
>> > > >
>> > > > On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor
><ash@apache.org>
>> > > wrote:
>> > > >
>> > > >> I'm also in favour of py-test (and it's what I use for most of
>my
>> > > >> development) which is why I created
>> > > >> https://issues.apache.org/jira/browse/AIRFLOW-4863, but I
>don't
>> think
>> > > >> non-user-facing/impacting changes need to go on the road map.
>> > > >>
>> > > >> -ash
>> > > >>
>> > > >> > On 24 Sep 2019, at 13:53, Tomasz Urbaszek <
>> > > tomasz.urbaszek@polidea.com>
>> > > >> wrote:
>> > > >> >
>> > > >> > I am thinking about proposing migration from nosetest to
>pytest
>> > > because
>> > > >> > it's "more up to date". I have even a POC but a lot of test
>fails
>> > due
>> > > to
>> > > >> > probably side effects.
>> > > >> >
>> > > >> > Best,
>> > > >> > Tomek
>> > > >> >
>> > > >> > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor
><ash@apache.org
>> >
>> > > >> wrote:
>> > > >> >
>> > > >> >> That formatted very badly in plain text. The list was:
>> > > >> >>
>> > > >> >>        • Knative Executor (AIP-25, currently draft.
Being
>worked
>> on
>> > > by
>> > > >> >> Daniel Imberman )
>> > > >> >>        • Improve Webserver performance (AIP-24, currently
>draft.
>> > > Being
>> > > >> >> worked on by myself, Kaxil Naik and Zhou Fang)
>> > > >> >>        • Enhanced real-time UI
>> > > >> >>        • Improve Scheduler performance
>> > > >> >>        • Extend/finish the API (AIP-13 is part of this,
but
>not
>> > all)
>> > > >> >>        • Production Docker image + Helm chart
>> > > >> >>
>> > > >> >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor
><ash@apache.org>
>> > wrote:
>> > > >> >>>
>> > > >> >>> Hi everyone,
>> > > >> >>>
>> > > >> >>> I'd like to start working on a concrete plan to get
>Airflow 2.0
>> > out,
>> > > >> and
>> > > >> >> as a result I've started updating
>> > > >> >>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
>> > > >> >>>
>> > > >> >>> In addition to all the tidy up work ("spring cleaning",
>finish
>> > tidy
>> > > up
>> > > >> >> after dropping Py2 etc) I'd propose the following 6 high
>level
>> > items:
>> > > >> >>>
>> > > >> >>> Knative Executor (AIP-25, currently draft. Being
worked on
>by
>> > Daniel
>> > > >> >> Imberman )
>> > > >> >>> Improve Webserver performance (AIP-24, currently
draft.
>Being
>> > worked
>> > > >> on
>> > > >> >> by myself, Kaxil Naik and Zhou Fang)
>> > > >> >>> Enhanced real-time UI
>> > > >> >>> Improve Scheduler performance
>> > > >> >>> Extend/finish the API (AIP-13 is part of this, but
not
>all)
>> > > >> >>> Production Docker image + Helm chart
>> > > >> >>> We at Astronomer are committing to work on these
in
>roughly this
>> > > order
>> > > >> >> if no one else gets to them first. I also propose that
we
>create
>> > SIGs
>> > > >> >> (Special Interest Groups) in slack with weekly/fortnightly
>(every
>> > 14
>> > > >> days)
>> > > >> >> "calls"/update sessions. We already have #sig-ui and
>> > > >> #sig-dag-serialisation.
>> > > >> >>>
>> > > >> >>> This roadmap is also not a promise that all of these
will
>be
>> done
>> > > >> before
>> > > >> >> Airflow 2.0 - we may decide later to push something back
to
>v2.1
>> > etc.
>> > > >> >>>
>> > > >> >>> Does anyone disagree strongly with these priorities,
or
>have
>> > > anything
>> > > >> >> they want to add that you are willing to work on?
>> > > >> >>>
>> > > >> >>> Thanks,
>> > > >> >>> Ash
>> > > >> >>
>> > > >> >>
>> > > >> >
>> > > >> > --
>> > > >> >
>> > > >> > Tomasz Urbaszek
>> > > >> > Polidea <https://www.polidea.com/> | Junior Software
>Engineer
>> > > >> >
>> > > >> > M: +48 505 628 493 <+48505628493>
>> > > >> > E: tomasz.urbaszek@polidea.com
><tomasz.urbaszeki@polidea.com>
>> > > >> >
>> > > >> > Unique Tech
>> > > >> > Check out our projects! <https://www.polidea.com/our-work>
>> > > >>
>> > > >>
>> > >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>> >
>>

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message