airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Proposed roadmap for Airflow 2.0
Date Mon, 30 Sep 2019 15:18:21 GMT
All those are very important and we are going to work on some of them as
well.

I think if there are breaking changes, we should rather try to fit them in
2.0 release - at least to the point that they can be base for extending it
in later versions in backwards-compatible way (maybe then we should adopt
SemVer officially and follow it).

J.


On Tue, Sep 24, 2019 at 11:52 PM James Meickle
<jmeickle@quantopian.com.invalid> wrote:

> My question with that is, how often do we want to do major version
> increments? There's a few  API breaking changes I'd love to see, but
> whether to propose them for 2.0 depends on what the wait until 3.0 looks
> like (or whether we'll allow more minor version breakages in the future)
>
> On Tue, Sep 24, 2019, 11:44 Dan Davydov <ddavydov@twitter.com.invalid>
> wrote:
>
> > I think along with "Improve Webserver Performance" we should solve the
> > serialization and task execution isolation problems a little bit more
> > completely since I imagine there could be backwards compatibility issues.
> > e.g. mapping each task JSON to a Docker image or other serialized
> > representation that workers would then consume. See the attached PDF,
> > AIP-24 is a subset of the DAG Definition Serialization work, but in my
> > opinion we should still work on DAG Isolation too. My only concern is
> that
> > the scope is too big for 2.0.
> >
> > cc @Sumit Maheshwari <smaheshwari@twitter.com> who is also looking at
> > tackling some of these problems.
> >
> > On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor <ash@apache.org>
> wrote:
> >
> >> I'm also in favour of py-test (and it's what I use for most of my
> >> development) which is why I created
> >> https://issues.apache.org/jira/browse/AIRFLOW-4863, but I don't think
> >> non-user-facing/impacting changes need to go on the road map.
> >>
> >> -ash
> >>
> >> > On 24 Sep 2019, at 13:53, Tomasz Urbaszek <
> tomasz.urbaszek@polidea.com>
> >> wrote:
> >> >
> >> > I am thinking about proposing migration from nosetest to pytest
> because
> >> > it's "more up to date". I have even a POC but a lot of test fails due
> to
> >> > probably side effects.
> >> >
> >> > Best,
> >> > Tomek
> >> >
> >> > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor <ash@apache.org>
> >> wrote:
> >> >
> >> >> That formatted very badly in plain text. The list was:
> >> >>
> >> >>        • Knative Executor (AIP-25, currently draft. Being worked
on
> by
> >> >> Daniel Imberman )
> >> >>        • Improve Webserver performance (AIP-24, currently draft.
> Being
> >> >> worked on by myself, Kaxil Naik and Zhou Fang)
> >> >>        • Enhanced real-time UI
> >> >>        • Improve Scheduler performance
> >> >>        • Extend/finish the API (AIP-13 is part of this, but not all)
> >> >>        • Production Docker image + Helm chart
> >> >>
> >> >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor <ash@apache.org>
wrote:
> >> >>>
> >> >>> Hi everyone,
> >> >>>
> >> >>> I'd like to start working on a concrete plan to get Airflow 2.0
out,
> >> and
> >> >> as a result I've started updating
> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
> >> >>>
> >> >>> In addition to all the tidy up work ("spring cleaning", finish
tidy
> up
> >> >> after dropping Py2 etc) I'd propose the following 6 high level items:
> >> >>>
> >> >>> Knative Executor (AIP-25, currently draft. Being worked on by Daniel
> >> >> Imberman )
> >> >>> Improve Webserver performance (AIP-24, currently draft. Being worked
> >> on
> >> >> by myself, Kaxil Naik and Zhou Fang)
> >> >>> Enhanced real-time UI
> >> >>> Improve Scheduler performance
> >> >>> Extend/finish the API (AIP-13 is part of this, but not all)
> >> >>> Production Docker image + Helm chart
> >> >>> We at Astronomer are committing to work on these in roughly this
> order
> >> >> if no one else gets to them first. I also propose that we create SIGs
> >> >> (Special Interest Groups) in slack with weekly/fortnightly (every 14
> >> days)
> >> >> "calls"/update sessions. We already have #sig-ui and
> >> #sig-dag-serialisation.
> >> >>>
> >> >>> This roadmap is also not a promise that all of these will be done
> >> before
> >> >> Airflow 2.0 - we may decide later to push something back to v2.1 etc.
> >> >>>
> >> >>> Does anyone disagree strongly with these priorities, or have
> anything
> >> >> they want to add that you are willing to work on?
> >> >>>
> >> >>> Thanks,
> >> >>> Ash
> >> >>
> >> >>
> >> >
> >> > --
> >> >
> >> > Tomasz Urbaszek
> >> > Polidea <https://www.polidea.com/> | Junior Software Engineer
> >> >
> >> > M: +48 505 628 493 <+48505628493>
> >> > E: tomasz.urbaszek@polidea.com <tomasz.urbaszeki@polidea.com>
> >> >
> >> > Unique Tech
> >> > Check out our projects! <https://www.polidea.com/our-work>
> >>
> >>
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message