airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Leslie-Waksman <geo...@cloverhealth.com.INVALID>
Subject Re: Airflow 2.0
Date Mon, 18 Dec 2017 18:48:33 GMT
I really do not think we should drop sqlite support. We use sqlite for
local testing/development; I used it when doing my initial evaluation of
Airflow; and there are regular comments in this group about people using
sqlite locally for their workflows. It feels like a critical feature for a
lot of use cases.

I also feel like we're kind of jumping the gun on pushing for 2.0 (is it
just because 9 is the last single digit?). There is a lot of the code base
that's pretty hairy and a number of things that are fairly non-performant /
buggy. I don't think that we NEED to break backward compatibility to clean
things up and fix things. I don't particularly like pickle or the old
import style but I don't feel like supporting them is terribly onerous (I
could be wrong).

New webserver, releasing the API, and even timezones are all things that
can be released in a non-breaking fashion.

--George

On Thu, Dec 14, 2017 at 2:20 PM Alek Storm <alek.storm@gmail.com> wrote:

> We use sqlite for developing Airflow DAGs locally, and doing basic checks
> for syntax/import errors in our CI/CD pipeline.
>
> Alek
>
> On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini <criccomini@apache.org>
> wrote:
>
> > Agree with Bolke that it's a good idea, but major work. I will push back
> on
> > this for 2.0 mostly due to time concerns. I don't want 2.0 to take months
> > and months to get out.
> >
> > > Btw: what about dropping sqlite support?
> >
> > I'm fine with this, but isn't this really useful for demos? Seems to me
> > that the out of the box experience from airflow is one of the things that
> > make sit so compelling.
> >
> > On Thu, Dec 14, 2017 at 2:12 PM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> > > That will take quite some work. It is a a good idea but also a major
> > > change. Not sure if we should target that.
> > >
> > > Btw: what about dropping sqlite support?
> > >
> > > Verstuurd vanaf mijn iPad
> > >
> > > > Op 14 dec. 2017 om 21:19 heeft Gael Magnan <gaelmagnan@gmail.com>
> het
> > > volgende geschreven:
> > > >
> > > > Hi,
> > > >
> > > > haven't been following much lately but on the import side of things,
> > > isn't
> > > > Airflow 2 the best moment to change to a pip plugin system for
> imports
> > of
> > > > third party stuff?
> > > > I.E being able to add a new type of credentials, operator etc..
> without
> > > > touching to the airflow code itself or having them in a special
> folder.
> > > >
> > > > Regards
> > > > Gael
> > > >
> > > >
> > > >
> > > > Le jeu. 14 déc. 2017 à 14:17, Chris Riccomini <criccomini@apache.org
> >
> > a
> > > > écrit :
> > > >
> > > >> I'm fine with sensor refactor. Added to Wiki.
> > > >>
> > > >> On Thu, Dec 14, 2017 at 11:16 AM, Chris Riccomini <
> > > criccomini@apache.org>
> > > >> wrote:
> > > >>
> > > >>> @Bolke,
> > > >>>
> > > >>>> Should we, before 2.0, start the graduation from the incubator?
> > > >>>
> > > >>> No, I'd rather keep them separate. We can certainly start
> graduation,
> > > but
> > > >>> I don't want to block 2.0. Can pursue them in parallel.
> > > >>>
> > > >>> On Thu, Dec 14, 2017 at 11:15 AM, Andy Hadjigeorgiou <
> > > >> andyxhadji@gmail.com
> > > >>>> wrote:
> > > >>>
> > > >>>> Does it make sense to include sensors.py refactor in 2.0,
so we
> can
> > > >>>> retire the old import structure easily and support the new
sensors
> > > >> package
> > > >>>> import structure?
> > > >>>>
> > > >>>> - Andy
> > > >>>>
> > > >>>> On Thu, Dec 14, 2017 at 2:12 PM, Driesprong, Fokko
> > > <fokko@driesprong.frl
> > > >>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> Good initiative. I would be happy to refactor the sensors
> package.
> > I
> > > >>>>> started on it but it changes a lot, all the imports will
break.
> > > >>>>>
> > > >>>>> https://github.com/apache/incubator-airflow/pull/2875
> > > >>>>>
> > > >>>>> What do you guys think?
> > > >>>>>
> > > >>>>> Cheers, Fokko
> > > >>>>>
> > > >>>>> 2017-12-14 20:09 GMT+01:00 Chris Riccomini <
> criccomini@apache.org
> > >:
> > > >>>>>
> > > >>>>>> I have created a wiki here:
> > > >>>>>>
> > > >>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
> > > >>>>>>
> > > >>>>>> To track features and progress.
> > > >>>>>>
> > > >>>>>> On Thu, Dec 14, 2017 at 11:08 AM, Chris Riccomini
<
> > > >>>>> criccomini@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>>> Re: #2: Is there a current ticket out for
removing the legacy
> > > >>>>> import
> > > >>>>>>> style?
> > > >>>>>>>
> > > >>>>>>> No, I don't think so, but you can create one!
:)
> > > >>>>>>>
> > > >>>>>>> On Thu, Dec 14, 2017 at 11:06 AM, Andy Hadjigeorgiou
<
> > > >>>>>> andyxhadji@gmail.com
> > > >>>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> This sounds great, something I'd like to see
updated for 2.0
> > > >>>>> release (or
> > > >>>>>>>> before) is the Airflow documentation
> > > >>>>>>>> <http://airflow.readthedocs.io/en/latest/installation.html>
(
> > > >>>>>>>> http://airflow.readthedocs.io/en/latest/installation.html).
> It
> > > >>>>> seems
> > > >>>>>> that
> > > >>>>>>>> updating the repo does not update this site
- and given that
> we
> > > >>>>> will be
> > > >>>>>>>> removing certain deprecated features I imagine
the docs will
> > > >> change
> > > >>>>>>>> substantially.
> > > >>>>>>>>
> > > >>>>>>>> Re: #2: Is there a current ticket out for
removing the legacy
> > > >> import
> > > >>>>>>>> style?
> > > >>>>>>>> I'm happy to help drive that forward.
> > > >>>>>>>>
> > > >>>>>>>> - Andy
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Dec 14, 2017 at 1:55 PM, Chris Riccomini
<
> > > >>>>> criccomini@apache.org
> > > >>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hey all,
> > > >>>>>>>>>
> > > >>>>>>>>> With 1.9.0 wrapping up soon (hopefully),
there's been some
> > > >>>>> discussion
> > > >>>>>> on
> > > >>>>>>>>> the having the next release be Airflow
2.0 (rather than
> 1.10).
> > > >>>>> This
> > > >>>>>>>> would
> > > >>>>>>>>> allow us to break compatibility, and clean
up some stuff.
> > > >> Proposed
> > > >>>>>>>> things
> > > >>>>>>>>> to include in 2.0 are:
> > > >>>>>>>>>
> > > >>>>>>>>> 1) New webserver that Joy Gao has been
working on.
> > > >>>>>>>>> 2) Remove the legacy import style that's
been deprecated
> since
> > > >> at
> > > >>>>>> least
> > > >>>>>>>> 1.8
> > > >>>>>>>>> 3) New timzone feature
> > > >>>>>>>>> 4) Move API out of experimental
> > > >>>>>>>>>
> > > >>>>>>>>> I want to keep the list fairly tight,
preferably to things
> that
> > > >>>>> have
> > > >>>>>>>>> already been done, so that we can ship
it fairly quickly (in
> > the
> > > >>>>> next
> > > >>>>>>>>> couple of months).
> > > >>>>>>>>>
> > > >>>>>>>>> Does this sound like a good list?
> > > >>>>>>>>>
> > > >>>>>>>>> Cheers,
> > > >>>>>>>>> Chris
> > > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message