airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alek Storm <alek.st...@gmail.com>
Subject Re: Airflow 2.0
Date Thu, 14 Dec 2017 22:20:15 GMT
We use sqlite for developing Airflow DAGs locally, and doing basic checks
for syntax/import errors in our CI/CD pipeline.

Alek

On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini <criccomini@apache.org>
wrote:

> Agree with Bolke that it's a good idea, but major work. I will push back on
> this for 2.0 mostly due to time concerns. I don't want 2.0 to take months
> and months to get out.
>
> > Btw: what about dropping sqlite support?
>
> I'm fine with this, but isn't this really useful for demos? Seems to me
> that the out of the box experience from airflow is one of the things that
> make sit so compelling.
>
> On Thu, Dec 14, 2017 at 2:12 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>
> > That will take quite some work. It is a a good idea but also a major
> > change. Not sure if we should target that.
> >
> > Btw: what about dropping sqlite support?
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 14 dec. 2017 om 21:19 heeft Gael Magnan <gaelmagnan@gmail.com> het
> > volgende geschreven:
> > >
> > > Hi,
> > >
> > > haven't been following much lately but on the import side of things,
> > isn't
> > > Airflow 2 the best moment to change to a pip plugin system for imports
> of
> > > third party stuff?
> > > I.E being able to add a new type of credentials, operator etc.. without
> > > touching to the airflow code itself or having them in a special folder.
> > >
> > > Regards
> > > Gael
> > >
> > >
> > >
> > > Le jeu. 14 déc. 2017 à 14:17, Chris Riccomini <criccomini@apache.org>
> a
> > > écrit :
> > >
> > >> I'm fine with sensor refactor. Added to Wiki.
> > >>
> > >> On Thu, Dec 14, 2017 at 11:16 AM, Chris Riccomini <
> > criccomini@apache.org>
> > >> wrote:
> > >>
> > >>> @Bolke,
> > >>>
> > >>>> Should we, before 2.0, start the graduation from the incubator?
> > >>>
> > >>> No, I'd rather keep them separate. We can certainly start graduation,
> > but
> > >>> I don't want to block 2.0. Can pursue them in parallel.
> > >>>
> > >>> On Thu, Dec 14, 2017 at 11:15 AM, Andy Hadjigeorgiou <
> > >> andyxhadji@gmail.com
> > >>>> wrote:
> > >>>
> > >>>> Does it make sense to include sensors.py refactor in 2.0, so we
can
> > >>>> retire the old import structure easily and support the new sensors
> > >> package
> > >>>> import structure?
> > >>>>
> > >>>> - Andy
> > >>>>
> > >>>> On Thu, Dec 14, 2017 at 2:12 PM, Driesprong, Fokko
> > <fokko@driesprong.frl
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Good initiative. I would be happy to refactor the sensors package.
> I
> > >>>>> started on it but it changes a lot, all the imports will break.
> > >>>>>
> > >>>>> https://github.com/apache/incubator-airflow/pull/2875
> > >>>>>
> > >>>>> What do you guys think?
> > >>>>>
> > >>>>> Cheers, Fokko
> > >>>>>
> > >>>>> 2017-12-14 20:09 GMT+01:00 Chris Riccomini <criccomini@apache.org
> >:
> > >>>>>
> > >>>>>> I have created a wiki here:
> > >>>>>>
> > >>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
> > >>>>>>
> > >>>>>> To track features and progress.
> > >>>>>>
> > >>>>>> On Thu, Dec 14, 2017 at 11:08 AM, Chris Riccomini <
> > >>>>> criccomini@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>>> Re: #2: Is there a current ticket out for removing
the legacy
> > >>>>> import
> > >>>>>>> style?
> > >>>>>>>
> > >>>>>>> No, I don't think so, but you can create one! :)
> > >>>>>>>
> > >>>>>>> On Thu, Dec 14, 2017 at 11:06 AM, Andy Hadjigeorgiou
<
> > >>>>>> andyxhadji@gmail.com
> > >>>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> This sounds great, something I'd like to see updated
for 2.0
> > >>>>> release (or
> > >>>>>>>> before) is the Airflow documentation
> > >>>>>>>> <http://airflow.readthedocs.io/en/latest/installation.html>
(
> > >>>>>>>> http://airflow.readthedocs.io/en/latest/installation.html).
It
> > >>>>> seems
> > >>>>>> that
> > >>>>>>>> updating the repo does not update this site - and
given that we
> > >>>>> will be
> > >>>>>>>> removing certain deprecated features I imagine
the docs will
> > >> change
> > >>>>>>>> substantially.
> > >>>>>>>>
> > >>>>>>>> Re: #2: Is there a current ticket out for removing
the legacy
> > >> import
> > >>>>>>>> style?
> > >>>>>>>> I'm happy to help drive that forward.
> > >>>>>>>>
> > >>>>>>>> - Andy
> > >>>>>>>>
> > >>>>>>>> On Thu, Dec 14, 2017 at 1:55 PM, Chris Riccomini
<
> > >>>>> criccomini@apache.org
> > >>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hey all,
> > >>>>>>>>>
> > >>>>>>>>> With 1.9.0 wrapping up soon (hopefully), there's
been some
> > >>>>> discussion
> > >>>>>> on
> > >>>>>>>>> the having the next release be Airflow 2.0
(rather than 1.10).
> > >>>>> This
> > >>>>>>>> would
> > >>>>>>>>> allow us to break compatibility, and clean
up some stuff.
> > >> Proposed
> > >>>>>>>> things
> > >>>>>>>>> to include in 2.0 are:
> > >>>>>>>>>
> > >>>>>>>>> 1) New webserver that Joy Gao has been working
on.
> > >>>>>>>>> 2) Remove the legacy import style that's been
deprecated since
> > >> at
> > >>>>>> least
> > >>>>>>>> 1.8
> > >>>>>>>>> 3) New timzone feature
> > >>>>>>>>> 4) Move API out of experimental
> > >>>>>>>>>
> > >>>>>>>>> I want to keep the list fairly tight, preferably
to things that
> > >>>>> have
> > >>>>>>>>> already been done, so that we can ship it fairly
quickly (in
> the
> > >>>>> next
> > >>>>>>>>> couple of months).
> > >>>>>>>>>
> > >>>>>>>>> Does this sound like a good list?
> > >>>>>>>>>
> > >>>>>>>>> Cheers,
> > >>>>>>>>> Chris
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message