airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Davydov <dan.davy...@airbnb.com.INVALID>
Subject Re: Airflow 1.8.0 Alpha 1
Date Tue, 03 Jan 2017 13:38:54 GMT
I have also started on this effort, recently Alex Guziel and I have been
pushing Airbnb's custom cherries onto master to get Airbnb back onto master
in order for us to do a release.

I think it might make sense to wait for these two commits to get merged in
since they would be quite nice to have for all Airflow users and seem like
they will be merged soon:
Schedule all pending DAG runs in a single scheduler loop -
https://github.com/apache/incubator-airflow/pull/1906
Add Support for dag.backfill=(True|False) Option -
https://github.com/apache/incubator-airflow/pull/1830
Impersonation Support + Cgroups - https://github.com/apache/
incubator-airflow/pull/1934 (this is kind of important from the Airbnb side
so that we can help test the new master without having to cherrypick this
PR on top of it which would make the testing unreliable for others).

If there are PRs that affect the core of Airflow that other committers
think are important to merge we could include these too. I can commit to
pushing out the Impersonation/Cgroups PR this week pending PR comments.
What do you think Bolke?

On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> Hey Alex,
>
> I have noticed the same, and it is also the reason why we have Alpha
> versions. For now I have noticed the following:
>
> * Tasks can get in limbo between scheduler and executor:
> https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948>
> * Try_number not increased due to reset in LocalTaskJob:
> https://github.com/apache/incubator-airflow/pull/1969 <
> https://github.com/apache/incubator-airflow/pull/1969>
> * one_failed trigger not executed
>
> My idea is to move to a Samba style of releases eventually, but for now I
> would like to get master into a state that we understand and therefore not
> accept any patches that do not address any bugs.
>
> If you (or anyone else) can review the above PRs and add your own as well
> then I can create another Alpha version. I’ll be on gitter as much as I can
> so we can speed up if needed.
>
> - Bolke
>
> > On 3 Jan 2017, at 08:51, Alex Van Boxel <alex@vanboxel.be> wrote:
> >
> > Hey Bolke,
> >
> > thanks for getting this moving. But I already have some blockers, since I
> > moved up master to this release (moved from end November to now)
> stability
> > has gone down (certainly on Celary). I'm trying to identify the core
> > problems and see if I can fix them.
> >
> > On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin <bdbruin@gmail.com
> <mailto:bdbruin@gmail.com>> wrote:
> >
> > Dear All,
> >
> > On the verge of the New Year, I decided to be a little bit cheeky and to
> > make available an Airflow 1.8.0 Alpha 1. We have been talking about it
> for
> > a long time now and by doing this I wanted bootstrap the process. It
> should
> > by no means be considered an Apache release yet. This is for testing
> > purposes in the dev community around Airflow, nothing else.
> >
> > The build is exactly the same as the state of master (git 410736d) plus
> the
> > change to version “1.8.0.alpha1” in version.py.
> >
> > I am dedicating quite some time next week and beyond to get a release
> out.
> > Hopefully we can get some help with testing, changelog etc. To make this
> > possible I would like to propose a freeze to adding new features for at
> > least two weeks - say until Jan 15.
> >
> > You can find the tar here: http://people.apache.org/~bolke/ <
> > http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/>> .
> It isn’t signed. Following versions
> > will be. SHA is available.
> >
> > Lastly, Alpha 1 does not have the fix for retries yet. So we will get an
> > Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
> > https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948> <
> > https://github.com/apache/incubator-airflow/pull/1948 <
> https://github.com/apache/incubator-airflow/pull/1948>> , but your
> feedback
> > is required as it is entrenched in new processing code that you are
> running
> > in production afaik - so I wonder what happens in your fork.
> >
> > Happy New Year!
> >
> > Bolke
> >
> >
> >
> > --
> >  _/
> > _/ Alex Van Boxel
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message