airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Airflow 1.8.0 Alpha 1
Date Tue, 03 Jan 2017 17:28:33 GMT
@bolke thanks for taking the lead on this. Let's get these cherries in
master and move forward with a release.

Happy new year everyone!

Max

On Tue, Jan 3, 2017 at 7:30 AM, Alex Van Boxel <alex@vanboxel.be> wrote:

> If they should make the first alpha, maybe they should be rebased so they
> can be merged in.
>
> On Tue, Jan 3, 2017 at 2:39 PM Dan Davydov <dan.davydov@airbnb.com.
> invalid>
> wrote:
>
> > I have also started on this effort, recently Alex Guziel and I have been
> > pushing Airbnb's custom cherries onto master to get Airbnb back onto
> master
> > in order for us to do a release.
> >
> > I think it might make sense to wait for these two commits to get merged
> in
> > since they would be quite nice to have for all Airflow users and seem
> like
> > they will be merged soon:
> > Schedule all pending DAG runs in a single scheduler loop -
> > https://github.com/apache/incubator-airflow/pull/1906
> > Add Support for dag.backfill=(True|False) Option -
> > https://github.com/apache/incubator-airflow/pull/1830
> > Impersonation Support + Cgroups - https://github.com/apache/
> > incubator-airflow/pull/1934 (this is kind of important from the Airbnb
> side
> > so that we can help test the new master without having to cherrypick this
> > PR on top of it which would make the testing unreliable for others).
> >
> > If there are PRs that affect the core of Airflow that other committers
> > think are important to merge we could include these too. I can commit to
> > pushing out the Impersonation/Cgroups PR this week pending PR comments.
> > What do you think Bolke?
> >
> > On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> > > Hey Alex,
> > >
> > > I have noticed the same, and it is also the reason why we have Alpha
> > > versions. For now I have noticed the following:
> > >
> > > * Tasks can get in limbo between scheduler and executor:
> > > https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948>
> > > * Try_number not increased due to reset in LocalTaskJob:
> > > https://github.com/apache/incubator-airflow/pull/1969 <
> > > https://github.com/apache/incubator-airflow/pull/1969>
> > > * one_failed trigger not executed
> > >
> > > My idea is to move to a Samba style of releases eventually, but for
> now I
> > > would like to get master into a state that we understand and therefore
> > not
> > > accept any patches that do not address any bugs.
> > >
> > > If you (or anyone else) can review the above PRs and add your own as
> well
> > > then I can create another Alpha version. I’ll be on gitter as much as I
> > can
> > > so we can speed up if needed.
> > >
> > > - Bolke
> > >
> > > > On 3 Jan 2017, at 08:51, Alex Van Boxel <alex@vanboxel.be> wrote:
> > > >
> > > > Hey Bolke,
> > > >
> > > > thanks for getting this moving. But I already have some blockers,
> > since I
> > > > moved up master to this release (moved from end November to now)
> > > stability
> > > > has gone down (certainly on Celary). I'm trying to identify the core
> > > > problems and see if I can fix them.
> > > >
> > > > On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin <bdbruin@gmail.com
> > > <mailto:bdbruin@gmail.com>> wrote:
> > > >
> > > > Dear All,
> > > >
> > > > On the verge of the New Year, I decided to be a little bit cheeky and
> > to
> > > > make available an Airflow 1.8.0 Alpha 1. We have been talking about
> it
> > > for
> > > > a long time now and by doing this I wanted bootstrap the process. It
> > > should
> > > > by no means be considered an Apache release yet. This is for testing
> > > > purposes in the dev community around Airflow, nothing else.
> > > >
> > > > The build is exactly the same as the state of master (git 410736d)
> plus
> > > the
> > > > change to version “1.8.0.alpha1” in version.py.
> > > >
> > > > I am dedicating quite some time next week and beyond to get a release
> > > out.
> > > > Hopefully we can get some help with testing, changelog etc. To make
> > this
> > > > possible I would like to propose a freeze to adding new features for
> at
> > > > least two weeks - say until Jan 15.
> > > >
> > > > You can find the tar here: http://people.apache.org/~bolke/ <
> > > > http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/>>
> .
> > > It isn’t signed. Following versions
> > > > will be. SHA is available.
> > > >
> > > > Lastly, Alpha 1 does not have the fix for retries yet. So we will get
> > an
> > > > Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in
> > > > https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948> <
> > > > https://github.com/apache/incubator-airflow/pull/1948 <
> > > https://github.com/apache/incubator-airflow/pull/1948>> , but your
> > > feedback
> > > > is required as it is entrenched in new processing code that you are
> > > running
> > > > in production afaik - so I wonder what happens in your fork.
> > > >
> > > > Happy New Year!
> > > >
> > > > Bolke
> > > >
> > > >
> > > >
> > > > --
> > > >  _/
> > > > _/ Alex Van Boxel
> > >
> > >
> >
> --
>   _/
> _/ Alex Van Boxel
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message