Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Bolke de Bruin <bdbruin@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_F9265C9D-B46E-45E6-A0D4-9BE1B8CB940F"
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Re: 1.8.1 release
Date: Wed, 5 Apr 2017 20:35:26 +0200
References: <CABYbY7e_AAHspfwkb5eXG=YLu9nap+ndVubpqwrFmtiRgK4vfg@mail.gmail.com>
 <5AF0B499-6CEB-4593-821B-291CB4B9F5DD@gmail.com>
 <3D5ACB81-209E-42B3-88AB-1DB6D13AE177@gmail.com>
 <CABYbY7exde2JbRsfLOgnz+eXAnR_zNnErUXLUOBwsgQ+=irCKQ@mail.gmail.com>
 <CABekQG5ouSq1TXAxOLzHdPzgmw_HbpSzFW3pRtudqQN6YK2KfA@mail.gmail.com>
 <CACdThQ2NwSWuBkM3C8a05S9NPExZEhU=kzndMunR0HXdEavAvg@mail.gmail.com>
 <CANLtMic2rduCjNRWJgUyG5kzKhRL8eY7BWstrs2hYWZSsieR3A@mail.gmail.com>
 <CACdThQ2UcAXmyiGr=dd8=LDB+XubwRFdkzvataC2WNVq=yNGig@mail.gmail.com>
 <CABYbY7fcR9xiMTdrU5a=s8W5zEwxispZOV1NE0Y6kP-aukcGoQ@mail.gmail.com>
 <0FE3ABD5-CB18-4D18-8B50-65391B59C8C7@gmail.com>
 <CABYbY7f0SysFsKUFRjcQ3PaDqVVFNsXibOL0bffc5kVvrXL-fw@mail.gmail.com>
 <CABYbY7dRYiGc_Dty20NL0wpoTmf2wpiR8nNXDo1ajHO85n4OXA@mail.gmail.com>
 <CANLtMidVknumA1qpXT=mtRqCdmHqF-=UxoVvfgBRK7B+Gg0SBg@mail.gmail.com>
 <CANLtMicMiFN=9F02MmQ2vEQR8j=dMLjfEn8bmHbF+uhBsptnCw@mail.gmail.com>
 <07507B63-C330-4C3C-BEEB-3613FE3705E8@gmail.com>
 <FF9329A2-BEF3-48C0-95EB-6B409D235CCE@gmail.com>
 <0E11AE55-EA83-4F2C-89CE-D4AC913C8E12@gmail.com>
 <CACdThQ3pWqVa9dR-S8pFRfZZ-ZJ029Jsjvk1AG4umiaohLhknQ@mail.gmail.com>
 <CABYbY7fKCtMhjXmgFJ_mnw2c4-QoA+9zvNc35o-qY4yAspQWwg@mail.gmail.com>
To: dev@airflow.incubator.apache.org
In-Reply-To: <CABYbY7fKCtMhjXmgFJ_mnw2c4-QoA+9zvNc35o-qY4yAspQWwg@mail.gmail.com>
Message-Id: <8EB1C0FB-293F-454A-B61C-33DDFA12FF1E@gmail.com>
archived-at: Wed, 05 Apr 2017 18:35:42 -0000

--Apple-Mail=_F9265C9D-B46E-45E6-A0D4-9BE1B8CB940F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

I have a PR out for this (AIRFLOW-1033). =
https://github.com/apache/incubator-airflow/pull/2220

 It seems large but that is due to activation of the ti_deps tests. See =
for the actual change: =
https://github.com/apache/incubator-airflow/pull/2220/commits/e16678103bf7=
502e98689cd32f3636cbb67e492f#diff-09c898c9cd54512de2fb3a05bae9724c =
<https://github.com/apache/incubator-airflow/pull/2220/commits/e16678103bf=
7502e98689cd32f3636cbb67e492f#diff-09c898c9cd54512de2fb3a05bae9724c>

Bolke

> On 5 Apr 2017, at 19:16, Chris Riccomini <criccomini@apache.org> =
wrote:
>=20
> I've added it to 1.8.1 as a blocker.
>=20
> On Wed, Apr 5, 2017 at 10:11 AM, Ruslan Dautkhanov =
<dautkhanov@gmail.com>
> wrote:
>=20
>> https://issues.apache.org/jira/browse/AIRFLOW-1033
>>=20
>> Is also 1.8 and not 1.8.0 blocker ?
>>=20
>> Thanks.
>>=20
>>=20
>>=20
>> --
>> Ruslan Dautkhanov
>>=20
>> On Mon, Apr 3, 2017 at 3:20 PM, Bolke de Bruin <bdbruin@gmail.com> =
wrote:
>>=20
>>> Hi Chris,
>>>=20
>>> Please add:
>>>=20
>>> https://issues.apache.org/jira/browse/AIRFLOW-1011 <
>>> https://issues.apache.org/jira/browse/AIRFLOW-1011> (PR:
>>> https://github.com/apache/incubator-airflow/pull/2179 <
>>> https://github.com/apache/incubator-airflow/pull/2179>)
>>>=20
>>> To the blocker list. It was missed due to the =E2=80=9CAirflow =
1.8=E2=80=9D affected
>>> version tag instead of =E2=80=9C1.8.0=E2=80=9D. I fixed that for =
you.
>>>=20
>>> Bolke.
>>>=20
>>>> On 3 Apr 2017, at 22:06, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>>>=20
>>>> I have a PR out for
>>>>=20
>>>> * AIRFLOW-1001: =
https://github.com/apache/incubator-airflow/pull/2213
>> <
>>> https://github.com/apache/incubator-airflow/pull/2213>
>>>> * AIRFLOW-1018: =
https://github.com/apache/incubator-airflow/pull/2212
>> <
>>> https://github.com/apache/incubator-airflow/pull/2212>
>>>>=20
>>>> Furthermore, Sid, I checked AIRFLOW-1053 and while it is annoying I
>>> don=E2=80=99t think it is a blocker: it happens only with @once dags =
that have a
>>> SLA, hardly very common. Nevertheless a fix would be nice obviously.
>>>>=20
>>>> Bolke
>>>>=20
>>>>> On 3 Apr 2017, at 11:05, Bolke de Bruin <bdbruin@gmail.com =
<mailto:
>>> bdbruin@gmail.com>> wrote:
>>>>>=20
>>>>> Hey Chris,
>>>>>=20
>>>>> AIRFLOW-1000 has a PR out. You might want to discuss it on the =
list
>>> what the impact is though.
>>>>> AIRFLOW-1018 I=E2=80=99ll have a PR out for shortly that allows =
=E2=80=9Cstdout=E2=80=9D for
>>> the scheduler log files. I did downgrade from blocker to critical.
>>>>> AIRFLOW-719 Has a PR, including much needed increased test =
coverage,
>>> that I am pretty sure is working, but needs verification (plz @Alex)
>>>>>=20
>>>>> I would downgrade AIRFLOW-1019 to critical - especially as Dan is =
not
>>> around at the moment.
>>>>>=20
>>>>> Furthermore, do you have any timelines for getting to the release? =
Can
>>> I help in any way? You might want to chase a couple of times ;-).
>>>>>=20
>>>>> Bolke
>>>>>=20
>>>>>> On 30 Mar 2017, at 19:48, siddharth anand <sanand@apache.org
>> <mailto:
>>> sanand@apache.org>> wrote:
>>>>>>=20
>>>>>> Chris,
>>>>>> I've submitted PRs for :
>>>>>>=20
>>>>>> - PR [AIRFLOW-1013] :
>>>>>> https://github.com/apache/incubator-airflow/pull/2203 <
>>> https://github.com/apache/incubator-airflow/pull/2203>
>>>>>> - PR [AIRFLOW-1054]:
>>>>>> https://github.com/apache/incubator-airflow/pull/2201 <
>>> https://github.com/apache/incubator-airflow/pull/2201>
>>>>>>=20
>>>>>> And filed a blocker for a new issue. Essentially, @once DAGs =
cannot
>> be
>>>>>> created if catchup=3DFalse :
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1055 <
>>> https://issues.apache.org/jira/browse/AIRFLOW-1055>
>>>>>>=20
>>>>>> I have a PR that works for this, but will need to add unit tests =
for
>>> it as
>>>>>> well as for AIRFLOW-1013.
>>>>>>=20
>>>>>> -s
>>>>>>=20
>>>>>> On Wed, Mar 29, 2017 at 3:24 PM, siddharth anand =
<sanand@apache.org>
>>> wrote:
>>>>>>=20
>>>>>>> Didn't realize =
https://issues.apache.org/jira/browse/AIRFLOW-1013
>>> was a
>>>>>>> blocker. I will have a PR shortly.
>>>>>>> -s
>>>>>>>=20
>>>>>>> On Wed, Mar 29, 2017 at 2:07 PM, Chris Riccomini <
>>> criccomini@apache.org>
>>>>>>> wrote:
>>>>>>>=20
>>>>>>>> The following three JIRAs were not merged into the v1-8-test
>> branch,
>>> but
>>>>>>>> are listed as part of the 1.8.1 release:
>>>>>>>>=20
>>>>>>>> AIRFLOW-1017 b2b9587cca9195229ab107394ad94b7702c70e37
>>>>>>>> AIRFLOW-906 bc47200711be4d2c0b36b772651dae4f5e01a204
>>>>>>>> AIRFLOW-858 94dc7fb0a6bb3c563d9df6566cd52a59bd0c4629
>>>>>>>> AIRFLOW-832 b0ae70d3a8e935dc9266b6853683ae5375a7390b
>>>>>>>>=20
>>>>>>>> I'm going to merge them in now.
>>>>>>>>=20
>>>>>>>> On Wed, Mar 29, 2017 at 1:53 PM, Chris Riccomini <
>>> criccomini@apache.org>
>>>>>>>> wrote:
>>>>>>>>=20
>>>>>>>>> Hey Bolke,
>>>>>>>>>=20
>>>>>>>>> Great. Assuming your PR is committed, that leaves five =
blockers:
>>>>>>>>>=20
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1000
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1001
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1013
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1018
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1019
>>>>>>>>>=20
>>>>>>>>> I've also got a list of all open 1.8.1 JIRAs [1].
>>>>>>>>>=20
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>>=20
>>>>>>>>> [1] https://issues.apache.org/jira/issues/?jql=3Dproject%20%
>>>>>>>>> 3D%20AIRFLOW%20AND%20status%20in%20(Open%2C%20%22In%
>>>>>>>>> 20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.8.1
>>>>>>>>>=20
>>>>>>>>> On Mon, Mar 27, 2017 at 8:59 PM, Bolke de Bruin <
>> bdbruin@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>=20
>>>>>>>>>> Hi Chris,
>>>>>>>>>>=20
>>>>>>>>>> I have a PR out for
>>>>>>>>>>=20
>>>>>>>>>> * Revert of 719, which makes 982 obsolete and removes 983 =
from
>> the
>>>>>>>>>> blockers list and just a new feature.
>>>>>>>>>>=20
>>>>>>>>>> See: https://github.com/apache/incubator-airflow/pull/2195 <
>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2195>
>>>>>>>>>>=20
>>>>>>>>>> Cc: @alexvanboxel
>>>>>>>>>>=20
>>>>>>>>>> Bolke
>>>>>>>>>>=20
>>>>>>>>>>> On 24 Mar 2017, at 10:21, Chris Riccomini <
>> criccomini@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>> Hey all,
>>>>>>>>>>>=20
>>>>>>>>>>> I've let this thread sit for a while. Here are a list of the
>>> issues
>>>>>>>> that
>>>>>>>>>>> were raised:
>>>>>>>>>>>=20
>>>>>>>>>>> BLOCKERS:
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-982
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-983
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1019
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1017
>>>>>>>>>>>=20
>>>>>>>>>>> NICE TO HAVE:
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1015
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1013
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1004
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1003
>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1001
>>>>>>>>>>>=20
>>>>>>>>>>> It looks like AIRFLOW-1017 is done, though the JIRA is not
>> closed.
>>>>>>>>>>>=20
>>>>>>>>>>> The rest remain open. I will wait on the release until the
>>> remaining
>>>>>>>>>>> blockers are finished. Dan/Daniel, can you comment on =
status?
>>>>>>>>>>>=20
>>>>>>>>>>> Ruslan, if you want to work on your nice to haves, and =
submit
>>>>>>>> patches,
>>>>>>>>>>> that's great, otherwise I don't believe they'll get fixed as
>> part
>>> of
>>>>>>>>>> 1.8.1.
>>>>>>>>>>>=20
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>>=20
>>>>>>>>>>> On Wed, Mar 22, 2017 at 9:19 AM, Ruslan Dautkhanov <
>>>>>>>>>> dautkhanov@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> Thank you Sid!
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Ruslan
>>>>>>>>>>>>=20
>>>>>>>>>>>> On Wed, Mar 22, 2017 at 12:01 AM, siddharth anand <
>>>>>>>> sanand@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>=20
>>>>>>>>>>>>> Ruslan,
>>>>>>>>>>>>> Thanks for sharing this list. I can pick a few up. I agree =
we
>>>>>>>> should
>>>>>>>>>> aim
>>>>>>>>>>>> to
>>>>>>>>>>>>> get some of them into 1.8.1.
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> -s
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> On Tue, Mar 21, 2017 at 2:29 PM, Ruslan Dautkhanov <
>>>>>>>>>> dautkhanov@gmail.com
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Some of the issues I ran into while testing 1.8rc5 :
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1015
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1013
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1004
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1003
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1001
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1015
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> It would be great to have at least some of them fixed in
>> 1.8.1.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ruslan Dautkhanov
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> On Tue, Mar 21, 2017 at 3:02 PM, Dan Davydov <
>>>>>>>> dan.davydov@airbnb.com
>>>>>>>>>> .
>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Here is my list for targeted 1.8.1 fixes:
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-982
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-983
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1019 (and =
in
>>>>>>>> general
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> slow
>>>>>>>>>>>>>>> startup time from this new logic of orphaned/reset task)
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1017 =
(which I
>>> will
>>>>>>>>>>>>>> hopefully
>>>>>>>>>>>>>>> have a fix out for soon just finishing up tests)
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> We are also hitting a new issue with subdags with rc5 =
that
>> we
>>>>>>>>>> weren't
>>>>>>>>>>>>>>> hitting with rc4 where subdags will occasionally just =
hang
>>> (had
>>>>>>>> to
>>>>>>>>>>>> roll
>>>>>>>>>>>>>>> back from rc5 to rc4), I'll try to spin up a JIRA for it
>> soon
>>>>>>>> which
>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>> be on the list too.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> On Tue, Mar 21, 2017 at 1:54 PM, Chris Riccomini <
>>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> Agreed. I'm looking for a list of checksums/JIRAs that =
we
>>> want
>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> bugfix release.
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> On Tue, Mar 21, 2017 at 12:54 PM, Bolke de Bruin <
>>>>>>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> On 21 Mar 2017, at 12:51, Bolke de Bruin <
>>> bdbruin@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> My suggestion, as we are using semantic versioning =
is:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> 1) no new features in the 1.8 branch
>>>>>>>>>>>>>>>>>> 2) only bug fixes in the 1.8 branch
>>>>>>>>>>>>>>>>>> 3) new features to land in 1.9
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> This allows companies to
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Have a "known" version and can move to the new branch =
when
>>> they
>>>>>>>>>>>>> want
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> get new features. Obviously we only support N-1, so =
when
>>> 1.10
>>>>>>>>>>>> comes
>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> stop supporting 1.8.X.
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> On 21 Mar 2017, at 11:22, Chris Riccomini <
>>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> I suggest that we start a 1.8.1 Airflow release now. =
The
>>> goal
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> be:
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> 1) get a second release under our belt
>>>>>>>>>>>>>>>>>>> 2) patch known issues with the 1.8.0 release
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> I'm happy to run it, but I saw Maxime mentioning =
that
>>> Airbnb
>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>> to.
>>>>>>>>>>>>>>>>>>> @Max et al, can you comment?
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> Also, can folks supply JIRAs for stuff that think =
needs
>>> to be
>>>>>>>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> 1.8.1
>>>>>>>>>>>>>>>>>>> bugfix release?
>>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>=20
>>>>=20
>>>=20
>>>=20
>>=20


--Apple-Mail=_F9265C9D-B46E-45E6-A0D4-9BE1B8CB940F--