Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Bolke de Bruin <bdbruin@gmail.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Date: Sun, 10 Sep 2017 22:09:49 +0200
Subject: Re: 1.9.0 PR mini-rundown
Message-Id: <B903CAA3-BFB1-4342-BD5E-0DFF2FAE8128@gmail.com>
References: <CABYbY7dXuVPv5LsLhgcON42Mp6=TjS-WPQSwchsw5Xj54kiNHA@mail.gmail.com> <F1397416-354A-4ACE-8542-A26D31A00BEB@gmail.com> <CAHEEp7U6Fi8dmxZVqSLTZcKyASR8RLHWWfO2mv3jWr4Efz0ivQ@mail.gmail.com> <CAL+gSoJTAwLhxze+K1xNMTKRRRXDaE_VjnOOUBmccYdqihfFdg@mail.gmail.com> <ABEF89A9-4939-41CC-BA9C-32F717E10F76@firemirror.com> <CABYbY7f6cbQkfBbN0d7cfC59Q5XfOZ5GjDwakmGd-sG4+6-JCw@mail.gmail.com> <11EC7029-AD94-4ECC-A0E2-8ED66A60D122@gmail.com> <CABYbY7cen_gepP4CJ_=0ch5RgLkJLv5gqOjJi4V3VJg1o7mZaA@mail.gmail.com> <CAMdzn8ujBHyAzasHF77824rmgK=9W7rXE1Cjg6rj1LJFSvSHYw@mail.gmail.com> <CAE1CdVK79FT5UNE9LU7Mtpa4NO5dQm6R7eSvQiVNELA6i5cRCQ@mail.gmail.com> <521C97EB-5486-4B25-9B01-E54EB6A045FD@gmail.com> <CABYbY7e6+cpqc4_03q37nNgPnW+QWk_AMyCxA=hbsQU5mTQzfA@mail.gmail.com> <CABYbY7eoZj+0VA-ygL+EE7u86af8n1gsAu5D51x390OuFnUrNw@mail.gmail.com> <225D061A-F32D-4662-9388-3EF9130B9018@firemirror.com>
In-Reply-To: <225D061A-F32D-4662-9388-3EF9130B9018@firemirror.com>
To: dev@airflow.incubator.apache.org
archived-at: Sun, 10 Sep 2017 20:09:57 -0000

Nice run down. There is also a security patch we need to integrate (also to 1=
.8.3).=20

Bolke

Verstuurd vanaf mijn iPad

> Op 8 sep. 2017 om 11:31 heeft Ash Berlin-Taylor <ash_airflowlist@firemirro=
r.com> het volgende geschreven:
>=20
> There are a few PRs I would also like to see in this release, I've gone th=
rough the open Github pull requests and picked some out. I've only listed on=
es that I would accept if I was reviewing them.
>=20
> One of these two (one fixes the specific Celery setting I needed, the othe=
r allows much more general customization of Celery config):
>=20
> - https://github.com/apache/incubator-airflow/pull/2572
> - https://github.com/apache/incubator-airflow/pull/2542 or https://github.=
com/apache/incubator-airflow/pull/2558 (both address the same Jira in slight=
ly different ways.)
>=20
>=20
> https://github.com/apache/incubator-airflow/pull/2578/files looks like it m=
ight be an important fix for a change on master, though is lacking tests cov=
ering the bug.
>=20
> https://github.com/apache/incubator-airflow/pull/2540 `Variable.setdefault=
` raises an exception when the variable already exists(!), at least when usi=
ng JSON.
>=20
>=20
> ### Two PRs that look good but aren't quite ready yet
>=20
> https://github.com/apache/incubator-airflow/pull/2327/files The `--ignore_=
all_dependencies` flag wasn't working. (This seems to be failing due to a mi=
ssing license, but otherwise looks good and is an important fix. I have aske=
d the author to add the missing license to the new test file.)
>=20
> https://github.com/apache/incubator-airflow/pull/2531 Server-side paging o=
f DAGS - helps with large airflow instances. (Though this is currently marke=
d WIP, so I have asked the author for clarification on if this is ready or n=
ot)
>=20
> https://github.com/apache/incubator-airflow/pull/2562 - adding the 1.8.2 c=
hangelog to master branch.
>=20
>=20
>=20
> ### And then some nice to haves/simple changes:
>=20
> https://github.com/apache/incubator-airflow/pull/2532 Migrate the S3Hook t=
o use boto3.
>=20
> https://github.com/apache/incubator-airflow/pull/2509/files a tiny refacto=
r to move a calculation only to where it is needed.
>=20
> https://github.com/apache/incubator-airflow/pull/2479 fix syntax errors in=
 the docs
>=20
> https://github.com/apache/incubator-airflow/pull/2405 exposes a new option=
 to `airflow worker` for `--celery_hostname` to support running multiple wor=
kers on the same node.
>=20
> https://github.com/apache/incubator-airflow/pull/2507/files - properly cle=
ars hidden fields in the Connection UI when changing type.
>=20
> https://github.com/apache/incubator-airflow/pull/2237/files - Fix issue wi=
th Gunicorn 19.4 and upgrade to it. Probably a good idea to not be stuck on o=
lder versions of modules? (has conflicts, but its two one-line changes so lo=
oks like it would be easy to resolve at merge time.)
>=20
> https://github.com/apache/incubator-airflow/pull/2189/files - replace an i=
nstance of hardcoded `scheduled_` prefix with ID_PREFIX variable. No semanti=
c meaning but seems to be the last instance of this string that isn't using t=
he variable.
>=20
> I hope this was helpful.
>=20
> -ash
>=20
>=20
>> On 7 Sep 2017, at 20:26, Chris Riccomini <criccomini@apache.org> wrote:
>>=20
>> Double negative for the win. Rephrase: Highly unlikely K8s will make 1.9.=
0.
>>=20
>> On Thu, Sep 7, 2017 at 12:26 PM, Chris Riccomini <criccomini@apache.org>
>> wrote:
>>=20
>>> Highly unlikely K8s will not make 1.9.0.
>>>=20
>>>> On Thu, Sep 7, 2017 at 12:19 PM, Bolke de Bruin <bdbruin@gmail.com> wro=
te:
>>>>=20
>>>> Depends on how fast it stabilizes...
>>>>=20
>>>> Sent from my iPhone
>>>>=20
>>>>> On 7 Sep 2017, at 21:11, Sergei Iakhnin <llevar@gmail.com> wrote:
>>>>>=20
>>>>> The Kubernetes features would be exciting to have. Do you think these
>>>> will
>>>>> make it into this release?
>>>>>=20
>>>>>> On Thu, Sep 7, 2017 at 8:22 PM Bolke de Bruin <bdbruin@gmail.com>
>>>> wrote:
>>>>>>=20
>>>>>> Would be appreciated!
>>>>>>=20
>>>>>> Op 7 sep. 2017 8:06 p.m. schreef "Chris Riccomini" <
>>>> criccomini@apache.org
>>>>>>> :
>>>>>>=20
>>>>>>> @Bolke, I can wait to cut the test branch until that PR is merged, i=
f
>>>> you
>>>>>>> like.
>>>>>>>=20
>>>>>>> On Thu, Sep 7, 2017 at 10:19 AM, Bolke de Bruin <bdbruin@gmail.com>
>>>>>> wrote:
>>>>>>>=20
>>>>>>>> One of my team members is preparing a comprehensive PR that reworks=

>>>> all
>>>>>>>> logging to use the standard framework that was recently introduced.=

>>>> It
>>>>>> is
>>>>>>>> expected to be become public tomorrow. So please don't cut too earl=
y
>>>>>> ;-).
>>>>>>>> In addition it would be nice to have a PR clean up? Some PRs have
>>>> been
>>>>>>>> waiting for some time.
>>>>>>>>=20
>>>>>>>> BTW: we will maintain 1.8.X for bug fixes until  1.10/2.0 comes
>>>> along.
>>>>>>>>=20
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>>>=20
>>>>>>>> Sent from my iPhone
>>>>>>>>=20
>>>>>>>>> On 7 Sep 2017, at 18:55, Chris Riccomini <criccomini@apache.org>
>>>>>>> wrote:
>>>>>>>>>=20
>>>>>>>>> Hey all,
>>>>>>>>>=20
>>>>>>>>> The workflow is documented here:
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow=

>>>>>>>>>=20
>>>>>>>>> I'll probably cut the 1.9.0 test branch today or tomorrow, dependi=
ng
>>>>>> on
>>>>>>>>> schedule. I'll let folks know when it's cut via dev@.
>>>>>>>>>=20
>>>>>>>>>> Once the branch is cut does that mean no more PRs will be include=
d
>>>>>> in
>>>>>>>> the
>>>>>>>>> release, or can some changes still be cherry-picked/merged in?
>>>>>>>>>=20
>>>>>>>>> We can still cherry-pick PRs in. Very quickly, though, we begin
>>>>>>> rejecting
>>>>>>>>> any PRs that aren't bug fixes. It's a big exercise to stabilize
>>>> these
>>>>>>>>> releases, so I will start rejecting merging non-bug fix PRs pretty=

>>>>>>>> quickly.
>>>>>>>>> Probably within a couple of weeks.
>>>>>>>>>=20
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>>=20
>>>>>>>>> On Thu, Sep 7, 2017 at 9:31 AM, Ash Berlin-Taylor <
>>>>>>>>> ash_airflowlist@firemirror.com> wrote:
>>>>>>>>>=20
>>>>>>>>>> What is the workflow? Once the branch is cut does that mean no mo=
re
>>>>>>> PRs
>>>>>>>>>> will be included in the release, or can some changes still be
>>>>>>>>>> cherry-picked/merged in?
>>>>>>>>>>=20
>>>>>>>>>> (Is this documented somewhere I can read?)
>>>>>>>>>>=20
>>>>>>>>>> Oh, and thing someone in gitter chat pointed out: the CHANGELOG o=
n
>>>>>>>> master
>>>>>>>>>> doesn't include entries for 1.8.2.
>>>>>>>>>>=20
>>>>>>>>>> -ash
>>>>>>>>>>> On 7 Sep 2017, at 17:28, Andrew Chen <andrewchen@databricks.com>=

>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>> Any estimate when branch 1.9 will be cut?
>>>>>>>>>>>=20
>>>>>>>>>>> On Thu, Sep 7, 2017 at 9:25 AM, Maxime Beauchemin <
>>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> Go for it! Still trying to wrap up 1.8.2 ...
>>>>>>>>>>>>=20
>>>>>>>>>>>> Max
>>>>>>>>>>>>=20
>>>>>>>>>>>> On Thu, Sep 7, 2017 at 8:31 AM, Bolke de Bruin <
>>>> bdbruin@gmail.com
>>>>>>>=20
>>>>>>>>>> wrote:
>>>>>>>>>>>>=20
>>>>>>>>>>>>> Awesome!
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> On 7 Sep 2017, at 17:04, Chris Riccomini <
>>>> criccomini@apache.org
>>>>>>>=20
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> I want to get the ball rolling on 1.9.0, now that 1.8.2 is ou=
t
>>>>>> the
>>>>>>>>>>>> door.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> @Max, I recall you mentioning driving this, but if you're bur=
nt
>>>>>>> out
>>>>>>>>>>>> after
>>>>>>>>>>>>>> the 1.8.2 work, Joy and I can drive 1.9.0. Just let me know.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>>=20
>>>>> --
>>>>>=20
>>>>> Sergei
>>>>=20
>>>=20
>>>=20
>=20