airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: 1.9.0 PR mini-rundown
Date Sun, 10 Sep 2017 20:09:49 GMT
Nice run down. There is also a security patch we need to integrate (also to 1.8.3). 

Bolke

Verstuurd vanaf mijn iPad

> Op 8 sep. 2017 om 11:31 heeft Ash Berlin-Taylor <ash_airflowlist@firemirror.com>
het volgende geschreven:
> 
> There are a few PRs I would also like to see in this release, I've gone through the open
Github pull requests and picked some out. I've only listed ones that I would accept if I was
reviewing them.
> 
> One of these two (one fixes the specific Celery setting I needed, the other allows much
more general customization of Celery config):
> 
> - https://github.com/apache/incubator-airflow/pull/2572
> - https://github.com/apache/incubator-airflow/pull/2542 or https://github.com/apache/incubator-airflow/pull/2558
(both address the same Jira in slightly different ways.)
> 
> 
> https://github.com/apache/incubator-airflow/pull/2578/files looks like it might be an
important fix for a change on master, though is lacking tests covering the bug.
> 
> https://github.com/apache/incubator-airflow/pull/2540 `Variable.setdefault` raises an
exception when the variable already exists(!), at least when using JSON.
> 
> 
> ### Two PRs that look good but aren't quite ready yet
> 
> https://github.com/apache/incubator-airflow/pull/2327/files The `--ignore_all_dependencies`
flag wasn't working. (This seems to be failing due to a missing license, but otherwise looks
good and is an important fix. I have asked the author to add the missing license to the new
test file.)
> 
> https://github.com/apache/incubator-airflow/pull/2531 Server-side paging of DAGS - helps
with large airflow instances. (Though this is currently marked WIP, so I have asked the author
for clarification on if this is ready or not)
> 
> https://github.com/apache/incubator-airflow/pull/2562 - adding the 1.8.2 changelog to
master branch.
> 
> 
> 
> ### And then some nice to haves/simple changes:
> 
> https://github.com/apache/incubator-airflow/pull/2532 Migrate the S3Hook to use boto3.
> 
> https://github.com/apache/incubator-airflow/pull/2509/files a tiny refactor to move a
calculation only to where it is needed.
> 
> https://github.com/apache/incubator-airflow/pull/2479 fix syntax errors in the docs
> 
> https://github.com/apache/incubator-airflow/pull/2405 exposes a new option to `airflow
worker` for `--celery_hostname` to support running multiple workers on the same node.
> 
> https://github.com/apache/incubator-airflow/pull/2507/files - properly clears hidden
fields in the Connection UI when changing type.
> 
> https://github.com/apache/incubator-airflow/pull/2237/files - Fix issue with Gunicorn
19.4 and upgrade to it. Probably a good idea to not be stuck on older versions of modules?
(has conflicts, but its two one-line changes so looks like it would be easy to resolve at
merge time.)
> 
> https://github.com/apache/incubator-airflow/pull/2189/files - replace an instance of
hardcoded `scheduled_` prefix with ID_PREFIX variable. No semantic meaning but seems to be
the last instance of this string that isn't using the variable.
> 
> I hope this was helpful.
> 
> -ash
> 
> 
>> On 7 Sep 2017, at 20:26, Chris Riccomini <criccomini@apache.org> wrote:
>> 
>> Double negative for the win. Rephrase: Highly unlikely K8s will make 1.9.0.
>> 
>> On Thu, Sep 7, 2017 at 12:26 PM, Chris Riccomini <criccomini@apache.org>
>> wrote:
>> 
>>> Highly unlikely K8s will not make 1.9.0.
>>> 
>>>> On Thu, Sep 7, 2017 at 12:19 PM, Bolke de Bruin <bdbruin@gmail.com>
wrote:
>>>> 
>>>> Depends on how fast it stabilizes...
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On 7 Sep 2017, at 21:11, Sergei Iakhnin <llevar@gmail.com> wrote:
>>>>> 
>>>>> The Kubernetes features would be exciting to have. Do you think these
>>>> will
>>>>> make it into this release?
>>>>> 
>>>>>> On Thu, Sep 7, 2017 at 8:22 PM Bolke de Bruin <bdbruin@gmail.com>
>>>> wrote:
>>>>>> 
>>>>>> Would be appreciated!
>>>>>> 
>>>>>> Op 7 sep. 2017 8:06 p.m. schreef "Chris Riccomini" <
>>>> criccomini@apache.org
>>>>>>> :
>>>>>> 
>>>>>>> @Bolke, I can wait to cut the test branch until that PR is merged,
if
>>>> you
>>>>>>> like.
>>>>>>> 
>>>>>>> On Thu, Sep 7, 2017 at 10:19 AM, Bolke de Bruin <bdbruin@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> One of my team members is preparing a comprehensive PR that
reworks
>>>> all
>>>>>>>> logging to use the standard framework that was recently introduced.
>>>> It
>>>>>> is
>>>>>>>> expected to be become public tomorrow. So please don't cut
too early
>>>>>> ;-).
>>>>>>>> In addition it would be nice to have a PR clean up? Some
PRs have
>>>> been
>>>>>>>> waiting for some time.
>>>>>>>> 
>>>>>>>> BTW: we will maintain 1.8.X for bug fixes until  1.10/2.0
comes
>>>> along.
>>>>>>>> 
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>>> 
>>>>>>>> Sent from my iPhone
>>>>>>>> 
>>>>>>>>> On 7 Sep 2017, at 18:55, Chris Riccomini <criccomini@apache.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hey all,
>>>>>>>>> 
>>>>>>>>> The workflow is documented here:
>>>>>>>>> 
>>>>>>>>> 
>>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow
>>>>>>>>> 
>>>>>>>>> I'll probably cut the 1.9.0 test branch today or tomorrow,
depending
>>>>>> on
>>>>>>>>> schedule. I'll let folks know when it's cut via dev@.
>>>>>>>>> 
>>>>>>>>>> Once the branch is cut does that mean no more PRs
will be included
>>>>>> in
>>>>>>>> the
>>>>>>>>> release, or can some changes still be cherry-picked/merged
in?
>>>>>>>>> 
>>>>>>>>> We can still cherry-pick PRs in. Very quickly, though,
we begin
>>>>>>> rejecting
>>>>>>>>> any PRs that aren't bug fixes. It's a big exercise to
stabilize
>>>> these
>>>>>>>>> releases, so I will start rejecting merging non-bug fix
PRs pretty
>>>>>>>> quickly.
>>>>>>>>> Probably within a couple of weeks.
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>> 
>>>>>>>>> On Thu, Sep 7, 2017 at 9:31 AM, Ash Berlin-Taylor <
>>>>>>>>> ash_airflowlist@firemirror.com> wrote:
>>>>>>>>> 
>>>>>>>>>> What is the workflow? Once the branch is cut does
that mean no more
>>>>>>> PRs
>>>>>>>>>> will be included in the release, or can some changes
still be
>>>>>>>>>> cherry-picked/merged in?
>>>>>>>>>> 
>>>>>>>>>> (Is this documented somewhere I can read?)
>>>>>>>>>> 
>>>>>>>>>> Oh, and thing someone in gitter chat pointed out:
the CHANGELOG on
>>>>>>>> master
>>>>>>>>>> doesn't include entries for 1.8.2.
>>>>>>>>>> 
>>>>>>>>>> -ash
>>>>>>>>>>> On 7 Sep 2017, at 17:28, Andrew Chen <andrewchen@databricks.com>
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Any estimate when branch 1.9 will be cut?
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Sep 7, 2017 at 9:25 AM, Maxime Beauchemin
<
>>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Go for it! Still trying to wrap up 1.8.2
...
>>>>>>>>>>>> 
>>>>>>>>>>>> Max
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 7, 2017 at 8:31 AM, Bolke de
Bruin <
>>>> bdbruin@gmail.com
>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Awesome!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 7 Sep 2017, at 17:04, Chris Riccomini
<
>>>> criccomini@apache.org
>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I want to get the ball rolling on
1.9.0, now that 1.8.2 is out
>>>>>> the
>>>>>>>>>>>> door.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> @Max, I recall you mentioning driving
this, but if you're burnt
>>>>>>> out
>>>>>>>>>>>> after
>>>>>>>>>>>>>> the 1.8.2 work, Joy and I can drive
1.9.0. Just let me know.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> --
>>>>> 
>>>>> Sergei
>>>> 
>>> 
>>> 
> 

Mime
View raw message