airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Guziel <alex.guz...@airbnb.com.INVALID>
Subject Re: Discussion on Airflow 1.8.1 RC2
Date Thu, 04 May 2017 17:55:11 GMT
I don't think any of the fixes I did were regressions.

On Thu, May 4, 2017 at 8:11 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> I know of one that Alex wanted to get in, but wasn’t targeted for 1.8.1 in
> Jira and thus didn’t make the cut at RC time. There is is another one out
> that seems to have stalled a bit (https://github.com/apache/
> incubator-airflow/pull/2205).
>
> Reading the changelog of 1.8.1 I see bug fixes, apache requirements and
> one “new” feature (UI lightning bolt). Regressions could have happened but
> we have been quite vigilant on the fact that these bug fixes needed proper
> tests, so I am very interested in 1.8.0 -> 1.8.1 regressions. If it is a
> pre-backfill-change 1.8.0 to 1.8.1 regression then I would also like to
> know, cause I made that change and feel responsible for it.
>
> Cheers
> Bolke
>
>
> On 3 May 2017, at 22:13, Dan Davydov <dan.davydov@airbnb.com> wrote:
>
> cc Alex and Rui who were working on fixes, I'm not sure if their commits
> got in before 1.8.1.
>
> On Wed, May 3, 2017 at 1:09 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>
>> Hi Dan,
>>
>> (Thread renamed to make sure it does not clash, dev@ now added)
>>
>> It surprises me that you found regression from 1.8.0 to 1.8.1 as 1.8.1 is
>> very much focused on bug fixes. Were the regressions shared yet?
>>
>> The whole 1.8.X release will be bug fix focused (per release management)
>> and minor feature updates. The 1.9.0 release will be the first release with
>> major feature updates. So what you want, more robustness and focus on
>> stability, is now underway. I agree with beefing up tests and including the
>> major operators in this. Executors should also be on this list btw. Turning
>> on coverage reporting might be a first step in helping this (it isn’t the
>> solution obviously).
>>
>> Cheers
>> Bolke
>>
>>
>> On 3 May 2017, at 20:28, Dan Davydov <dan.davydov@airbnb.com> wrote:
>>
>> We saw several regressions moving from 1.8.0 to 1.8.1 the first time we
>> tried, and while I think we merged all our fixes to master (not sure if
>> they all made it into 1.8.1 however), we have put releasing on hold due to
>> stability issues from the last couple of releases. It's either the case
>> that:
>> A) Airbnb requires more robustness from new releases.
>> or
>> B) Most companies using Airflow require more robustness and we should
>> halt on feature work until we are more confident in our testing
>>
>> I think the biggest problem currently is the lack of unit testing
>> coverage, e.g. when the backfill framework was refactored (which was the
>> right long-term fix), it caused a lot of breakages that weren't caught by
>> tests. I think we need to audit the major operators/classes and beef up the
>> unit testing coverage. The coverage metric does not necessarily cover these
>> cases (e.g. cyclomatic complexity). Writing regression tests is good but we
>> shouldn't have so many new blocker issues in our releases.
>>
>> We are fighting some fires internally at the moment (not Airflow
>> related), but Alex and I have been working on some stuff that we will push
>> to the community once we are done. Alex is working on a good solution for
>> python package isolation, and I'm working on integration with Kubernetes at
>> the executor level.
>>
>> Feel free to forward any of my messages to the dev mailing list.
>>
>> On Wed, May 3, 2017 at 11:18 AM, Bolke de Bruin <bdbruin@gmail.com>
>> wrote:
>>
>>> Grrr, I seriously dislike to send button on the touch bar…here goes
>>> again.
>>>
>>> Hi Dan,
>>>
>>> (Please note I would like to forward the next message to dev@, but let
>>> me know if you don’t find it comfortable)
>>>
>>> I understand your point. The gap between 1.7.1 was large in terms of
>>> functionality changes etc. It was going to be a (bit?) rough and as you
>>> guys are using many of the edge cases you probably found more issues than
>>> any of us. Still, between 1.8.0 and 1.8.1 we have added many tests
>>> (coverage increased from 67% to close to 69%, which is a lot as you know).
>>> It would be nice if you can share where your areas of concern are so we can
>>> address those and a suggestion on how to proceed with integration tests is
>>> also welcome.
>>>
>>> You guys (=Airbnb) have been a bit quiet over the past couple of days,
>>> so I am getting a bit worried in terms of engagement. Is that warranted?
>>>
>>> Cheers
>>> Bolke
>>>
>>>
>>> On 3 May 2017, at 20:13, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>>
>>> Hi Dan,
>>>
>>> (Please note I would like to forward the next message to dev@, but let
>>> me know if you don’t find it comfortable)
>>>
>>> I understand your point. The gap between 1.7.1 was large in terms of
>>> functionality changes etc. It was going to be a (bit?) rough and as you
>>> guys are using many of the edge cases you probably found more issues than
>>> any of us. Still, between 1.8.0 and 1.8.1 we have added many tests
>>> (coverage increased from 67
>>>
>>> On 3 May 2017, at 19:41, Arthur Wiedmer <arthur.wiedmer@airbnb.com>
>>> wrote:
>>>
>>> As a counterpoint,
>>>
>>> I am comfortable voting +1 on this release in the sense that it fixes
>>> some of the issues with 1.8.0. It is unfortunate that we cannot test it on
>>> the Airbnb production for now and we should definitely invest in increasing
>>> testing coverage, but some of the fixes are needed for ease of use/adoption
>>> (See for instance AIRFLOW-832), and this release is a step in the right
>>> direction.
>>>
>>> Best,
>>> Arthur
>>>
>>> On Wed, May 3, 2017 at 10:30 AM, Dan Davydov <dan.davydov@airbnb.com>
>>> wrote:
>>>
>>>> I'm not comfortable voting without doing comprehensive staging and we
>>>> aren't comfortable doing an internal lease for now until we fix the state
>>>> of unit test coverage and integration tests.
>>>>
>>>> On May 3, 2017 8:42 AM, "Bolke de Bruin" <bdbruin@gmail.com> wrote:
>>>>
>>>>> Hey Guys,
>>>>>
>>>>> Chris has been preparing the 1.8.1 release and the vote is running for
>>>>> it. Only one day left though! Would you mind casting your vote? Only
Chris
>>>>> and I have voted binding until so far.
>>>>>
>>>>> (Please reply to the message on the list, not this message).
>>>>>
>>>>> Cheers!
>>>>> Bolke
>>>>>
>>>>> Begin forwarded message:
>>>>>
>>>>> *From: *Chris Riccomini <criccomini@apache.org>
>>>>> *Subject: **[VOTE] Release Airflow 1.8.1 RC2*
>>>>> *Date: *1 May 2017 at 19:58:41 GMT+2
>>>>> *To: *dev@airflow.incubator.apache.org
>>>>> *Reply-To: *dev@airflow.incubator.apache.org
>>>>>
>>>>> Dear All,
>>>>>
>>>>> _WARN: The package version for this RC is 1.8.1 (does not include RC2
>>>>> in
>>>>> version number). As such, any future 1.8.1 installatinos will have to
>>>>> be
>>>>> force installed. PIP will not be able to distinguish between RCs and
>>>>> final
>>>>> versions. Again, you'll have to force install the package. This can be
>>>>> done
>>>>> by adding `--force-reinstall` to your `pip install` commands._
>>>>>
>>>>> I've made Airflow 1.8.1 RC2 available at:
>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow, public keys
>>>>> are
>>>>> available at https://dist.apache.org/repos/
>>>>> dist/release/incubator/airflow.
>>>>>
>>>>> New issues fixed in 1.8.1 RC2:
>>>>>
>>>>> [AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependen
>>>>> [AIRFLOW-1004] `airflow webserver -D` runs in foreground
>>>>> [AIRFLOW-492] Insert into dag_stats table results into failed ta
>>>>>
>>>>> Issues fixed in 1.8.1 RC0/RC1, and included in RC2:
>>>>>
>>>>> [AIRFLOW-1138] Add licenses to files in scripts directory
>>>>> [AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE
>>>>> [AIRFLOW-1124] Do not set all task instances to scheduled on back
>>>>> [AIRFLOW-1120] Update version view to include Apache prefix
>>>>> [AIRFLOW-1062] DagRun#find returns wrong result if external_trigg
>>>>> [AIRFLOW-1054] Fix broken import on test_dag
>>>>> [AIRFLOW-1050] Retries ignored - regression
>>>>> [AIRFLOW-1033] TypeError: can't compare datetime.datetime to None
>>>>> [AIRFLOW-1017] get_task_instance should return None instead of th
>>>>> [AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs
>>>>> [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) fo
>>>>> [AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow
>>>>> [AIRFLOW-989] Clear Task Regression
>>>>> [AIRFLOW-974] airflow.util.file mkdir has a race condition
>>>>> [AIRFLOW-906] Update Code icon from lightning bolt to file
>>>>> [AIRFLOW-858] Configurable database name for DB operators
>>>>> [AIRFLOW-853] ssh_execute_operator.py stdout decode default to A
>>>>> [AIRFLOW-832] Fix debug server
>>>>> [AIRFLOW-817] Trigger dag fails when using CLI + API
>>>>> [AIRFLOW-816] Make sure to pull nvd3 from local resources
>>>>> [AIRFLOW-815] Add previous/next execution dates to available def
>>>>> [AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/jo
>>>>> [AIRFLOW-812] Scheduler job terminates when there is no dag file
>>>>> [AIRFLOW-806] UI should properly ignore DAG doc when it is None
>>>>> [AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_C
>>>>> [AIRFLOW-785] ImportError if cgroupspy is not installed
>>>>> [AIRFLOW-784] Cannot install with funcsigs > 1.0.0
>>>>> [AIRFLOW-780] The UI no longer shows broken DAGs
>>>>> [AIRFLOW-777] dag_is_running is initlialized to True instead of
>>>>> [AIRFLOW-719] Skipped operations make DAG finish prematurely
>>>>> [AIRFLOW-694] Empty env vars do not overwrite non-empty config v
>>>>> [AIRFLOW-139] Executing VACUUM with PostgresOperator
>>>>> [AIRFLOW-111] DAG concurrency is not honored
>>>>> [AIRFLOW-88] Improve clarity Travis CI reports
>>>>>
>>>>> I would like to raise a VOTE for releasing 1.8.1 based on release
>>>>> candidate
>>>>> 2.
>>>>>
>>>>> Please respond to this email by:
>>>>>
>>>>> +1,0,-1 with *binding* if you are a PMC member or *non-binding* if you
>>>>> are
>>>>> not.
>>>>>
>>>>> Vote will run for 72 hours (ends this Thursday).
>>>>>
>>>>> Thanks!
>>>>> Chris
>>>>>
>>>>> My VOTE: +1 (binding)
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Arthur Wiedmer
>>> (Pronouns: He, Him)
>>> Data Engineering, Airbnb
>>> www.airbnb.com
>>>
>>>
>>>
>>>
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message