Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Bolke de Bruin <bdbruin@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5
Date: Wed, 15 Mar 2017 14:37:31 -0700
References: <B1833A3A-05FB-4112-B395-135CAF930859@gmail.com>
 <CAF+3fpPCNh10FVa+TjLNC0ErkWsLLQDXnsd+LgPJQGR9X-OiDw@mail.gmail.com>
 <CADsgxrGrjFy95Z9n33rCzfV9j_rTb4Ob5k+CuzyJMPEF1cav5g@mail.gmail.com>
 <CABekQG7ZZexwk1gHsbBcDFnmFt+H=9AwDa55MTf7Ji1tu0RcOw@mail.gmail.com>
 <CANLtMifisYtsbUE57vb38qPEhmv-Gg3rwr+jQVyxxt2Le9E0CQ@mail.gmail.com>
 <CALCMntsT2jD=qG6-6bpitzQFa1XG_y3BYcOHJR1bSbkHjyi_fw@mail.gmail.com>
 <CAHEEp7WZPfp=4FM2e+PD9U1sNfeVUS6VtrCe=5i1eOO3d6eubw@mail.gmail.com>
 <CANLtMif5spPUQkZ-2NDRKTz6dvt2NJCrb_0DqzB_En6k9FnqBw@mail.gmail.com>
 <CABDm2Q9yt8vJxJXE=xonFR955uZ=kvSvNywbt=KAERKA6s1pyA@mail.gmail.com>
 <CANLtMifC-AH63z=iD3=O-UxFM6vQ_uG=4B7jED_CyUqFNkFGFg@mail.gmail.com>
 <CACdThQ3ZaVoJW6L_2N+Q9pBGrA5p-+LNMLv69T7cmVmEi7Asdw@mail.gmail.com>
 <3D35429E-4688-4A9F-B531-75BCB6DC2433@gmail.com>
 <CABYbY7dqTLCzwR=9G_q-HAAThhMzD=L3eX2zzWZ6St5ZgrG1AQ@mail.gmail.com>
 <CABYbY7dNHDZFTcizumCr6VMStTcd1i09nexDPXSFFkFE_aaPiw@mail.gmail.com>
 <CANLtMicp3Tj9Or=qMAMG6KSOxupTQ5QM3YbFUh67revaOtUwwA@mail.gmail.com>
 <997BD649-6668-4A8D-8A77-B93D7CF98214@gmail.com>
To: dev@airflow.incubator.apache.org
In-Reply-To: <997BD649-6668-4A8D-8A77-B93D7CF98214@gmail.com>
Message-Id: <D43ECB0D-BEE6-4D27-8285-2E833187582B@gmail.com>
archived-at: Wed, 15 Mar 2017 21:37:39 -0000

On second thought I do consider it a bug and can have a fix out pretty =
quickly, but I don=E2=80=99t consider it a blocker.

- B.

> On 15 Mar 2017, at 14:21, Bolke de Bruin <bdbruin@gmail.com> wrote:
>=20
> Just to be clear: Also in 1.7.1 the DagRun was marked successful, but =
its tasks continued to be scheduled. So one could also consider 1.7.1 =
behaviour a bug. I am not sure here, but I think it kind of makes sense =
to consider the behaviour of 1.7.1 a bug. It has been present throughout =
all the 1.8 rc/beta/apha series.
>=20
> So yes it is a change in behaviour whether it is a regression or an =
integrity improvement is up for discussion. Either way I don=E2=80=99t =
consider it a blocker.
>=20
> Bolke.
>=20
>> On 15 Mar 2017, at 14:06, siddharth anand <sanand@apache.org> wrote:
>>=20
>> Here's the JIRA :
>> https://issues.apache.org/jira/browse/AIRFLOW-989
>>=20
>> I confirmed it is a regression from 1.7.1.3, which I installed via =
pip and
>> tested against the same DAG in the JIRA.
>>=20
>> The issue occurs if a leaf / last / terminal downstream task is not
>> cleared. You won't see this issue if you clear the entire DAG Run or =
clear
>> a task and all of its downstream tasks. If you truly want to only =
clear and
>> rerun a task, but not its downstream tasks, you can use the CLI to =
execute
>> a specific task (e.g. vial airflow run).
>>=20
>> This is a change in behavior -- if we do go ahead with the release, =
then
>> this JIRA should be in a list of JIRAs of known issues related to the =
new
>> version.
>> -s
>>=20
>> On Wed, Mar 15, 2017 at 9:17 AM, Chris Riccomini =
<criccomini@apache.org>
>> wrote:
>>=20
>>> @Sid, does this happen if you clear downstream as well?
>>>=20
>>> On Wed, Mar 15, 2017 at 9:04 AM, Chris Riccomini =
<criccomini@apache.org>
>>> wrote:
>>>=20
>>>> Has anyone been able to reproduce Sid's issue?
>>>>=20
>>>> On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin =
<bdbruin@gmail.com>
>>>> wrote:
>>>>=20
>>>>> That is not an airflow error, but a Kerberos error. Try executing =
the
>>>>> kinit command on the command line by yourself.
>>>>>=20
>>>>> Bolke
>>>>>=20
>>>>> Sent from my iPhone
>>>>>=20
>>>>>> On 14 Mar 2017, at 23:11, Ruslan Dautkhanov =
<dautkhanov@gmail.com>
>>>>> wrote:
>>>>>>=20
>>>>>> `airflow kerberos` is broken in 1.8-rc5
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-987
>>>>>> Hopefully fix can be part of the 1.8 release.
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> --
>>>>>> Ruslan Dautkhanov
>>>>>>=20
>>>>>>> On Tue, Mar 14, 2017 at 6:19 PM, siddharth anand =
<sanand@apache.org>
>>>>> wrote:
>>>>>>>=20
>>>>>>> FYI,
>>>>>>> I've just hit a major bug in the release candidate related to =
"clear
>>>>> task"
>>>>>>> behavior.
>>>>>>>=20
>>>>>>> I've been running airflow in both stage and prod since yesterday =
on
>>>>> rc5 and
>>>>>>> have reproduced this in both environments. I will file a JIRA =
for
>>> this
>>>>>>> tonight, but wanted to send a note over email as well.
>>>>>>>=20
>>>>>>> In my example, I have a 2 task DAG. For a given DAG run that has
>>>>> completed
>>>>>>> successfully, if I
>>>>>>> 1) clear task2 (leaf task in this case), the =
previously-successful
>>> DAG
>>>>> Run
>>>>>>> goes back to Running, requeues, and executes the task =
successfully.
>>>>> The DAG
>>>>>>> Run the returns from Running to Success.
>>>>>>> 2) clear task1 (root task in this case), the =
previously-successful
>>> DAG
>>>>> Run
>>>>>>> goes back to Running, DOES NOT requeue or execute the task at =
all.
>>> The
>>>>> DAG
>>>>>>> Run the returns from Running to Success though it never ran the =
task.
>>>>>>>=20
>>>>>>> 1) is expected and previous behavior. 2) is a regression.
>>>>>>>=20
>>>>>>> The only workaround is to use the CLI to run the task cleared. =
Here
>>> are
>>>>>>> some images :
>>>>>>> *After Clearing the Tasks*
>>>>>>> https://www.dropbox.com/s/wmuxt0krwx6wurr/Screenshot%
>>>>>>> 202017-03-14%2014.09.34.png?dl=3D0
>>>>>>>=20
>>>>>>> *After DAG Runs return to Success*
>>>>>>> https://www.dropbox.com/s/qop933rzgdzchpd/Screenshot%
>>>>>>> 202017-03-14%2014.09.49.png?dl=3D0
>>>>>>>=20
>>>>>>> This is a major regression because it will force everyone to use =
the
>>>>> CLI
>>>>>>> for things that they would normally use the UI for.
>>>>>>>=20
>>>>>>> -s
>>>>>>>=20
>>>>>>>=20
>>>>>>> -s
>>>>>>>=20
>>>>>>>=20
>>>>>>>> On Tue, Mar 14, 2017 at 1:32 PM, Daniel Huang =
<dxhuang@gmail.com>
>>>>> wrote:
>>>>>>>>=20
>>>>>>>> +1 (non-binding)!
>>>>>>>>=20
>>>>>>>> On Tue, Mar 14, 2017 at 11:35 AM, siddharth anand <
>>> sanand@apache.org>
>>>>>>>> wrote:
>>>>>>>>=20
>>>>>>>>> +1 (binding)
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On Tue, Mar 14, 2017 at 8:42 AM, Maxime Beauchemin <
>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>=20
>>>>>>>>>> +1 (binding)
>>>>>>>>>>=20
>>>>>>>>>> On Tue, Mar 14, 2017 at 3:59 AM, Alex Van Boxel =
<alex@vanboxel.be
>>>>=20
>>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>>> +1 (binding)
>>>>>>>>>>>=20
>>>>>>>>>>> Note: we had to revert all our ONE_SUCCESS with ALL_SUCCESS
>>> trigger
>>>>>>>>> rules
>>>>>>>>>>> where the parent nodes where joining with a SKIP. But I can =
of
>>>>>>> should
>>>>>>>>>> have
>>>>>>>>>>> known this was coming. Apart of that I had a successful run =
last
>>>>>>>> night.
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> On Tue, Mar 14, 2017 at 1:37 AM siddharth anand <
>>> sanand@apache.org
>>>>>>>>=20
>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>> I'm going to deploy this to staging now. Fab work Bolke!
>>>>>>>>>>> -s
>>>>>>>>>>>=20
>>>>>>>>>>> On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov <
>>>>>>> dan.davydov@airbnb.com
>>>>>>>> .
>>>>>>>>>>> invalid
>>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> I'll test this on staging as soon as I get a chance (the =
testing
>>>>>>> is
>>>>>>>>>>>> non-blocking on the rc5). Bolke very much in particular :).
>>>>>>>>>>>>=20
>>>>>>>>>>>> On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin <
>>>>>>>> jlowin@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>=20
>>>>>>>>>>>>> +1 (binding) extremely impressed by the work and diligence =
all
>>>>>>>>>>>> contributors
>>>>>>>>>>>>> have put in to getting these blockers fixed, Bolke in
>>>>>>> particular.
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> On Mon, Mar 13, 2017 at 1:07 AM Arthur Wiedmer <
>>>>>>>> arthur@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> +1 (binding)
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Thanks again for steering us through Bolke.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Arthur
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> On Sun, Mar 12, 2017 at 9:59 PM, Bolke de Bruin <
>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>=20
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Dear All,
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Finally, I have been able to make the FIFTH RELEASE
>>>>>>> CANDIDATE
>>>>>>>>> of
>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>> 1.8.0 available at: https://dist.apache.org/repos/
>>>>>>>>>>>>>>> dist/dev/incubator/airflow/ <https://dist.apache.org/
>>>>>>>>>>>>>>> repos/dist/dev/incubator/airflow/> , public keys are
>>>>>>>> available
>>>>>>>>>> at
>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>> airflow/
>>>>>>>>> <
>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>> airflow/>
>>>>>>>>> .
>>>>>>>>>>> It
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> tagged with a local version =E2=80=9Capache.incubating=E2=80=
=9D so it
>>>>>>> allows
>>>>>>>>>>>> upgrading
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> earlier releases.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Issues fixed since rc4:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> [AIRFLOW-900] Double trigger should not kill original =
task
>>>>>>>>>> instance
>>>>>>>>>>>>>>> [AIRFLOW-900] Fixes bugs in LocalTaskJob for double run
>>>>>>>>>> protection
>>>>>>>>>>>>>>> [AIRFLOW-932] Do not mark tasks removed when backfilling
>>>>>>>>>>>>>>> [AIRFLOW-961] run onkill when SIGTERMed
>>>>>>>>>>>>>>> [AIRFLOW-910] Use parallel task execution for backfills
>>>>>>>>>>>>>>> [AIRFLOW-967] Wrap strings in native for py2 ldap
>>>>>>>> compatibility
>>>>>>>>>>>>>>> [AIRFLOW-941] Use defined parameters for psycopg2
>>>>>>>>>>>>>>> [AIRFLOW-719] Prevent DAGs from ending prematurely
>>>>>>>>>>>>>>> [AIRFLOW-938] Use test for True in task_stats queries
>>>>>>>>>>>>>>> [AIRFLOW-937] Improve performance of task_stats
>>>>>>>>>>>>>>> [AIRFLOW-933] use ast.literal_eval rather eval because
>>>>>>>>>>>> ast.literal_eval
>>>>>>>>>>>>>>> does not execute input.
>>>>>>>>>>>>>>> [AIRFLOW-919] Running tasks with no start date shouldn't
>>>>>>>> break
>>>>>>>>> a
>>>>>>>>>>> DAGs
>>>>>>>>>>>>> UI
>>>>>>>>>>>>>>> [AIRFLOW-897] Prevent dagruns from failing with =
unfinished
>>>>>>>>> tasks
>>>>>>>>>>>>>>> [AIRFLOW-861] make pickle_info endpoint be =
login_required
>>>>>>>>>>>>>>> [AIRFLOW-853] use utf8 encoding for stdout line decode
>>>>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is set for local
>>>>>>>> client
>>>>>>>>>>>>>>> [AIRFLOW-830][AIRFLOW-829][AIRFLOW-88] Reduce Travis log
>>>>>>>>>> verbosity
>>>>>>>>>>>>>>> [AIRFLOW-794] Access DAGS_FOLDER and SQL_ALCHEMY_CONN
>>>>>>>>> exclusively
>>>>>>>>>>>> from
>>>>>>>>>>>>>>> settings
>>>>>>>>>>>>>>> [AIRFLOW-694] Fix config behaviour for empty envvar
>>>>>>>>>>>>>>> [AIRFLOW-365] Set dag.fileloc explicitly and use for =
Code
>>>>>>>> view
>>>>>>>>>>>>>>> [AIRFLOW-931] Do not set QUEUED in TaskInstances
>>>>>>>>>>>>>>> [AIRFLOW-899] Tasks in SCHEDULED state should be white =
in
>>>>>>> the
>>>>>>>>> UI
>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>> of black
>>>>>>>>>>>>>>> [AIRFLOW-895] Address Apache release incompliancies
>>>>>>>>>>>>>>> [AIRFLOW-893][AIRFLOW-510] Fix crashing webservers when =
a
>>>>>>>>> dagrun
>>>>>>>>>>> has
>>>>>>>>>>>> no
>>>>>>>>>>>>>>> start date
>>>>>>>>>>>>>>> [AIRFLOW-793] Enable compressed loading in =
S3ToHiveTransfer
>>>>>>>>>>>>>>> [AIRFLOW-863] Example DAGs should have recent start =
dates
>>>>>>>>>>>>>>> [AIRFLOW-869] Refactor mark success functionality
>>>>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is set for local
>>>>>>>> client
>>>>>>>>>>>>>>> [AIRFLOW-814] Fix Presto*CheckOperator.__init__
>>>>>>>>>>>>>>> [AIRFLOW-844] Fix cgroups directory creation
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> No known issues anymore.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> I would also like to raise a VOTE for releasing 1.8.0 =
based
>>>>>>>> on
>>>>>>>>>>>> release
>>>>>>>>>>>>>>> candidate 5, i.e. just renaming release candidate 5 to
>>>>>>> 1.8.0
>>>>>>>>>>> release.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Please respond to this email by:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> +1,0,-1 with *binding* if you are a PMC member or
>>>>>>>> *non-binding*
>>>>>>>>>> if
>>>>>>>>>>>> you
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> not.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> My VOTE: +1 (binding)
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> --
>>>>>>>>>>> _/
>>>>>>>>>>> _/ Alex Van Boxel
>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>=20
>>>>=20
>>>>=20
>>>=20
>=20