airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5
Date Wed, 15 Mar 2017 16:19:31 GMT
I have asked Sid to create and Jira and to make it reproducible. Nevertheless, I do not consider
it a blocker as a workaround exists and it is relatively small in scope (while slightly annoying
I understand that). 

Let’s get 1.8 out and do bug fixes in 1.8.1. More bugs will inevitably pop up :).

- Bolke

> On 15 Mar 2017, at 09:04, Chris Riccomini <criccomini@apache.org> wrote:
> 
> Has anyone been able to reproduce Sid's issue?
> 
> On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
>> That is not an airflow error, but a Kerberos error. Try executing the
>> kinit command on the command line by yourself.
>> 
>> Bolke
>> 
>> Sent from my iPhone
>> 
>>> On 14 Mar 2017, at 23:11, Ruslan Dautkhanov <dautkhanov@gmail.com>
>> wrote:
>>> 
>>> `airflow kerberos` is broken in 1.8-rc5
>>> https://issues.apache.org/jira/browse/AIRFLOW-987
>>> Hopefully fix can be part of the 1.8 release.
>>> 
>>> 
>>> 
>>> --
>>> Ruslan Dautkhanov
>>> 
>>>> On Tue, Mar 14, 2017 at 6:19 PM, siddharth anand <sanand@apache.org>
>> wrote:
>>>> 
>>>> FYI,
>>>> I've just hit a major bug in the release candidate related to "clear
>> task"
>>>> behavior.
>>>> 
>>>> I've been running airflow in both stage and prod since yesterday on rc5
>> and
>>>> have reproduced this in both environments. I will file a JIRA for this
>>>> tonight, but wanted to send a note over email as well.
>>>> 
>>>> In my example, I have a 2 task DAG. For a given DAG run that has
>> completed
>>>> successfully, if I
>>>> 1) clear task2 (leaf task in this case), the previously-successful DAG
>> Run
>>>> goes back to Running, requeues, and executes the task successfully. The
>> DAG
>>>> Run the returns from Running to Success.
>>>> 2) clear task1 (root task in this case), the previously-successful DAG
>> Run
>>>> goes back to Running, DOES NOT requeue or execute the task at all. The
>> DAG
>>>> Run the returns from Running to Success though it never ran the task.
>>>> 
>>>> 1) is expected and previous behavior. 2) is a regression.
>>>> 
>>>> The only workaround is to use the CLI to run the task cleared. Here are
>>>> some images :
>>>> *After Clearing the Tasks*
>>>> https://www.dropbox.com/s/wmuxt0krwx6wurr/Screenshot%
>>>> 202017-03-14%2014.09.34.png?dl=0
>>>> 
>>>> *After DAG Runs return to Success*
>>>> https://www.dropbox.com/s/qop933rzgdzchpd/Screenshot%
>>>> 202017-03-14%2014.09.49.png?dl=0
>>>> 
>>>> This is a major regression because it will force everyone to use the CLI
>>>> for things that they would normally use the UI for.
>>>> 
>>>> -s
>>>> 
>>>> 
>>>> -s
>>>> 
>>>> 
>>>>> On Tue, Mar 14, 2017 at 1:32 PM, Daniel Huang <dxhuang@gmail.com>
>> wrote:
>>>>> 
>>>>> +1 (non-binding)!
>>>>> 
>>>>> On Tue, Mar 14, 2017 at 11:35 AM, siddharth anand <sanand@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> +1 (binding)
>>>>>> 
>>>>>> 
>>>>>> On Tue, Mar 14, 2017 at 8:42 AM, Maxime Beauchemin <
>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>> 
>>>>>>> +1 (binding)
>>>>>>> 
>>>>>>> On Tue, Mar 14, 2017 at 3:59 AM, Alex Van Boxel <alex@vanboxel.be>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> +1 (binding)
>>>>>>>> 
>>>>>>>> Note: we had to revert all our ONE_SUCCESS with ALL_SUCCESS
trigger
>>>>>> rules
>>>>>>>> where the parent nodes where joining with a SKIP. But I can
of
>>>> should
>>>>>>> have
>>>>>>>> known this was coming. Apart of that I had a successful run
last
>>>>> night.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Tue, Mar 14, 2017 at 1:37 AM siddharth anand <sanand@apache.org
>>>>> 
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> I'm going to deploy this to staging now. Fab work Bolke!
>>>>>>>> -s
>>>>>>>> 
>>>>>>>> On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov <
>>>> dan.davydov@airbnb.com
>>>>> .
>>>>>>>> invalid
>>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I'll test this on staging as soon as I get a chance (the
testing
>>>> is
>>>>>>>>> non-blocking on the rc5). Bolke very much in particular
:).
>>>>>>>>> 
>>>>>>>>> On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin <
>>>>> jlowin@apache.org>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> +1 (binding) extremely impressed by the work and
diligence all
>>>>>>>>> contributors
>>>>>>>>>> have put in to getting these blockers fixed, Bolke
in
>>>> particular.
>>>>>>>>>> 
>>>>>>>>>> On Mon, Mar 13, 2017 at 1:07 AM Arthur Wiedmer <
>>>>> arthur@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> +1 (binding)
>>>>>>>>>>> 
>>>>>>>>>>> Thanks again for steering us through Bolke.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Arthur
>>>>>>>>>>> 
>>>>>>>>>>> On Sun, Mar 12, 2017 at 9:59 PM, Bolke de Bruin
<
>>>>>> bdbruin@gmail.com
>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Dear All,
>>>>>>>>>>>> 
>>>>>>>>>>>> Finally, I have been able to make the FIFTH
RELEASE
>>>> CANDIDATE
>>>>>> of
>>>>>>>>>> Airflow
>>>>>>>>>>>> 1.8.0 available at: https://dist.apache.org/repos/
>>>>>>>>>>>> dist/dev/incubator/airflow/ <https://dist.apache.org/
>>>>>>>>>>>> repos/dist/dev/incubator/airflow/> , public
keys are
>>>>> available
>>>>>>> at
>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/
>>>>> airflow/
>>>>>> <
>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/
>>>>> airflow/>
>>>>>> .
>>>>>>>> It
>>>>>>>>> is
>>>>>>>>>>>> tagged with a local version “apache.incubating”
so it
>>>> allows
>>>>>>>>> upgrading
>>>>>>>>>>> from
>>>>>>>>>>>> earlier releases.
>>>>>>>>>>>> 
>>>>>>>>>>>> Issues fixed since rc4:
>>>>>>>>>>>> 
>>>>>>>>>>>> [AIRFLOW-900] Double trigger should not kill
original task
>>>>>>> instance
>>>>>>>>>>>> [AIRFLOW-900] Fixes bugs in LocalTaskJob
for double run
>>>>>>> protection
>>>>>>>>>>>> [AIRFLOW-932] Do not mark tasks removed when
backfilling
>>>>>>>>>>>> [AIRFLOW-961] run onkill when SIGTERMed
>>>>>>>>>>>> [AIRFLOW-910] Use parallel task execution
for backfills
>>>>>>>>>>>> [AIRFLOW-967] Wrap strings in native for
py2 ldap
>>>>> compatibility
>>>>>>>>>>>> [AIRFLOW-941] Use defined parameters for
psycopg2
>>>>>>>>>>>> [AIRFLOW-719] Prevent DAGs from ending prematurely
>>>>>>>>>>>> [AIRFLOW-938] Use test for True in task_stats
queries
>>>>>>>>>>>> [AIRFLOW-937] Improve performance of task_stats
>>>>>>>>>>>> [AIRFLOW-933] use ast.literal_eval rather
eval because
>>>>>>>>> ast.literal_eval
>>>>>>>>>>>> does not execute input.
>>>>>>>>>>>> [AIRFLOW-919] Running tasks with no start
date shouldn't
>>>>> break
>>>>>> a
>>>>>>>> DAGs
>>>>>>>>>> UI
>>>>>>>>>>>> [AIRFLOW-897] Prevent dagruns from failing
with unfinished
>>>>>> tasks
>>>>>>>>>>>> [AIRFLOW-861] make pickle_info endpoint be
login_required
>>>>>>>>>>>> [AIRFLOW-853] use utf8 encoding for stdout
line decode
>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is
set for local
>>>>> client
>>>>>>>>>>>> [AIRFLOW-830][AIRFLOW-829][AIRFLOW-88] Reduce
Travis log
>>>>>>> verbosity
>>>>>>>>>>>> [AIRFLOW-794] Access DAGS_FOLDER and SQL_ALCHEMY_CONN
>>>>>> exclusively
>>>>>>>>> from
>>>>>>>>>>>> settings
>>>>>>>>>>>> [AIRFLOW-694] Fix config behaviour for empty
envvar
>>>>>>>>>>>> [AIRFLOW-365] Set dag.fileloc explicitly
and use for Code
>>>>> view
>>>>>>>>>>>> [AIRFLOW-931] Do not set QUEUED in TaskInstances
>>>>>>>>>>>> [AIRFLOW-899] Tasks in SCHEDULED state should
be white in
>>>> the
>>>>>> UI
>>>>>>>>>> instead
>>>>>>>>>>>> of black
>>>>>>>>>>>> [AIRFLOW-895] Address Apache release incompliancies
>>>>>>>>>>>> [AIRFLOW-893][AIRFLOW-510] Fix crashing webservers
when a
>>>>>> dagrun
>>>>>>>> has
>>>>>>>>> no
>>>>>>>>>>>> start date
>>>>>>>>>>>> [AIRFLOW-793] Enable compressed loading in
S3ToHiveTransfer
>>>>>>>>>>>> [AIRFLOW-863] Example DAGs should have recent
start dates
>>>>>>>>>>>> [AIRFLOW-869] Refactor mark success functionality
>>>>>>>>>>>> [AIRFLOW-856] Make sure execution date is
set for local
>>>>> client
>>>>>>>>>>>> [AIRFLOW-814] Fix Presto*CheckOperator.__init__
>>>>>>>>>>>> [AIRFLOW-844] Fix cgroups directory creation
>>>>>>>>>>>> 
>>>>>>>>>>>> No known issues anymore.
>>>>>>>>>>>> 
>>>>>>>>>>>> I would also like to raise a VOTE for releasing
1.8.0 based
>>>>> on
>>>>>>>>> release
>>>>>>>>>>>> candidate 5, i.e. just renaming release candidate
5 to
>>>> 1.8.0
>>>>>>>> release.
>>>>>>>>>>>> 
>>>>>>>>>>>> Please respond to this email by:
>>>>>>>>>>>> 
>>>>>>>>>>>> +1,0,-1 with *binding* if you are a PMC member
or
>>>>> *non-binding*
>>>>>>> if
>>>>>>>>> you
>>>>>>>>>>> are
>>>>>>>>>>>> not.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> Bolke
>>>>>>>>>>>> 
>>>>>>>>>>>> My VOTE: +1 (binding)
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> _/
>>>>>>>> _/ Alex Van Boxel
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 


Mime
View raw message