airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.8.0 alpha 2
Date Wed, 04 Jan 2017 19:47:58 GMT
Some issues remain:

* one_failed not executed as dag run is marked failed seemingly prematurely (@chris yes you
should see this, see below for an example that is not working properly), confirmed regression
* celery instability Alex
* Wrong DAG state after failure inside branch

So Alpha 2 is definitely not ready for production, but please do put in your canary dags and
let them run. I am still quite concerned about the scheduler integrity and stability.

- Bolke



one_failed_not_executed.py
==== 
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2016,10,5,19),
    'email': ['airflow@airflow.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(seconds=1),
}

dag = DAG('tutorial', default_args=default_args, schedule_interval='@once')

task1 = BashOperator(
    task_id='first_one',
    bash_command='date',
    dag=dag)

task2 = BashOperator(
    task_id='second_one',
    bash_command='this_should_not_work',
    dag=dag)

task2.set_upstream(task1)



task3 = BashOperator(
    task_id='third_one',
    bash_command='random_command_third',
    dag=dag)

task3.set_upstream(task2)

fail_task = DummyOperator(
    task_id='one_failed',
    trigger_rule='one_failed',
    dag=dag)

fail_task.set_upstream([task1,task2,task3])

> On 4 Jan 2017, at 20:35, Chris Riccomini <criccomini@apache.org> wrote:
> 
> Bolke, can you describe the current state of the alpha 2 release? I saw
> some comments from Alex yesterday about celery instability. If I'm running
> on LocalExecutor, should I be seeing any issues?
> 
> On Wed, Jan 4, 2017 at 8:20 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
>> Hi All,
>> 
>> I have put up Airflow 1.8.0 alpha 2 in https://people.apache.org/~bolke/ <
>> https://people.apache.org/~bolke/> .
>> 
>> Note: This still cannot be considered an Apache release. Working on this.
>> 
>> This build is signed (note it is served over https).
>> 
>> Changes are in the area of scheduler stability.
>> 
>> - Bolke


Mime
View raw message