airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-883) Assigning operator to DAG via bitwise composition does not pickup default args
Date Wed, 15 Mar 2017 17:06:41 GMT


Jeremiah Lowin commented on AIRFLOW-883:

I'm not totally sure this is a "bug" per se, though it is confusing. 

"default_args" are arguments that are passed to Operators by the parent DAG. Critically, that
happens when the Operators are created. While bitshift operators allow deferred DAG assignment,
the Operator in question has already been created. The reason the distinction matters is that
the Operator's __init__ may include logic related to its arguments. If we pass/assign those
arguments after initialization, the logic won't run. 

However, if we do want to tackle this:
1. The simplest thing would be to walk "default_args" and replace any matching Operator attributes
that are None.
2. The more proper thing would be to defer Operator initialization until it is added to a
DAG. This would require a bit of a refactor though.

> Assigning operator to DAG via bitwise composition does not pickup default args
> ------------------------------------------------------------------------------
>                 Key: AIRFLOW-883
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: models
>            Reporter: Daniel Huang
>            Assignee: Jeremiah Lowin
>            Priority: Minor
> This is only the case when the operator does not specify {{dag=dag}} and is not initialized
within a DAG's context manager (due to
> Example:
> {code}
> default_args = {
>     'owner': 'airflow', 
>     'start_date': datetime(2017, 2, 1)
> }
> dag = DAG('my_dag', default_args=default_args)
> dummy = DummyOperator(task_id='dummy')
> dag >> dummy
> {code}
> This will raise a {{Task is missing the start_date parameter}}. I _think_ this should
probably be allowed because I assume the purpose of supporting {{dag >> op}} was to
allow delayed assignment of an operator to a DAG. 
> I believe to fix this, on assignment, we would need to go back and go through dag.default_args
to see if any of those attrs weren't explicitly set on task...not the cleanest. 

This message was sent by Atlassian JIRA

View raw message