airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: Global Custom Template Variables? Or is there another best practise here?
Date Wed, 29 May 2019 08:39:43 GMT
You can slightly improve it by using the @contextfunction decorator from Jinja and user_defined_macros
at the DAG level:

```
from datetime import datetime
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from jinja2 import contextfunction
import pendulum
import airflow.utils.timezone


@contextfunction
def ts_no_tz(context):
    dag = context['dag']

    return context['execution_date'].strftime("%Y-%m-%dT%H:%M:%S")


start = datetime(2019, 2, 18, 0, 0, 0, tzinfo=pendulum.timezone("America/Los_Angeles"))
dag = DAG(
    'test',
    start_date=start,
    catchup=False,
    max_active_runs=1,
    default_args={
        'retries': 2,
        'retry_delay': 60,
    },
    user_defined_macros={'ts_no_tz': ts_no_tz},
)

with dag:
    tasks = [BashOperator(task_id='test_' + str(i), bash_command='echo {{ts}} {{ ts_no_tz()
}}') for i in range(6)]
```

You could also use the policy() hook <http://airflow.apache.org/concepts.html#cluster-policy
<http://airflow.apache.org/concepts.html#cluster-policy>> to set this globally:

```
def policy(task):
    if not task.dag.user_defined_macros:
        task.dag.user_defined_macros = {}
    task.dag.user_defined_macros.setdefault('ts_no_tz', ts_no_tz)
```

HTH,
Ash
      

> On 28 May 2019, at 18:46, Shaw, Damian P. <damian.shaw.2@credit-suisse.com> wrote:
> 
> Thanks, I will test this and if it works generally switch my own non-user facing code
to this approach.
>  
> However for exposing Airflow to my end users  it has 2 drawbacks that make it unlikely
to be used: 1) For date variables the approach appears to be to create a function that you
pass the task instance to. 2) The name spacing is more verbose. E.g. My current custom template
variable “local_cal_ds” becomes “macros.local_cal_ds(ti)”
> This might seem small but awkward apis slow down or halt adoption when dealing with users
who are only lightly technical.
>  
> Regards
> Damian
>                                                                                     
                     
> From: James Meickle [mailto:jmeickle@quantopian.com] 
> Sent: Tuesday, May 28, 2019 1:16 PM
> To: users@airflow.apache.org
> Subject: Re: Global Custom Template Variables? Or is there another best practise here?
>  
> You can do this via the "macros" section in a plugin: https://airflow.apache.org/plugins.html#example
<https://airflow.apache.org/plugins.html#example>
>  
> On Tue, May 28, 2019 at 12:42 PM Shaw, Damian P. <damian.shaw.2@credit-suisse.com
<mailto:damian.shaw.2@credit-suisse.com>> wrote:
> Hi all,
>  
> Is there a way to create template variables (e.g. ‘ds’, ‘ds_nodash’, etc.) that
are available to every situation where Airflow renders templates? Or otherwise is there a
best practice on providing these?
>  
> My motivation is I am developing an Airflow environment for my business users. To make
it easy for them I have created several custom date variables (they are set at the end of
the tasks processing period in the timezone of the DAG)  that they can access so they don’t
have to write their own date logic.
>  
> My current implementation is to subclass all the operators and sensors I need and to
override the render_template and execute method injecting the extra variables in to the context
dictionary. This has 2 major drawbacks, it needs to be done for every Operator and Sensor
we are using, and it is not available to logic outside Operators and Sensors such as setting
email alert configuration:https://airflow.apache.org/concepts.html#email-configuration <https://airflow.apache.org/concepts.html#email-configuration>
>  
> Thanks for any input,
> Damian
>  
> 
> ==============================================================================
> Please access the attached hyperlink for an important electronic communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html <http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html>
> ==============================================================================
> 
> 
> ==============================================================================
> Please access the attached hyperlink for an important electronic communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html <http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html>
> ==============================================================================


Mime
View raw message