airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaw, Damian P. " <damian.sha...@credit-suisse.com>
Subject RE: Global Custom Template Variables? Or is there another best practise here?
Date Tue, 28 May 2019 17:46:56 GMT
Thanks, I will test this and if it works generally switch my own non-user facing code to this
approach.

However for exposing Airflow to my end users  it has 2 drawbacks that make it unlikely to
be used: 1) For date variables the approach appears to be to create a function that you pass
the task instance to. 2) The name spacing is more verbose. E.g. My current custom template
variable “local_cal_ds” becomes “macros.local_cal_ds(ti)”
This might seem small but awkward apis slow down or halt adoption when dealing with users
who are only lightly technical.

Regards
Damian

From: James Meickle [mailto:jmeickle@quantopian.com]
Sent: Tuesday, May 28, 2019 1:16 PM
To: users@airflow.apache.org
Subject: Re: Global Custom Template Variables? Or is there another best practise here?

You can do this via the "macros" section in a plugin: https://airflow.apache.org/plugins.html#example

On Tue, May 28, 2019 at 12:42 PM Shaw, Damian P. <damian.shaw.2@credit-suisse.com<mailto:damian.shaw.2@credit-suisse.com>>
wrote:
Hi all,

Is there a way to create template variables (e.g. ‘ds’, ‘ds_nodash’, etc.) that are
available to every situation where Airflow renders templates? Or otherwise is there a best
practice on providing these?

My motivation is I am developing an Airflow environment for my business users. To make it
easy for them I have created several custom date variables (they are set at the end of the
tasks processing period in the timezone of the DAG)  that they can access so they don’t
have to write their own date logic.

My current implementation is to subclass all the operators and sensors I need and to override
the render_template and execute method injecting the extra variables in to the context dictionary.
This has 2 major drawbacks, it needs to be done for every Operator and Sensor we are using,
and it is not available to logic outside Operators and Sensors such as setting email alert
configuration: https://airflow.apache.org/concepts.html#email-configuration

Thanks for any input,
Damian

==============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================


=============================================================================== 
Please access the attached hyperlink for an important electronic communications disclaimer:

http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html 
=============================================================================== 

Mime
View raw message