airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: Good jinja templating in airflow examples
Date Thu, 19 Jan 2017 14:22:27 GMT
Maxime, I have a related question. Can you explain how template_dict work
with PythonOperator? Documentation is very vague about it. Example would be
nice. I could not find a way to use jinja template with Python operator
i.e. passing templated parameters to a pythonoperator callable. Someone
told me the only way to do it is to create a wrapper function and use xcom
or macro from there

On Wed, Jan 18, 2017 at 10:09 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> Here's the list of variables and macros exposed in the jinja context:
> https://airflow.apache.org/code.html#macros
>
> Those are all exposed by the framework, meaning they are usable for any
> templated field without doing anything special.
>
> There are ways to pass your own variables and methods using the `params`
> attribute of any operator as shown in the tutorial here
> <https://airflow.apache.org/tutorial.html#example-pipeline-definition>,
> and
> there's also a `user_defined_macros` parameter as a dict you can pass while
> creating the DAG object. Every key in that dictionary is made available in
> the global jinja namespace.
>
> https://airflow.apache.org/code.html#airflow.models.DAG
>
> Max
>
> On Wed, Jan 18, 2017 at 11:31 AM, Boris Tyukin <boris@boristyukin.com>
> wrote:
>
> > Hi Guilherme,
> >
> > I guess it depends what exactly you want to do as not everything works
> with
> > jinja.
> >
> > From documentation:
> >
> > https://pythonhosted.org/airflow/concepts.html#jinja-templating
> >
> > You can use Jinja templating with every parameter that is marked as
> > “templated” in the documentation.
> >
> > You can open source code for operators and see what parameters are
> actually
> > templated.
> >
> > For example, if you open source code for BashOperator
> > https://pythonhosted.org/airflow/_modules/bash_
> operator.html#BashOperator
> > you will see this line of code:
> > template_fields = ('bash_command', 'env')
> >
> > It means only bash_command and env will work with jinja templates.
> >
> > On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
> > guilherme.marthe@enjoei.com.br> wrote:
> >
> > > Hey folks!
> > >
> > > I am trying to write a dag that works well with jinja templating, since
> > > through my study of the documentation, is the best way to ensure
> > > compatibility with the back-fill function.
> > >
> > > Are you guys aware of any examples online with this functionality
> > working?
> > > The, documentation is still a bit dry on insightful examples, and I am
> a
> > > noobie developer, so I am trying to make the best use of the tool in
> the
> > > "correct way".
> > >
> > > Thank you in advance for any resources you guys can share!
> > >
> > > Gui
> > >
> > > Ps: I ve been told I can email you guys with questions like these :)
> Hope
> > > this is not a hassle ^_^
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message