airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Or Sher <or.sh...@gmail.com>
Subject Re: Iterating task xcom keys inside template
Date Thu, 16 Jun 2016 06:39:46 GMT
Thanks Jeremiah!

Looking at the code it seems like get_many can return only many xcom values
from different task_ids and not different keys from the same task_id.
But it doesn't really matter as I would really like to do the iteration
somehow inside the template.
I feel like building the template inside the previous python operator is
kind of missing the whole point of templates as I can already create the
whole "already rendered" email message at the same time.

I tried pushing one big json xcom to reflect all of the tests, but now
I'm struggling
with iterating string json inside the template.
The following template fails on the "for" line:
VALIDATION_EMAIL_CONTENT = """
<b>Date:</b> {{ ds }}<br>
<p>
    <table>
    {% for key, value in
task_instance.xcom_pull(task_ids='data_validation_tests',
key='tests_results') %}
        {{ key|string() }}
        {{ value|string() }}
    {% endfor %}
..
..

File "<template>", line 5, in top-level template code
ValueError: too many values to unpack

I'm guessing it's because the xcom_pull returns a string but I didn't
managed to cast it to a json yet..

On Wed, Jun 15, 2016 at 5:33 PM Jeremiah Lowin <jlowin@apache.org> wrote:

> Hi Or,
>
> There is support for pulling multiple XComs at once (see XCom.get_many())
> but it won't be directly available inside a jinja template, where you you
> are typically calling ti.xcom_pull() and returning a single XCom at a time,
> using a key you know in advance.
>
> I would suggest that you do this processing either inside your existing
> PythonOperator or in a subsequent PythonOperator that calls XCom.get_many()
> and produces the final template for your email operator. It is
> theoretically possible for you to do this in your email template using
> template macros, but I only have a basic knowledge of that (and it's a
> convoluted process) so I don't want to give you bad advice. If you can do
> your preprocessing in your PythonOperator I think you will be happier,
> especially because the result will be visible in the Airflow UI and logs.
>
> Best,
> Jeremiah
>
> On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or.sher1@gmail.com> wrote:
>
> > Hi all,
> >
> > For the purpose of data validation tests I have two adjacent operators:
> > Python - Parsing predefined json of tests and saves the result as an xcom
> > where each key is a test id.
> >
> > Email - I'd like to be able to iterate over all keys from the previous
> task
> > inside the predefined email template so I won't need to change it every
> > time I change the tests json.
> >
> > It seems like it's not supported. Is that right?
> > What are my choices?
> > If I won't find anything else, I assume I can always use one key for all
> of
> > the tests combined into one json.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message