airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Iterating task xcom keys inside template
Date Thu, 16 Jun 2016 20:50:29 GMT
I *think* it might be because xcom_pull only returns one element per task:


https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_xcom.py#L35

You're calling with just a single task_id, so I think you should be doing
`for element in ...` rather than `for key, value in ...`.

On Thu, Jun 16, 2016 at 11:51 AM, Jeremiah Lowin <jlowin@apache.org> wrote:

> Or, I'm afraid my jinja-fu is sorely lacking and I'm not going to be much
> help to you here... perhaps Maxime or one of the other devs is more
> experienced with templates and can offer some advice?
>
> Apologies!
> Jeremiah
>
> On Thu, Jun 16, 2016 at 2:40 AM Or Sher <or.sher1@gmail.com> wrote:
>
> > Thanks Jeremiah!
> >
> > Looking at the code it seems like get_many can return only many xcom
> values
> > from different task_ids and not different keys from the same task_id.
> > But it doesn't really matter as I would really like to do the iteration
> > somehow inside the template.
> > I feel like building the template inside the previous python operator is
> > kind of missing the whole point of templates as I can already create the
> > whole "already rendered" email message at the same time.
> >
> > I tried pushing one big json xcom to reflect all of the tests, but now
> > I'm struggling
> > with iterating string json inside the template.
> > The following template fails on the "for" line:
> > VALIDATION_EMAIL_CONTENT = """
> > <b>Date:</b> {{ ds }}<br>
> > <p>
> >     <table>
> >     {% for key, value in
> > task_instance.xcom_pull(task_ids='data_validation_tests',
> > key='tests_results') %}
> >         {{ key|string() }}
> >         {{ value|string() }}
> >     {% endfor %}
> > ..
> > ..
> >
> > File "<template>", line 5, in top-level template code
> > ValueError: too many values to unpack
> >
> > I'm guessing it's because the xcom_pull returns a string but I didn't
> > managed to cast it to a json yet..
> >
> > On Wed, Jun 15, 2016 at 5:33 PM Jeremiah Lowin <jlowin@apache.org>
> wrote:
> >
> > > Hi Or,
> > >
> > > There is support for pulling multiple XComs at once (see
> XCom.get_many())
> > > but it won't be directly available inside a jinja template, where you
> you
> > > are typically calling ti.xcom_pull() and returning a single XCom at a
> > time,
> > > using a key you know in advance.
> > >
> > > I would suggest that you do this processing either inside your existing
> > > PythonOperator or in a subsequent PythonOperator that calls
> > XCom.get_many()
> > > and produces the final template for your email operator. It is
> > > theoretically possible for you to do this in your email template using
> > > template macros, but I only have a basic knowledge of that (and it's a
> > > convoluted process) so I don't want to give you bad advice. If you can
> do
> > > your preprocessing in your PythonOperator I think you will be happier,
> > > especially because the result will be visible in the Airflow UI and
> logs.
> > >
> > > Best,
> > > Jeremiah
> > >
> > > On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or.sher1@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > For the purpose of data validation tests I have two adjacent
> operators:
> > > > Python - Parsing predefined json of tests and saves the result as an
> > xcom
> > > > where each key is a test id.
> > > >
> > > > Email - I'd like to be able to iterate over all keys from the
> previous
> > > task
> > > > inside the predefined email template so I won't need to change it
> every
> > > > time I change the tests json.
> > > >
> > > > It seems like it's not supported. Is that right?
> > > > What are my choices?
> > > > If I won't find anything else, I assume I can always use one key for
> > all
> > > of
> > > > the tests combined into one json.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message