airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Regarding getting the result of another task in your current task
Date Thu, 22 Dec 2016 21:57:49 GMT
That's right Jeremiah.

Keep in mind that XComs should only be used for small amount of metadata.

In general, pipelines' structure (similarly to data structures) shouldn't
change based on the data that is coming through them. There are ways to
author data pipelines and workflows so that the DAG and tasks logic are
unaffected by what is coming going on inside the pipeline.

What is the nature of the metadata you are moving around with XComs?

Max

On Thu, Dec 22, 2016 at 8:49 AM, Jeremiah Lowin <jlowin@apache.org> wrote:

> Could you provide a little more detail on the type of Operator you want to
> use? It's certainly easiest with PythonOperator. However, the `ti` object
> is also available in jinja templates, so for example you can reference it
> in a BashOperator as (I believe) {{ ti.xcom_pull("t1") }}.
>
> On Thu, Dec 22, 2016 at 11:32 AM twinkle <twinkle.sachdeva@gmail.com>
> wrote:
>
> > Hi,
> >
> > I want to get the result of one task t1, inside task t2.
> >
> > I have set the result of t1, by returning a value, which is set for the
> key
> > result_value.
> >
> > While using the PythonOperator as t2, i am able to get the value of the
> > result of task t1.
> > Code for doing this in PythonOperator is:
> >
> > pull = PythonOperator(
> >
> >     task_id='puller', dag=main_dag, provide_context=True,
> > python_callable=puller)
> >
> >
> > def puller(**kwargs):
> >
> >     ti = kwargs['ti']
> >
> >     v1 = ti.xcom_pull(key=None, task_ids=[t1])
> >
> >     logging.info("v1 :%s" %v1)
> >
> > How can i do that in any other operator?
> > I tried, accessing 'ti' from context as well as kwargs, for both of them
> i
> > get the following error:
> >
> > task_instance = context['ti']
> >
> >     task_instance = context['ti']
> > TypeError: string indices must be integers
> >
> >
> > Any pointers?
> >
> > Regards,
> > Twinkle
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message