airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Napolitano <joseph.napolit...@blueapron.com.INVALID>
Subject Re: Suggested way of passing "input parameters" to a DAG run?
Date Wed, 03 Aug 2016 22:42:03 GMT
What I can say is that we use it a lot, but very lightly.  We basically use
it to communicate the S3 key for a flat file between operators.

Definitely don't use it to send actual data :)

Cheers

On Wed, Aug 3, 2016 at 6:21 PM, Andrew Phillips <andrewp@apache.org> wrote:

> Let me know if that helps, or if I completely misunderstood :)
>>
>
> That helps, indeed - thanks, Joe! We were in fact going down exactly this
> path as an alternative; we were just a bit hesitant to use XComs based on
> the following comment in the docs [1]:
>
> "If it absolutely can’t be avoided, Airflow does have a feature for
> operator cross-communication called XCom that is described elsewhere in
> this document."
>
> The statements talks about sharing information *between* tasks, but we
> weren't sure if this should be read as "stay away from XComs unless there's
> no other option". Curious to hear the community's thoughts on that.
>
> Thanks for the quick response!
>
> ap
>
> [1] https://pythonhosted.org/airflow/concepts.html#operators
>



-- 
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message