airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <lance.nors...@gmail.com>
Subject Re: A question for the Airflow community
Date Sat, 06 Aug 2016 04:22:26 GMT
2. There is a feature 'XCom' which allows you to use the Airflow database
as a key-value store. If you wish DAG and task instances to see particular
data items, you can store them via 'XCom'. I have not done this.



On Fri, Aug 5, 2016 at 7:25 PM, Andrew Phillips <andrewp@apache.org> wrote:

> I'd like to ask you all a question in turn: what do you
>> know now that you wish you knew when you first deployed
>> Airflow?
>>
>
> Mainly questions around suggestions/best practice for addressing
> reasonably common (?) integration challenges:
>
> 1. Talking to Airflow from remote/external systems (e.g. on a cloud CI/CD
> service) where the Airflow CLI is not available and/or it is not desirable
> to expose access to the DB directly.
>
> An HTTP API, as already on the roadmap, as far as I know, would be an
> obvious candidate here, I guess, but how are people doing this currently?
> (Cf. [1])
>
> 2. Passing parameters to DAG runs beyond the run ID that remain stable
> across the DAG run. Cf. [2]
>
> A description somewhere of what happens when a DAG run actually takes
> place (when and where is the DAG object instantiated, when are task
> instances instantiated, when does template resolution happen, etc.) would
> have helped us to avoid running into gotchas based on mistaken assumptions.
>
> For example, we had initially assumed that the DAG object would be created
> *once*, at the beginning of the DAG run, and that any parameters passed to
> it would be "frozen" at that time.
>
> As far as we can tell, the DAG object is actually instantiated repeatedly,
> so e.g. the value passed as the params argument is not "frozen" at the
> beginning of the DAG run, and later task instances may see different values
> from earlier task instances.
>
> Thanks for kicking off this initiative!
>
> Regards
>
> ap
>
> [1] http://markmail.org/message/3yowkhufo7hegzoo
> [2] http://markmail.org/message/5jwz7c64ndnhzqyq
>



-- 
Lance Norskog
lance.norskog@gmail.com
Redwood City, CA

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message