airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soma S Dhavala <>
Subject feature request: push/save DAG objects directly
Date Sun, 27 May 2018 04:16:36 GMT
Hi AirFlow Developer,

I am Dr. Soma Dhavala, ML architect. At EkStep <>, a
non-profit organisation, we are developing an open source ML workbench and
we are using AirFlow as one of the core components, on the top of which are
writing an application layer. The idea is to make this tool available to
millions of users who use our SunBird <>

We compose DAGs programmatically - that is an agent auto-composes a DAG
based on a specification (i.e., DAGS are not handcrafted but are generated
based a on template), so we dont have any python script that literally
constructs a DAG object. So it was less than ideal for us to drop .py files
into the DAG folder. An ability to save/write DAG objects directly to the
DB or make them available to the DagBag would ideal.

Right now, we drop a py script that will load pickled DAG objects and makes
them available to the DagBag upon its instantiation. The work around we are
using can be seen here

I am wondering if you have any plans to inject DAG objects via methods
available on the "dag" class or something similar.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message