airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ludovic Claude <>
Subject Re: Programatically generating DAGs
Date Mon, 16 Jan 2017 17:00:52 GMT
Hello Arvind,

DAGs in Airflow are simple Python objects that only need to be registered
in the globals namespace. It's easy to generate DAGs from some
configuration or database, I'm using that for example to generate several
image processing pipelines based on configuration.

Here is an example:


2017-01-16 12:28 GMT+01:00 Arvind Nedumaran <>:

> Hello everyone,
> I'm new to Airflow and I'm evaluating it for implementation at our org. I
> find that there are a lot of use cases where workflows are not known during
> development. Is there a way to programatically generate, and manage DAGs.
> Better yet, is there a way to manage them directly from a DB instead of
> generating them from templates and managing them as files. We're about to
> spend considerable time implementing this, so I just want to know about the
> various existing options.
> Does it help if the tasks are pre-defined and the DAGs only happen to be
> some permutation of the pre-existing tasks?
> Anybody who's tried something like before, I'd be grateful for any input on
> this issue.
> Thanks,
> Arvind

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message