airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremiah Lowin <jlo...@apache.org>
Subject Re: Airflow Newbie Question
Date Tue, 21 Mar 2017 19:38:03 GMT
Indeed, Airflow has an implicit assumption that your DAGs are "slowly
changing" at worst. There's absolutely nothing wrong with using a factory
to generate multiple DAGs at once, but as Jorge says you will run in to
trouble if you are generating different DAGs each time the server,
scheduler, executor, or worker parses the file (which is often).

In other words, if your factory function generates 10 DAGs for 10 lob's,
that's great -- just try to have it always be the same 10, or at least make
changes infrequently.

On Tue, Mar 21, 2017 at 2:50 PM Jorge Alpedrinha Ramos <
jalpedrinharamos@gmail.com> wrote:

> Hi Andrew,
>
> I've ran into the same problem and I found out that the best way to do this
> is to use a dag_factory that takes your parameter and builds a dag for each
> lob in the "list".
>
> This way you'll have one dag per lob, and they will run independently as I
> believe its your intention.
>
> This solution will define which dags are active or inactive not at DAG
> runtime, but at scheduler boot time. The problem you may have or not, is if
> the lob "list" is somehow dynamic, which means that your dags will be
> active or inactive depending on the lob being in the "list" at the time the
> scheduler instantiates your dag and not necessarily at the time your dag
> runs.
>
> I hope I've helped, I believe this is a point where best practices are
> still in some kind of grey area and is definitely something airflow should
> have one clear way of doing.
>
>
>
>
> On Tue, Mar 21, 2017 at 4:06 PM Andrew Maguire <andrewm4894@gmail.com>
> wrote:
>
> > Sorry if this is not the right place for this.
> >
> > I'm wondering if anyone might be able to help me with this question?
> >
> >
> >
> http://stackoverflow.com/questions/42930221/best-way-to-loop-through-parameters-in-airflow
> >
> > Feel free to ignore if this is not the right place to try find help.
> >
> > Loving airflow btw!
> >
> > Cheers,
> > Andy
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message