airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joy Gao <j...@wepay.com>
Subject Re: Multiple similar DAGs
Date Wed, 01 Nov 2017 18:45:52 GMT
Hi Michael,

You could leverage the Variable feature in Airflow: start by setting a
different environment variable in each env, and then in your DAG file,
instead of hard-coding the schedule, call a method that looks up which
environment it is (using Variable.get) and returns the desired one.

Hope this helps!

On Wed, Nov 1, 2017 at 11:10 AM, Michael Crawford <
michael.crawford@modernizingmedicine.com> wrote:

> Hi All,
>
> Is there a best practice regarding registering similar DAGs which only
> differ in small variables.
>
> For instance say I have an certain ETL that I want to run on several
> different environments on different schedules.
>
> All of the DAGs would essentially be the exactly the same just with a few
> different parameters.
>
> It doesn’t seem like duplicating the DAG code over and over is the right
> way to do this.
>
> Thanks,
> Mike
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message