airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Standish <dpstand...@gmail.com>
Subject Re: reuse dags
Date Wed, 14 Aug 2019 23:13:14 GMT
I would say the main reusable objects in airflow are the operator and the
hook, not the dag.  These are the primary building blocks that you use to
construct your pipelines.

That said you may find it useful to look at TriggerDagRunOperator.  You can
have your dag's behavior change based on execution date and dag_run conf,
which are available to all tasks in as a template context variable.

You may also look into generating your dags dynamically.




On Wed, Aug 14, 2019 at 2:58 PM Nicolas Paris <nicolas.paris@riseup.net>
wrote:

> Hi Damian
>
> > I believe there are some big improvements coming in Airflow 2 for
> subdags
>
> All right, I won't use them until v2 is available.
>
>
> > In terms of what you are saying there you have dags depend on each, we
> use a DummyOperator to be an End Point for the DAG that will always be the
> same name, and ExternalTaskSensor which will wait for the End Point to be
> successfully finished. Through a warning, you have to be on exactly the
> same schedule for this to work or you need to do some datetime conversion
> logic to provide the correct execution_date.
>
> Interesting but quite complex. I am looking for a built-in simple way to
> import dags.
>
> Reusing and factorizing code is quite common need. So far airflow does
> not provide yet a solution for such purpose apparently
>
> On Mon, Aug 12, 2019 at 01:34:11PM +0000, Shaw, Damian P.  wrote:
> > Hi Nicolas,
> >
> > I don't have a lot of experience with subdags as there's a lot of
> warnings around them, I would strongly recommend reading this before using
> them: https://www.astronomer.io/guides/subdags/
> >
> > I believe there are some big improvements coming in Airflow 2 for
> subdags where they will act more like regular dags and work in the way you
> expect.
> >
> > In terms of what you are saying there you have dags depend on each, we
> use a DummyOperator to be an End Point for the DAG that will always be the
> same name, and ExternalTaskSensor which will wait for the End Point to be
> successfully finished. Through a warning, you have to be on exactly the
> same schedule for this to work or you need to do some datetime conversion
> logic to provide the correct execution_date.
> >
> > Regards,
> > Damian
> >
> > -----Original Message-----
> > From: Nicolas Paris [mailto:nicolas.paris@riseup.net]
> > Sent: Sunday, August 11, 2019 7:44 PM
> > To: users@airflow.apache.org
> > Subject: reuse dags
> >
> > Hi
> >
> > From the documentation I guess I spotted the way of reusing dags from
> > other python files: this can be done by creating "subdags".
> >
> > I have created several pipelines (dag1.py, dag2.py...) in several python
> files
> > Right now, I d'like to  build a meta-pipeline dag3 = dag1 >> dag2
> >
> > Do I have to convert dag1 and dag2 to subdag and import them in dag3 ?
> > Or is there a more friendly way
> >
> > Thanks
> > --
> > nicolas
> >
> >
> >
> ===============================================================================
>
> > Please access the attached hyperlink for an important electronic
> communications disclaimer:
> > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> >
> ===============================================================================
>
> >
>
> --
> nicolas
>

Mime
View raw message