airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Paris <nicolas.pa...@riseup.net>
Subject Re: reuse dags
Date Wed, 14 Aug 2019 21:58:29 GMT
Hi Damian

> I believe there are some big improvements coming in Airflow 2 for subdags 

All right, I won't use them until v2 is available.


> In terms of what you are saying there you have dags depend on each, we use a DummyOperator
to be an End Point for the DAG that will always be the same name, and ExternalTaskSensor which
will wait for the End Point to be successfully finished. Through a warning, you have to be
on exactly the same schedule for this to work or you need to do some datetime conversion logic
to provide the correct execution_date.

Interesting but quite complex. I am looking for a built-in simple way to
import dags. 

Reusing and factorizing code is quite common need. So far airflow does
not provide yet a solution for such purpose apparently

On Mon, Aug 12, 2019 at 01:34:11PM +0000, Shaw, Damian P.  wrote:
> Hi Nicolas,
> 
> I don't have a lot of experience with subdags as there's a lot of warnings around them,
I would strongly recommend reading this before using them: https://www.astronomer.io/guides/subdags/

> 
> I believe there are some big improvements coming in Airflow 2 for subdags where they
will act more like regular dags and work in the way you expect.
> 
> In terms of what you are saying there you have dags depend on each, we use a DummyOperator
to be an End Point for the DAG that will always be the same name, and ExternalTaskSensor which
will wait for the End Point to be successfully finished. Through a warning, you have to be
on exactly the same schedule for this to work or you need to do some datetime conversion logic
to provide the correct execution_date.
> 
> Regards,
> Damian
> 
> -----Original Message-----
> From: Nicolas Paris [mailto:nicolas.paris@riseup.net] 
> Sent: Sunday, August 11, 2019 7:44 PM
> To: users@airflow.apache.org
> Subject: reuse dags
> 
> Hi
> 
> From the documentation I guess I spotted the way of reusing dags from
> other python files: this can be done by creating "subdags".
> 
> I have created several pipelines (dag1.py, dag2.py...) in several python files
> Right now, I d'like to  build a meta-pipeline dag3 = dag1 >> dag2
> 
> Do I have to convert dag1 and dag2 to subdag and import them in dag3 ?
> Or is there a more friendly way
> 
> Thanks
> -- 
> nicolas
> 
> 
> =============================================================================== 
> Please access the attached hyperlink for an important electronic communications disclaimer:

> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html 
> =============================================================================== 
> 

-- 
nicolas

Mime
View raw message