airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaw, Damian P. " <damian.sha...@credit-suisse.com>
Subject RE: reuse dags
Date Mon, 12 Aug 2019 13:34:11 GMT
Hi Nicolas,

I don't have a lot of experience with subdags as there's a lot of warnings around them, I
would strongly recommend reading this before using them: https://www.astronomer.io/guides/subdags/


I believe there are some big improvements coming in Airflow 2 for subdags where they will
act more like regular dags and work in the way you expect.

In terms of what you are saying there you have dags depend on each, we use a DummyOperator
to be an End Point for the DAG that will always be the same name, and ExternalTaskSensor which
will wait for the End Point to be successfully finished. Through a warning, you have to be
on exactly the same schedule for this to work or you need to do some datetime conversion logic
to provide the correct execution_date.

Regards,
Damian

-----Original Message-----
From: Nicolas Paris [mailto:nicolas.paris@riseup.net] 
Sent: Sunday, August 11, 2019 7:44 PM
To: users@airflow.apache.org
Subject: reuse dags

Hi

>From the documentation I guess I spotted the way of reusing dags from
other python files: this can be done by creating "subdags".

I have created several pipelines (dag1.py, dag2.py...) in several python files
Right now, I d'like to  build a meta-pipeline dag3 = dag1 >> dag2

Do I have to convert dag1 and dag2 to subdag and import them in dag3 ?
Or is there a more friendly way

Thanks
-- 
nicolas


=============================================================================== 
Please access the attached hyperlink for an important electronic communications disclaimer:

http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html 
=============================================================================== 

Mime
View raw message