airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Batista <>
Subject Halt other DAGs until a certain DAG has completed
Date Thu, 11 May 2017 12:54:05 GMT
Hi everyone,

I asked a similar question before, but I thought about rephrasing/detailing
it with an example, maybe someone else has this issue:

Let's say I have DAG_1 running and for some reason (e.g.: due to failing
and re-tries, took some more time due to huge amounts of data, etc.)
Airflow triggers DAG_2 because the time that matches the crontab for DAG_2
was reached.

What happens now, is that Airflow will start running tasks from bot DAG_1
and DAG_2. In a scenario where there are limited resources, I would like to
put all tasks on DAG_2 on hold until DAG_1 finishes completely.

I know that one way to do this is to have an External Sensor on DAG_2
poking for the success of DAG_1. But, is there a way to tell Airflow, that
even if the time reached to run other DAGs, if a specific DAG still has not
complete put everything else on hold?



[image: logo]   
  <>   <>
   <>   <>   
*HelloFresh App –Download Now!* 
*We're active in:* 
US <>
 |  DE 
<> |  
|  NL 
<> |  
 |  BE 
<> |  
AT <> 
|  CH 
<> | 
CA <> 
We are hiring around the world – Click here to join us 


HelloFresh SE, Berlin (Sitz der Gesellschaft) | Vorstände: Dominik S. 
Richter (Vorsitzender), Thomas W. Griesel, Christian Gärtner | Vorsitzender 
des Aufsichtsrats: Jeffrey Lieberman | Eingetragen beim Amtsgericht 
Charlottenburg, HRB 182382 B | USt-Id Nr.: DE 302210417

*CONFIDENTIALITY NOTICE:* This message (including any attachments) is 
confidential and may be privileged. It may be read, copied and used only by 
the intended recipient. If you have received it in error please contact the 
sender (by return e-mail) immediately and delete this message. Any 
unauthorized use or dissemination of this message in whole or in parts is 
strictly prohibited.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message