airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Hodgkinson <nik.hodgkin...@collectivehealth.com>
Subject Determine if a DAG is running
Date Wed, 15 Feb 2017 23:51:26 GMT
I'm using the TriggerDagRunOperator within a Sensor DAG to automatically
sense and start loads, however the problem I'm coming across, given that I
want to run this Sensor DAG often (every 5 minutes), is the same run can be
triggered multiple times. Is there a way to avoid this? I've set
max_active_runs to 1, but that doesn't seem to be respected when doing
manual triggers. I suppose I could write out a file when I trigger the job,
have the job clean it up, and then not trigger if that file exists, but I
feel like there should be a better way to accomplish this; not to mention a
failure in the dag would break this process.

Thoughts? I've been trying to figure this out for a while and Gitter and
the web haven't been much help.

-N
nik.hodgkinson@collectivehealth.com
(913) 927-4891

-- 


Read our founder's story. 
<https://collectivehealth.com/blog/started-collective-health/>

*This message may contain confidential, proprietary, or protected 
information.  If you are not the intended recipient, you may not review, 
copy, or distribute this message. If you received this message in error, 
please notify the sender by reply email and delete this message.*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message