airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth <srikanth...@gmail.com>
Subject ExternalTaskSensor usage
Date Fri, 30 Jun 2017 17:51:56 GMT
Hello,

I have a use case where a couple of dags(dag A & B) run at set intervals.
Both dags have sensors that trigger a BashOperator. I have a third dag that
is a long running process that needs to be started once(I use '@once').
This dag depends both dags A and B to be run at least once. So I added
ExternalTaskSensors. Is there a better way to do this?

I noticed that the ExternalTaskSensor poke() never succeeded. Looking at
the code, there seems to be an explicit check for execution time match.
https://github.com/apache/incubator-airflow/blob/7b620391a4e71fd19dff037a859dd39f132edb8c/airflow/operators/sensors.py#L246
Comment for "execution_delta" seems to suggest that this check should be
for an execution time after certain reference. Should't the check be
for "TI.execution_date
> dttm,"?

If this is a bug, I'd be happy to fix and PR. If not, what is the intent
behind this??

Srikanth

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message