airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Proposal for new task state "WAITING_ON_CALLBACK"
Date Wed, 08 Feb 2017 06:36:25 GMT
Hi All,

Now that we have an API in place. I would like to propose a new state for tasks named “WAITING_ON_CALLBACK”.
Currently, we have tasks that have a kind of polling mechanism (ie. Sensors) that wait for
an action to happen and check if that action happened by regularly polling a particular backend.
This will always use a slot from one of the workers and could starve an airflow cluster for
resources. What if a callback to Airflow could happen that task to change its status by calling
a callback mechanism without taking up a worker slot. A timeout could (should) be associated
with the required callback so that the task can fail if required. So a bit more visual:


Task X from DAG Z  does some work and sets “WAITING_ON_CALLBACK” -> API post to /dags/Z/dag_runs/20170101T00:00:00/tasks/X
with payload “set status to SUCCESS”

DAG Z happily continues.

Or

Task X from DAG Z sets “WAITING_ON_CALLBACK” with timeout of 300s -> time passes ->
scheduler sets task to FAILED.


Any thoughts?

- Bolke
Mime
View raw message