airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Proposal for new task state "WAITING_ON_CALLBACK"
Date Wed, 08 Feb 2017 13:33:46 GMT
On this proposal? No, not yet. Just popped in my mind yesterday. API there is a bit on the
wiki.

> On 8 Feb 2017, at 14:31, Jeremiah Lowin <jlowin@apache.org> wrote:
> 
> Makes a lot of sense. At the NY meetup there was considerable interest in
> using the API (and quite a few hacks around exposing the CLI!) -- is there
> more complete documentation anywhere?
> 
> Thanks Bolke
> 
> On Wed, Feb 8, 2017 at 1:36 AM Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
>> Hi All,
>> 
>> Now that we have an API in place. I would like to propose a new state for
>> tasks named “WAITING_ON_CALLBACK”. Currently, we have tasks that have a
>> kind of polling mechanism (ie. Sensors) that wait for an action to happen
>> and check if that action happened by regularly polling a particular
>> backend. This will always use a slot from one of the workers and could
>> starve an airflow cluster for resources. What if a callback to Airflow
>> could happen that task to change its status by calling a callback mechanism
>> without taking up a worker slot. A timeout could (should) be associated
>> with the required callback so that the task can fail if required. So a bit
>> more visual:
>> 
>> 
>> Task X from DAG Z  does some work and sets “WAITING_ON_CALLBACK” -> API
>> post to /dags/Z/dag_runs/20170101T00:00:00/tasks/X with payload “set status
>> to SUCCESS”
>> 
>> DAG Z happily continues.
>> 
>> Or
>> 
>> Task X from DAG Z sets “WAITING_ON_CALLBACK” with timeout of 300s -> time
>> passes -> scheduler sets task to FAILED.
>> 
>> 
>> Any thoughts?
>> 
>> - Bolke


Mime
View raw message