airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Trigger DagRun Task based on notification
Date Fri, 01 Feb 2019 06:27:28 GMT
Hi All,

In our workflows we trigger big data jobs which run from few hours to few days. Currently
our Airflow operator submits the job and keeps on polling its status. Depending upon its status
next task in the workflow is triggered by Airflow scheduler.
So currently operator is not doing any useful work but is occupying one worker slot.
we are exploring if we can do following 
 -> Operator Submits the Big Data Job and mark itself success
 -> Once that job finishes it sends a notification either through callback or through some
 -> Based on notification Dependent task in the workflow is triggered and starts running.

We might need to introduce new operator state like waiting ..
Does it make sense?. 
Is there any way/workaround to achieve this.

Raman Gupta

View raw message