airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] YingboWang commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
Date Sat, 05 Oct 2019 18:05:38 GMT
YingboWang commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
URL: https://github.com/apache/airflow/pull/5499#issuecomment-538674356
 
 
   @seelmann  We are deploying the smart sensor inside our company right now. Testing and
monitoring it for further refinement. I will update the code for this PR and AIP document
with the overall structure and some statistics based on our testing for the past several weeks.

   
   @BasPH @feng-tao, Thanks for your feedbacks. It's my bad not update this PR on time. 
   The smart sensor operator itself is only used to create a control dag for taking over all
sensors in the airflow cluster when the cluster is enable to "use_smart_sensor" by configuration.
The smart sensor task is running as a service keep refreshing all active sensor tasks from
DB and poking them. Users doesn't need to change their original sensor tasks when writing
DAGs. 
   
   When use smart sensor, the sensor tasks state changing is different with regular sensor.
The scheduler will persist sensor's poke context to DB and set the state to "Smart_pending"
so that the task will not be queued and taking a separate process in any worker. Smart sensor
task instance which is a long running process on the work will refresh DB to get all sensors
that are in "Smart" state (smart_running or smart_pending) and poke them. If the poke return
True it will set the corresponding task_instance state to "success" so that the scheduler
will be able to kick off the downstream task_instance in the next parsing loop. There is only
a cluster configuration change for using smart sensor. No need to write DAGs in a different
way. I will update the PR and the description with latest change and also update the AIP to
include all details for smart sensor design. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message