airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] yuqian90 commented on issue #6392: [AIRFLOW-5648] Add ClearTaskOperator for clearing tasks in a DAG
Date Thu, 31 Oct 2019 13:53:35 GMT
yuqian90 commented on issue #6392: [AIRFLOW-5648] Add ClearTaskOperator for clearing tasks
in a DAG
URL: https://github.com/apache/airflow/pull/6392#issuecomment-548385907
 
 
   > TBH it feels wrong if you have to clear tasks state on a regular interval! If you
have to do that it means you are doing something wrong in your original DAG and it's task.
   > 
   > We have the `clear` functionality in the UI and CLI so that is something unexpected
happens you can clear the task state and run it again!
   
   Thanks @kaxil . You are right to point out that there's some inherent problems in the tasks.
The underlying problem I often face is that there are some cycles in the tasks that I can't
break in the near future, so we  have to go back and rerun tasks that already finished earlier
on a regular basis. In the [AIRFLOW-5648](https://issues.apache.org/jira/browse/AIRFLOW-5648)
example, if I need to run tasks `A, C, E, G, H, I, J` one more time after `Sensor` passes,
is the best way to do this to copy those tasks into `A1, C1, E1, G1, H1, I1, J1` ? Is there
a better alternative provided by Airflow?
   
   ```
   A >> C >> E >> G >> H >> I >> J >> K >>
L >> M >> Finish
                  ^                   ^          
                  |                   |__________         
   B >> D >> F>>>>                               |
                                                 |
   Sensor >> A1 >> C1 >> E1 >> G1 >> H1 >> I1 >>
J1
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message