airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arpan Dhandhania <ar...@infoworks.io>
Subject Re: Canceling a DAG run
Date Wed, 01 Mar 2017 13:45:51 GMT
So if I want to cancel a DAG run, I will need to update the iterate over the running tasks
and “clear” them?

I can write a function to “clear” the tasks, but I am wondering why it hasn’t been written
thus far. IMO, canceling a DAG would be a basic requirement for any ETL use-case (and others
would have required this too). Since there is no easy way to do this already, I am wondering
if I am trying to do something wrong.

Arpan Dhandhania
e: arpan@infoworks.io
m: +91 782 939 5618

—
Perfection is achieved, not when there is nothing more to add, but when there is nothing left
to take away.

> On Mar 1, 2017, at 12:50 AM, Maxime Beauchemin <maximebeauchemin@gmail.com> wrote:
> 
> Changing the state of a dag run in the UI will only stop the scheduler from
> evaluating that dag run, it does not sigterm or anything like that.
> "Clearing" running task instances (UI/CLI) will result in failing the task
> instances. Also as of 1.8.x if the state of the task in changed externally
> from running to some other state, the task will kill itself.
> 
> Max
> 
> On Tue, Feb 28, 2017 at 9:35 AM, Arpan Dhandhania <arpan@infoworks.io>
> wrote:
> 
>> Hi folks,
>> 
>> We have been using Airflow to run ETL jobs but sometimes require to cancel
>> a DAG run. Since I could not find a way to cancel a DAG run (from CLI or
>> UI), we are using a crude work-around that is to cancel the tasks that are
>> running and thus failing the DAG run.
>> 
>> What I expect (and I might be wrong here) is that there should be a way to
>> cancel a DAG run which would send a SIGTERM signal to the running tasks and
>> after they are terminated, the DAG run should go into a “canceled” or
>> “finished” state.
>> 
>> Is this something that others find useful as well or am I missing
>> something?
>> 
>> Arpan Dhandhania
>> e: arpan@infoworks.io
>> m: +91 782 939 5618
>> 
>> —
>> Perfection is achieved, not when there is nothing more to add, but when
>> there is nothing left to take away.
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message