airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Gozali <i...@lecida.com>
Subject Re: Canceling a DAG run
Date Mon, 15 May 2017 18:08:47 GMT
Hello!

Sorry for reviving an old thread, but after following the conversation, I'm
still not fully clear on how to reliably cancel a DAG run.

What's the *least hacky* way to reliably cancel the entire DAG run in
Airflow 1.8.x? I tried using the Web UI, going to Graph View of some DAG,
and *clear*ing the very first task by clicking on the node, and clicking
Clear (and confirming on the prompt page), but the scheduler seems to keep
retrying the cleared task instance. Is this because to cancel entire DAG
runs using this method of *clearing task instances*, I have to explicitly
disable retries on tasks?

Appreciate any insight!


--
Regards,


Ivan Gozali
Lecida
Email: ivan@lecida.com

On Wed, Mar 1, 2017 at 11:24 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> You can clear manually from the CLI or the UI. You can also do it in Python
> code using `DAG.clear`, or in other languages by reverse-engineering the
> UI's endpoint [until we offer a proper REST endpoint].
>
> Max
>
> On Wed, Mar 1, 2017 at 5:45 AM, Arpan Dhandhania <arpan@infoworks.io>
> wrote:
>
> > So if I want to cancel a DAG run, I will need to update the iterate over
> > the running tasks and “clear” them?
> >
> > I can write a function to “clear” the tasks, but I am wondering why it
> > hasn’t been written thus far. IMO, canceling a DAG would be a basic
> > requirement for any ETL use-case (and others would have required this
> too).
> > Since there is no easy way to do this already, I am wondering if I am
> > trying to do something wrong.
> >
> > Arpan Dhandhania
> > e: arpan@infoworks.io
> > m: +91 782 939 5618
> >
> > —
> > Perfection is achieved, not when there is nothing more to add, but when
> > there is nothing left to take away.
> >
> > > On Mar 1, 2017, at 12:50 AM, Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> > >
> > > Changing the state of a dag run in the UI will only stop the scheduler
> > from
> > > evaluating that dag run, it does not sigterm or anything like that.
> > > "Clearing" running task instances (UI/CLI) will result in failing the
> > task
> > > instances. Also as of 1.8.x if the state of the task in changed
> > externally
> > > from running to some other state, the task will kill itself.
> > >
> > > Max
> > >
> > > On Tue, Feb 28, 2017 at 9:35 AM, Arpan Dhandhania <arpan@infoworks.io>
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> We have been using Airflow to run ETL jobs but sometimes require to
> > cancel
> > >> a DAG run. Since I could not find a way to cancel a DAG run (from CLI
> or
> > >> UI), we are using a crude work-around that is to cancel the tasks that
> > are
> > >> running and thus failing the DAG run.
> > >>
> > >> What I expect (and I might be wrong here) is that there should be a
> way
> > to
> > >> cancel a DAG run which would send a SIGTERM signal to the running
> tasks
> > and
> > >> after they are terminated, the DAG run should go into a “canceled”
or
> > >> “finished” state.
> > >>
> > >> Is this something that others find useful as well or am I missing
> > >> something?
> > >>
> > >> Arpan Dhandhania
> > >> e: arpan@infoworks.io
> > >> m: +91 782 939 5618
> > >>
> > >> —
> > >> Perfection is achieved, not when there is nothing more to add, but
> when
> > >> there is nothing left to take away.
> > >>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message