airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: Fix on_kill command for operators
Date Mon, 08 Jan 2018 13:15:17 GMT
Yes, for Spark this should work. Depending on the operator and the
implementation:
https://github.com/apache/incubator-airflow/blob/3e6babe8ed8f8f281b67aa3f4e03bf3cfc1bcbaa/airflow/contrib/hooks/spark_submit_hook.py#L412-L428

However this is a big change in behaviour. I'm curious about the opinion of
others.

Cheers,
Fokko


2018-01-08 14:12 GMT+01:00 Milan van der Meer <
milan.vandermeer@realimpactanalytics.com>:

> Any help? :)
>
> On Thu, Dec 14, 2017 at 8:12 PM, Milan van der Meer <
> milan.vandermeer@realimpactanalytics.com> wrote:
>
> > I recently openend the following PR: https://github.com/apache/
> > incubator-airflow/pull/2877
> >
> > The problem is that on_kill is not called for operators when you clear a
> > task from the UI.
> > Thats problematic when working with ex. spark clusters as the jobs on the
> > cluster need to be killed.
> >
> > The issue is in the core code of Airflow and Im not familiar enough with
> > the inner workings there. So I could use some directions on this one from
> > people who are familiar.
> >
> > For more info, check out the PR.
> >
> > Kind regards,
> > Milan
> >
>
>
>
> --
>
> *Milan van der Meer*
>
> *Real**Impact* Analytics *| *Big Data Consultant
> www.realimpactanalytics.com
>
> *BE *+32 498 45 96 22 <0032498459622>* | Skype *milan.vandermeer.ria
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message