From dev-return-4378-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Mon Jan 8 14:15:24 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id ED6A1180607 for ; Mon, 8 Jan 2018 14:15:24 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id DD7AF160C2C; Mon, 8 Jan 2018 13:15:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2E8C2160C1E for ; Mon, 8 Jan 2018 14:15:24 +0100 (CET) Received: (qmail 93953 invoked by uid 500); 8 Jan 2018 13:15:23 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 93936 invoked by uid 99); 8 Jan 2018 13:15:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Jan 2018 13:15:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 63B371805D6 for ; Mon, 8 Jan 2018 13:15:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.229 X-Spam-Level: ** X-Spam-Status: No, score=2.229 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=driesprongen-nl.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Ov-Kq-ygEqYv for ; Mon, 8 Jan 2018 13:15:20 +0000 (UTC) Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id CC58B5F24E for ; Mon, 8 Jan 2018 13:15:19 +0000 (UTC) Received: by mail-wm0-f41.google.com with SMTP id b76so14259479wmg.1 for ; Mon, 08 Jan 2018 05:15:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=driesprongen-nl.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=cwVE9dfL/dMuNVvRbrQZcGJ/v+Xex4GEXe2nfF3aNms=; b=vuhH8AeTRFS739Cwkf1T1sw2iPgKMh4Gep47dcllJE85u6gg/jcKJR85maCJDJhJB2 aNkg3SkHXJliz0udBtIyV2kU2avJ5PQkY9AwMNzrqJGdAINfI06s4joF33wdQ6+t3235 pg0fndxc5Ic8hyLWje4RvoNom1JesqA56Bi2G4Pa9zKaqiTvsmDYApUZwB+JuSmQ0gSJ Jxfy7GZCUYopCoZON1tdYJ46XauMQHpXpRckmZcBH2KuHcmqI+V9xcWtb9DFlryGgs3c wwhGsKr9bZGamY+1ITcmE+cWwPmKJcbbl1SmvupnaeeM3oUVuXXcREDRtA+i0KGKv2K7 iAyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=cwVE9dfL/dMuNVvRbrQZcGJ/v+Xex4GEXe2nfF3aNms=; b=SuJpWP9jmzcHt8TVISEVGft61HqmedcEhr7vznGiIUQ8pehtvuIzixyeSZc0TjEHbd Ac7al14htfBMiw0PELee7hNdaF6RUtcdtVg4PmT5+LyxwpYbUD3WjNMj/Q6AthC/FMs/ j3yUKOyuEtz+FpdvHXLYZqMojk0SCsorfSK6QIyuURnfrFA8HAMXCp3/R5MGc1IkFHQ/ Dq0Zv79YIM3QhJqcT59O4Th0/EVlKDTtvDza/E2Alv60SOopCXqqS4IlGofFaKNns3Gy ADWrxxN38uI+aT2JV4GHq+YoJJHB75x7vXxl57IgmNIj02eIhfZns1HxSmQYQTQift0u MM1A== X-Gm-Message-State: AKGB3mLuHKMxnJ7OZhYFAGMqEjikbicFqp9IsecgCi93cIcaDQSGlDmd FAo8xm3BwXOj84Cmij4hzoW4suqA0i9yLqJLrlNqTtFk X-Google-Smtp-Source: ACJfBotfHNtvimEHsNCXp/An/M86gyFyy4VLbCro+WYVAcnFvyx0egJXxJB2C+Vj0xR5Pj8HpT644YW34mIepRJLJxY= X-Received: by 10.80.146.83 with SMTP id j19mr16721411eda.205.1515417318404; Mon, 08 Jan 2018 05:15:18 -0800 (PST) MIME-Version: 1.0 Sender: fokko@driesprongen.nl Received: by 10.80.153.93 with HTTP; Mon, 8 Jan 2018 05:15:17 -0800 (PST) In-Reply-To: References: From: "Driesprong, Fokko" Date: Mon, 8 Jan 2018 14:15:17 +0100 X-Google-Sender-Auth: esJx5mlxwwEp2Bg0GVp_nLmKBek Message-ID: Subject: Re: Fix on_kill command for operators To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="94eb2c0c3ce8f349a405624396af" --94eb2c0c3ce8f349a405624396af Content-Type: text/plain; charset="UTF-8" Yes, for Spark this should work. Depending on the operator and the implementation: https://github.com/apache/incubator-airflow/blob/3e6babe8ed8f8f281b67aa3f4e03bf3cfc1bcbaa/airflow/contrib/hooks/spark_submit_hook.py#L412-L428 However this is a big change in behaviour. I'm curious about the opinion of others. Cheers, Fokko 2018-01-08 14:12 GMT+01:00 Milan van der Meer < milan.vandermeer@realimpactanalytics.com>: > Any help? :) > > On Thu, Dec 14, 2017 at 8:12 PM, Milan van der Meer < > milan.vandermeer@realimpactanalytics.com> wrote: > > > I recently openend the following PR: https://github.com/apache/ > > incubator-airflow/pull/2877 > > > > The problem is that on_kill is not called for operators when you clear a > > task from the UI. > > Thats problematic when working with ex. spark clusters as the jobs on the > > cluster need to be killed. > > > > The issue is in the core code of Airflow and Im not familiar enough with > > the inner workings there. So I could use some directions on this one from > > people who are familiar. > > > > For more info, check out the PR. > > > > Kind regards, > > Milan > > > > > > -- > > *Milan van der Meer* > > *Real**Impact* Analytics *| *Big Data Consultant > www.realimpactanalytics.com > > *BE *+32 498 45 96 22 <0032498459622>* | Skype *milan.vandermeer.ria > --94eb2c0c3ce8f349a405624396af--