From dev-return-4952-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Thu Apr 26 20:39:28 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id AE894180648 for ; Thu, 26 Apr 2018 20:39:27 +0200 (CEST) Received: (qmail 2017 invoked by uid 500); 26 Apr 2018 18:39:26 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 2005 invoked by uid 99); 26 Apr 2018 18:39:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Apr 2018 18:39:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 80AECC1D30 for ; Thu, 26 Apr 2018 18:39:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.869 X-Spam-Level: * X-Spam-Status: No, score=1.869 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id h23vBqaxq3wa for ; Thu, 26 Apr 2018 18:39:24 +0000 (UTC) Received: from mail-qk0-f180.google.com (mail-qk0-f180.google.com [209.85.220.180]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 364345F254 for ; Thu, 26 Apr 2018 18:39:24 +0000 (UTC) Received: by mail-qk0-f180.google.com with SMTP id b131so22113774qkg.2 for ; Thu, 26 Apr 2018 11:39:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=qEdxcUaOlJgTodYFu2HXmppyU8bAALVMe+6ONmIN0ak=; b=qPabRgw3z+L+pVg8hHvgr5xq7WLdXIUzJd7Xa9Z3H0VY3i4x69xo7vBBQq4elCo2Gx YIioVjKzjEBH5WKmQds+c7iqMiIkBYmE9uo/GnrYWRSWNN69Q/noz4sFTq0ROLQgDYqt VDhNrRH2H3aL/3540dIOGHd1jdsgvY6EI5V81bcaJ69FiLxh+0qqpE44ZfBEilFM+G8p OLc+HX2GP2fgKOMPx2R5uf3k58oEV+2rzJGxpeKLrQBKqTeyHd2Wtm89W05GDD92of0E PxbAhZFQcDYmhnEw/fefhYXNJmE491lFWjx+uqog+YZp8F8OfnacHSCuQT9BYctjU3Z+ jeDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=qEdxcUaOlJgTodYFu2HXmppyU8bAALVMe+6ONmIN0ak=; b=HxGm0Zmlskvm1hthUVHZqgt1sb0v8ydJkRhKmJPB11gZwQy4afwWJimKD9vCOes2XE 92hF5KHlItcF73jEYLtXJSn2D1bfOp/4orsHwWtI9v99OMdIOD+nQaRh9jMV8XtmQ0/N cDKRXzZyMY653HvS4ZtMzWJKWzkQz6Bp1jnPV169KjNpmXt5PyLlQ624gl1id4KFx6nK Z62DksGKl1W0+2pzas4jPe4kzTRE3NORjMs0wOjIHabOhc1LzOhzSwoRvUeoNDXzOH4s K3QJ6KoE/DjY+yJ+oohpaZwS+n4XcsMmuQNk9wxOyTuXx+uha00ZoRV9GZ2dBbVYY5sf LttA== X-Gm-Message-State: ALQs6tCY1v2Vn1KVET78AOPUc1fehEI8lUpkdLkmckXG+vgHx9axO9W4 fyUlDu03eLxQAcwI3Gmeq2P04+2edI84T6/Z5Urig1Qz X-Google-Smtp-Source: AB8JxZpJfepl0QFNdvwwyrAr/rGyK8bFA2oIWP0n3vuJ8kHAsPNGwlBxOMyDiv6Tv10HH010oxtGu+CJIVaetiM28cs= X-Received: by 10.55.99.133 with SMTP id x127mr35808654qkb.182.1524767962856; Thu, 26 Apr 2018 11:39:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.12.231.139 with HTTP; Thu, 26 Apr 2018 11:39:22 -0700 (PDT) In-Reply-To: References: From: Maxime Beauchemin Date: Thu, 26 Apr 2018 11:39:22 -0700 Message-ID: Subject: Re: About how to pause the running task To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="94eb2c05b21aca84e2056ac4b4ee" --94eb2c05b21aca84e2056ac4b4ee Content-Type: text/plain; charset="UTF-8" There are no semantics or concept of pause within a task, though you can clear/kill tasks which is essentially sending a poison pill to kill the task subprocess. Even if BaseOperator (common to all tasks) was to implement pausing semantics, they probably wouldn't be implemented for most operators. Is it even possible to pause a bash script externally (through SIGTSTP , but does that always work)? a python script? Hive job? A Spark job? Personally I don't think that feature would get much use. You end up freezing and not releasing a worker slot. You can pause a DAG though. FYI when a DAG is paused, it only means it will stop scheduling new tasks. Max On Thu, Apr 26, 2018 at 10:14 AM, Song Liu wrote: > Hi, > > A DAG is composed of many tasks, when this DAG is started, how to pause > the current running task ? > > Thanks, > Song > --94eb2c05b21aca84e2056ac4b4ee--