From dev-return-8114-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Fri Apr 12 15:42:34 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4CD07180621 for ; Fri, 12 Apr 2019 17:42:34 +0200 (CEST) Received: (qmail 85090 invoked by uid 500); 12 Apr 2019 15:42:33 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 85078 invoked by uid 99); 12 Apr 2019 15:42:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Apr 2019 15:42:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 76146180FAE for ; Fri, 12 Apr 2019 15:42:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.8 X-Spam-Level: * X-Spam-Status: No, score=1.8 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Wj_bTMoZfMll for ; Fri, 12 Apr 2019 15:42:30 +0000 (UTC) Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 1160F5FC18 for ; Fri, 12 Apr 2019 15:42:30 +0000 (UTC) Received: by mail-wm1-f66.google.com with SMTP id z6so746614wmi.0 for ; Fri, 12 Apr 2019 08:42:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=LdC2RE3t9g7PAUeEWn5rVLW+HEC3qgCYX6DVxgIAXY8=; b=sfw0guIX/TjIj7aMRC8uko2hmtDuICgsUSHEvRaWcvKrFo+zgFeFoqeTw3J/h83qeX Z3PYfCK0Aa22M1zlfnBgfeF60TK4Gu73/mAzdql7fYzLpTc9H7yVxcvg5uOy6FWsK0Kw iVyF4zcwrkJdokKFT7t/0pzrUFFhaT0NHuqx0J4+SvV4UoHL154WTjjKPYA6tk4Tee// F4OD7OvYg/617UgVLKhSIy4DdHyZDyDzVk+HkdA9m+VND4hiLieWiXUyVXeA5s9sfZ8t FCMjVGsQuj2fttSLGdo/T4glVbQp6+hShw1qUpbqsF5GknYwiMRAAyKCL+eslGvAYKsC 5zGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=LdC2RE3t9g7PAUeEWn5rVLW+HEC3qgCYX6DVxgIAXY8=; b=cvD4ZTX0CwUYuak0iMr5hzFu4/AO0yGyQA34/zO/MbVC1HYtrDxf8GyZu2UI0/Oo+6 UQoh2TtstFJeFRPZxYzdLAYRE+41FoVoRE5t6jrfU15ZY4HJpte3G2z4MT+DkO8SgJju OftU2Wr2b3XapPBUekZmFa4uFUFS0z+jiECLsyjaT8ibuQexI+0EGsWIJjPwilBbwzzf qjOCO0Lko8CF4oEUQEZJkazK5/tiuDwxeuAZQ3vyVbCAqr5YN3VHHr2ILWFcxypEgDGh YrSwoHxSwGDyXF1XQyaqzK4IHscqqT37b+NlPL1cAiJ+GQFfRTvcrVJ7i6xiSWSRiBvV BKRw== X-Gm-Message-State: APjAAAXNSYV3U8TVLmtkHo+xwtjG6cKFOndYDVmrasLzpXML9ab/hQkG rn6zDSLbToRlE65iG9/Hqfhut0jUzSwrNIjwM8HNcQ== X-Google-Smtp-Source: APXvYqwHvF5nYl2bZulc1Q8V9JHdN/PrRCbVs0r/Av7p4jroaAA2kWdYGyKuaH+X97iZGCALsoDnlf1+ss5SCtXifjo= X-Received: by 2002:a7b:c5ce:: with SMTP id n14mr12051855wmk.18.1555083749585; Fri, 12 Apr 2019 08:42:29 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Chen Tong Date: Fri, 12 Apr 2019 11:42:12 -0400 Message-ID: Subject: Re: [2.0 spring cleaning] Deprecate subdags To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="0000000000007d77c005865726d1" --0000000000007d77c005865726d1 Content-Type: text/plain; charset="UTF-8" Is it possible to re-implement it in the view-level, not in operator level? And this operator is just define a different view in GUI, that these tasks will be collapsed into another view. On Fri, Apr 12, 2019 at 11:31 AM James Meickle wrote: > I have avoided using them because of outstanding issues like the open JIRA > issues I linked above, or similar issues that I've read about on blog > posts. If it were just GUI or UX issues I'd use them, but many people have > reported issues which affect concurrency/stability, consistency, or > correctness of results. I believe that it's working for you, but for me, > it's not worth the risk to build using them in my environment (even though > they could be handy for many of our workflows). > > On Fri, Apr 12, 2019 at 11:18 AM Kaxil Naik wrote: > > > I have been using SubDags in production and haven't had much problem with > > it. > > > > Can you list the issues you had? > > > > Regards, > > Kaxil > > > > > > On Fri, Apr 12, 2019, 16:16 James Meickle > .invalid> > > wrote: > > > > > Given their bad reputation, would it be appropriate to deprecate > subDAGs > > > now to advertise that they're no longer considered a suitable > > > implementation? If a new and better implementation is created, would it > > > even be similar enough to subDAGs that we'd want to continue to call > the > > > feature that? > > > > > > They feel like a "new Airflow user trap" right now - I have had to tell > > my > > > team never to use them, because they seem appealing and are in the > > official > > > docs. > > > > > > On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor > > wrote: > > > > > > > I'd like to find time to fix subdags as they do provide a useful > > > > abstraction - but I agree right now they aren't great (I avoid them > > > because > > > > of this) > > > > > > > > I have half thoughts of how to it should work, I just need to look at > > the > > > > code in depth to see if that makes sense. Now 1.10.3 is out I might > > have > > > a > > > > bit more time to do this. > > > > > > > > -ash > > > > > > > > > On 12 Apr 2019, at 15:48, James Meickle > > .INVALID> > > > > wrote: > > > > > > > > > > I think we should deprecate SubDAGs given the complexity they add > and > > > the > > > > > limited usage and use cases. Or, we should invest effort in > > redesigning > > > > > their API and implementation. I think that having to account for > > > > > subdag-introduced complexity makes Airflow's code much harder to > > > maintain > > > > > and buggier, looking at how many open issues there are that > reference > > > > > subdags (and how unrelated in topic they are): > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22 > > > > > > > > > > > > > > --0000000000007d77c005865726d1--