From dev-return-11732-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Thu Jun 18 16:33:50 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3362A18065B for ; Thu, 18 Jun 2020 18:33:50 +0200 (CEST) Received: (qmail 19188 invoked by uid 500); 18 Jun 2020 16:33:49 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 19176 invoked by uid 99); 18 Jun 2020 16:33:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2020 16:33:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1E69EC0A55 for ; Thu, 18 Jun 2020 16:33:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=twitter.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id kxWinlvfdpzi for ; Thu, 18 Jun 2020 16:33:46 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::131; helo=mail-il1-x131.google.com; envelope-from=ddavydov@twitter.com; receiver= Received: from mail-il1-x131.google.com (mail-il1-x131.google.com [IPv6:2607:f8b0:4864:20::131]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 030A77DD07 for ; Thu, 18 Jun 2020 16:33:45 +0000 (UTC) Received: by mail-il1-x131.google.com with SMTP id e11so6387805ilr.4 for ; Thu, 18 Jun 2020 09:33:45 -0700 (PDT) X-ASF-DKIM-Sig: v=1; a=rsa-sha256; c=relaxed/relaxed; d=twitter.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ug1CNn9ADydD5JfpS2AkVW49yQMJw5F5Xvk3Pyc57+Q=; b=ckJOx9lYNMVTL0/ZtaXBYqRpNpd4dnBx6yQrIttXBJ6MUb2750Swxz80HIvnGyJXZn oENCSfWWoJmcZhJ1AVtg8Qqz6zZT12SJRFF0iw2S4a+Qw1IXnWlIbqSG7vLyGMcIfIA8 Whbuf3waBp+WkKg4XsKuwxpHUNjlz7/Xkwo0Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ug1CNn9ADydD5JfpS2AkVW49yQMJw5F5Xvk3Pyc57+Q=; b=sywcDnYzRFnVwYKez1eD4at7r7c3l+wbBdnqSdnwxbb3b6t8r5bSFcqIdCMBTI16cr W+mVb8FwBUy2URSE0F5L5xtBPSCxZh1IL9udVi9miBJTUqH5gkPmquHJHFfVOKFVXFsl etkALdCGNhazM0Efofc3YA/dSzJcmFHCcvQuK/HHI3SyPM+ob1dzYylQmYve4+WeNRoB aiWptRKXgoRonuFayE64rU9yKpj3BWf5VHjDgGFgUZXsnbP+r90dYAdEiT2N+piwjNqH kFommsZGzT1hy9mFfPfpEEzTz9i74WQ3u0yx//qIRxxficC2snJ2lyMA+OYg/VKrvb2/ MVPw== X-Gm-Message-State: AOAM531yfvRl2iuy1UR3Us3rwC1Wc6WghITxEEfhadyDmHBuO/cPBQEJ vOu+QlO67BVy5Pq0FMvupszWAMpM0B0U/jKVBv4/DXTZ X-Google-Smtp-Source: ABdhPJxBcLBS/O7elqztpA8px/69GwFq7U8ZTRYOmSoWdZrm4jcOCrSirckSf2iejA15rDxKk20YoqRRM3opN1X9j5U= X-Received: by 2002:a92:290b:: with SMTP id l11mr5050831ilg.145.1592498018756; Thu, 18 Jun 2020 09:33:38 -0700 (PDT) MIME-Version: 1.0 References: <60bbb611-aaf9-4e75-9a1f-b5fcf5cb434b@Spark> <779d8e8a-f9f7-416e-b34d-f3f0c05fe15f@Spark> <44419d36-7dc5-44af-beaa-31b2fbc1ed06@gmail.com> In-Reply-To: <44419d36-7dc5-44af-beaa-31b2fbc1ed06@gmail.com> From: Dan Davydov Date: Thu, 18 Jun 2020 12:33:03 -0400 Message-ID: Subject: Re: [UPDATE] AIP-31 .output update To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="000000000000b6f60505a85e5620" --000000000000b6f60505a85e5620 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable +1 (binding) On Thu, Jun 18, 2020 at 12:32 PM Daniel Imberman wrote: > +1 (binding) > > via Newton Mail [ > https://cloudmagic.com/k/d/mailapp?ct=3Ddx&cv=3D10.0.50&pv=3D10.14.6&sour= ce=3Demail_footer_2 > ] > On Wed, Jun 17, 2020 at 12:13 AM, Tomasz Urbaszek > wrote: > +1 (binding) > > On Wed, Jun 17, 2020 at 3:39 AM =E8=92=8B=E6=99=93=E5=B3=B0 wrote: > > > +1(not binding) > > > > On Wed, Jun 17, 2020 at 3:03 AM Gerard Casas Saez > > wrote: > > > > > Hi everyone, > > > > > > Sending an email here to consolidate an update to the AIP-31 that has > > > happened while we have been implementing this. > > > > > > When AIP-31 was proposed, the proposal mentioned that all operators > would > > > have a __call__ method which would be used to define DAG in a > functional > > > manner. While implementing this with the SIG Functional Ops team, we > > > decided to change that requirement and replace it with a better > > integration > > > with templated_fields and a new property in BaseOperator called outpu= t. > > The > > > idea is that (see AIP DAG example for variable reference) get_ip() = =3D=3D > > > get_ip.output. > > > > > > I updated the AIP-31 doc to reflect those changes. Also took the chan= ce > > to > > > clean the doc a bit to reword some smaller stuff, in order to make it > > > easier for consumption by people outside of this group in the future. > > > Update diff: > > > > > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageI= d=3D148638736&selectedPageVersions=3D12&selectedPageVersions=3D11 > > > Updated AIP-31: > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=3D148638= 736 > > > > > > Regarding the output property change, you can follow this discussion = in > > > the current Airflow PR between me and Ash, to understand better the > > > motivations behind it: > > > https://github.com/apache/airflow/pull/8962/files#r441004317. > > > > > > A short summary: > > > The main reason for this change was that sometimes templated args > > > in operators are required on initialization > > > (example: subject on EmailOperator). This meant that you needed to ad= d > a > > > placeholder value on initialization, and then specify the XComArg tha= t > > > needed to be used instead in the __call__ method. This seemed a bit > > > confusing, so when implementing we decided to rollback > > > the __call__ addition to BaseOperator. > > > > > > This made it difficult to get XComArg from an operator as the > > > initializer returns an instance of the operator instead of > > > an XComArg compared to the proposed __call__ method. For that, we > > proposed > > > to add a new property output to BaseOperator that would generate an > > XComArg > > > for the operator with return_value as the default key. > > > > > > An example of how different the DAG looks between the previous AIP an= d > > the > > > new update can be found here: > > > https://github.com/apache/airflow/pull/8962/files#r441013469 > > > > > > With that said, I will follow Ash recommendation and try to get a laz= y > > > consensus on this change: > > > Allow 48h for committers and developers to comment/bring up > > > objections on the change. > > > > > > If no objection is raised by Thursday June 18th at 1PM MST, we will > move > > > forward with the change and merge > > > https://github.com/apache/airflow/pull/8962 PR. > > > > > > Thanks everyone for all the work on this change! =F0=9F=8E=89 > > > > > > Gerard Casas Saez > > > Twitter | Cortex | @casassaez > > > > > --000000000000b6f60505a85e5620--