From dev-return-9008-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Sat Jul 27 07:07:52 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EA05118064E for ; Sat, 27 Jul 2019 09:07:51 +0200 (CEST) Received: (qmail 21723 invoked by uid 500); 27 Jul 2019 07:07:49 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 21711 invoked by uid 99); 27 Jul 2019 07:07:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Jul 2019 07:07:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 13D42180C7B for ; Sat, 27 Jul 2019 07:07:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.253 X-Spam-Level: ** X-Spam-Status: No, score=2.253 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=driesprong-frl.20150623.gappssmtp.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GqqmxlC6WiLA for ; Sat, 27 Jul 2019 07:07:43 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=209.85.219.196; helo=mail-yb1-f196.google.com; envelope-from=fokko@driesprongen.nl; receiver= Received: from mail-yb1-f196.google.com (mail-yb1-f196.google.com [209.85.219.196]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id EF038BC7E1 for ; Sat, 27 Jul 2019 07:07:42 +0000 (UTC) Received: by mail-yb1-f196.google.com with SMTP id j6so11847313ybm.7 for ; Sat, 27 Jul 2019 00:07:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=driesprong-frl.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=05jwsAJiS5K328WqmPwDXlPi4JBJ6EgO1EyxYhQxeI0=; b=jmNyxmaeFSdX0vsQ2kRCS8POlqDKcYoF6wIVn8i6jjkWyrNwyrEYMJI88JrrdUuSYD KHMmi+VFMzSX96gCtNNf2+zW0Trww5E6s0tXNAx1MhKOWGefy6H+Ue4+9/qXtTi2bTY+ yFugkLo3xqqITXk8+YQVwtOuuEeTkvkogkblj8/PHmnmgHSrUVQytsv9OLV+VPbyH/CR lHLHSjEweDD63YDzQAMOB9W86SBFBVhft5+l9gIE0FMX7hX8YgPbup5FQhBaPFj+4Ru9 TBqAZTPAob8AgZqL1j++AvpKL7r0JaDgUa/meiteC0Nkmq07MsyYDbD3w9ubmhET1FUN z4TA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=05jwsAJiS5K328WqmPwDXlPi4JBJ6EgO1EyxYhQxeI0=; b=M24s0E1XftWfK0r76YDxrFANTawgI/+/LOu6ck54uozqUJ6UUaUTcQPdBXhqP8KSdO KZkLT4zt2Z8BHUNN5SXadw1rXmroZtJUFu1Yw3psQ0qI/c9zmWn5m8JTbD3fogH1LyUk HIenhDV0A5wS2gTNiGdy/JOxAT/UkBo9JX9ZI1WZADIZxdP6iZCOXw+7KbcXuG0x4fxw 6ahoE5NlkXW9LZRszutPbludvoUg/nawGynHDLIYK5+CbPQidB08dCT3Dpmn3h56Ao6N iIPXOAPaUBgEBnGY3JS6YaAwWoU6LN5kc/Q+7MIKstYLmQIgvvK6fkpWL0ZBAgl2n++K T67Q== X-Gm-Message-State: APjAAAUwlbDJHbqRgZgSni6A2olia09ccTAq5PUiK0ype4SEocVDnDOj hBhXFWnIPr3STaZ4gk5tPl6AlDBm5CmtIWHUxSM= X-Google-Smtp-Source: APXvYqwVVzh0TilgI2CwfOJcjamyc/VZN7MTDGCwWGLp3TBmL50eR/SahsGhCf1SZSGIAZdDKn4G7FYgSafmqHPWLHY= X-Received: by 2002:a5b:7d1:: with SMTP id t17mr40227172ybq.281.1564211261779; Sat, 27 Jul 2019 00:07:41 -0700 (PDT) MIME-Version: 1.0 References: <20E5C0CB-5EA6-49CE-80E8-D944116E409A@apache.org> <6C792643-C3EE-430B-8769-83B9EACE9890@apache.org> In-Reply-To: From: "Driesprong, Fokko" Date: Sat, 27 Jul 2019 09:07:30 +0200 Message-ID: Subject: Re: [VOTE] Changes in import paths To: Chen Tong Cc: dev@airflow.apache.org, Kaxil Naik , Jarek Potiuk Content-Type: multipart/alternative; boundary="0000000000009cc803058ea45065" --0000000000009cc803058ea45065 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I agree with all, this is a bit chaotic. For the sake of clarity, I would suggest moving the Google Doc to the Wiki as an AIP. Create a structural code improvement AIP and then and add a matrix of users/cases where everybody can add his vote per case. This gives also the opportunity for changing your vote on a specific case. WDYT? Cheers, Fokko Op vr 26 jul. 2019 om 22:28 schreef Chen Tong : > I agree with Ash. It is clear to vote on each case rather than all > together. > > On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor wrote: > >> A number of proposals overlap or add on to each other, so I think (?) a >> single vote makes sense. >> >> To be clear, my vote is -1 until it's clear exactly what we are voting o= n. >> >> On 26 July 2019 20:39:56 BST, Kaxil Naik wrote: >> >I agree with Ash and instead of voting on all 7 Cases together, how >> >about >> >separate vote emails for all the cases? Each email can have the >> >description >> >copied over from the doc. >> > >> >It would probably be a bit easier to track a decision in the future as >> >well. >> > >> >What do you guys think? @Jarek Potiuk >> >@Driesprong, >> >Fokko >> > >> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik wrote: >> > >> >> *Case #1* airflow.contrib.{resources} : *Solution A * >> >> >> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A * >> >> >> >> *Case #3* {aws/azure/gcp}_* : *Solution C* >> >> >> >> *Case #4* Separate namespace for resources : >> >> *Solution A* >> >> >> >> *Case #5* *Operator : *Solution A* >> >> >> >> *Case #7* : *Solution A* >> >> >> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish >> > >> >> wrote: >> >> >> >>> I have tried to add some clarification to Jarek's summary, but I am >> >a >> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek can >> >further >> >>> clarify. >> >>> >> >>> According to my reading, in this motion cases 4,5,6 are all either >> >>> proposal >> >>> *rejections* or otherwise have no effect, so attention can be >> >focused on >> >>> cases 1,2,3 and 7. >> >>> >> >>> *Motion is to adopt the following:* >> >>> >> >>> Case 1 - Solution A - Get rid of contrib >> >>> All objects will be moved out of contrib folder and into respective >> >>> non-contrib folder. >> >>> >> >>> Case 2 - Solution A - Remove object type suffix from modules >> >>> Example: >> >>> Airflow.operators.foo_operator.FooOperator becomes >> >>> airflow.operators.foo.FooOperator >> >>> Example: >> >>> Airflow.hooks.foo_hook.FooOperator becomes airflow.hooks.foo.FooHook >> >>> >> >>> Case 3 - conventions for cloud provider etc prefixes >> >>> *I am not sure exactly what the proposal is.* >> >>> Example case is airflow/contrib/operators/gcp_bigtable_operator.py >> >>> Since motion adopts case 1 solution A, we can assume it is now named >> >like >> >>> so: airflow/operators/gcp_bigtable_operator.py >> >>> So what is proposed for this case? >> >>> >> >>> Case 4 - Solution B - *no change* >> >>> This proposal was about namespaces but the motion is to reject the >> >>> proposal. >> >>> >> >>> Case 5 - Solution B - *no change* >> >>> The proposal was to remove class type suffix e.g. remove the >> >Operator in >> >>> FooOperator, but the motion is to reject this proposal -- i.e. >> >motion is >> >>> no >> >>> change on this case, i.e. we resolve to keep "Operator" (and >> >"Sensor") >> >>> suffixes on class names >> >>> >> >>> Case 6 - *No change* >> >>> This case concerns object naming inconsistencies, but motion does >> >not >> >>> enact >> >>> any specific proposal. >> >>> >> >>> Case 7 - Solution A - use warnings.warn template for deprecation >> >warnings >> >>> (instead of zope) >> >>> Example: >> >>> >> >>> # in old_package/old_mod.py >> >>> >> >>> import warnings >> >>> from new_package.new_mod import * >> >>> warnings.warn("old_package.old_mod has moved to new_package.new_mod. >> >>> Import >> >>> of " >> >>> "old_package.old_mod will become unsupported in version 2", >> >>> DeprecationWarning, 2) >> >>> >> >>> Reference implementation here: >> >>> >> >>> >> > >> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/soluti= on1 >> >>> >> >>> >> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3. >> >>> >> >>> I am very happy that this motion now gets rid of contrib. Thank you >> >>> Sergio >> >>> for turning the tide with your effective argumentation ;) >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor >> >wrote: >> >>> >> >>> > Could someone summarise what the proposed name changes are, here, >> >in >> >>> this >> >>> > thread? Pointing at a google doc and only giving partial examples >> >is 1) >> >>> a >> >>> > bit difficult to quickly work out what we are voting on, and 2) >> >using a >> >>> > google doc isn't great for a longer term record. >> >>> > >> >>> > Thanks, >> >>> > Ash >> >>> > >> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk >> > >> >>> wrote: >> >>> > > >> >>> > > I would like to recast the vote. Let's start from 0 :) . I would >> >like >> >>> to >> >>> > > keep the July 30th 6pm CEST date, and at least three +1 >> >(binding) >> >>> votes >> >>> > are >> >>> > > needed to pass. I marked in red the modifications to the >> >previous >> >>> > proposal. >> >>> > > >> >>> > > Here is the proposal (details here >> >>> > > < >> >>> > >> >>> >> > >> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9j= X57Ef6A/edit#heading=3Dh.j4jc3i70qszo >> >>> > > >> >>> > > ) >> >>> > > >> >>> > > - *Case 1: A: Get rid of contrib,* >> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator could >> >>> > > become airflow.operators.foo.FooOperator* >> >>> > > - *Case 3: C: >> >>> > > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >> >>> could >> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator* >> >>> > > - *Case 4: B: no namespace introduction* >> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on class >> >>> names* >> >>> > > - *Case 6: We will treat isolated cases on case-by-case (and >> >my team >> >>> > can >> >>> > > do the job of GCP-related operators)* >> >>> > > - *Case 7: Native python solution (with better IDE >> >integration)* >> >>> > > >> >>> > > This is my binding (+1) vote. >> >>> > > >> >>> > > >> >>> > > >> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk < >> >>> Jarek.Potiuk@polidea.com> >> >>> > > wrote: >> >>> > > >> >>> > >> Hey Felix, >> >>> > >> >> >>> > >> For 7* -> I am in favour of trying the native one as well (I >> >use IDE >> >>> > quite >> >>> > >> often ;) ) >> >>> > >> >> >>> > >> J. >> >>> > >> >> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko >> >>> > >>> > > >> >>> > >> wrote: >> >>> > >> >> >>> > >>> Thanks Kamil for putting the document together. I wasn't able >> >to >> >>> > respond >> >>> > >>> earlier since I was giving Airflow workshops throughout Europe >> >:-) >> >>> > >>> >> >>> > >>> *Case 1: *Solution A =E2=86=92 Remove everything from contrib = into a >> >single >> >>> > >>> package. >> >>> > >>> >> >>> > >>> To make it easier to the end-user, my preference would be to >> >get >> >>> rid of >> >>> > >>> the >> >>> > >>> contrib package altogether. What's in contrib or not is a >> >tricky >> >>> > question, >> >>> > >>> I think this is already reflected in the document since >> >"maturity" >> >>> is >> >>> > in >> >>> > >>> quotes. What would be the moment to transfer the operator to >> >the >> >>> core >> >>> > or >> >>> > >>> vice versa? I'm afraid that renaming contrib to incubating >> >will >> >>> confuse >> >>> > >>> the >> >>> > >>> user even more. For people who aren't as known with Airflow it >> >isn't >> >>> > that >> >>> > >>> transparent which operator lives where, especially if you >> >don't use >> >>> an >> >>> > IDE >> >>> > >>> such as Ash ;) Besides that I don't think it is worth changing >> >the >> >>> > public >> >>> > >>> API just for a different name. >> >>> > >>> >> >>> > >> >> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It will >> >make >> >>> > easier >> >>> > >> as we will not have to define other rules as those that we >> >already >> >>> have. >> >>> > >> >> >>> > >> >> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0) >> >>> > >>> >> >>> > >>> I like to search on Github with t, and then search for >> >BashOperator. >> >>> > After >> >>> > >>> the change, you should search for Operator/Bash. >> >>> > >>> >> >>> > >>> Jarek, I'm confused with your vote on this, from your mail you >> >>> state: - >> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator could >> >become >> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is >> >keeping it >> >>> > the >> >>> > >>> same, so that would mean - *Case 2: B: >> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay >> >>> > >>> airflow.operators.foo_operator*.FooOperator* >> >>> > >>> >> >>> > >>> My mistake. It was of course A. I think there is a little >> >confusion >> >>> > here. >> >>> > >> This case is not for changing BashOperator into Bash but for >> >changing >> >>> > the >> >>> > >> *module* name not the *class* name. So >> >bash_operator.BashOperator -> >> >>> > >> bash.BashOperator :) . you will still search for BashOperator >> >in this >> >>> > case. >> >>> > >> >> >>> > >> >> >>> > >>> *Case 3:* I'm leaning towards B >> >>> > >>> >> >>> > >>> Mostly because it isn't as trivial to decide when it gets its >> >own >> >>> > package >> >>> > >>> or not. Besides the cloud providers, there are companies like >> >>> > Databricks >> >>> > >>> and Qubole which might also end up with their own package >> >because >> >>> their >> >>> > >>> integration is getting richer over time. Moving this is a bit >> >>> painful >> >>> > >>> because we need to deprecate the old path and move everything >> >which >> >>> > >>> introduces noise into version control etc. >> >>> > >> >> >>> > >> >> >>> > >> I'd say, we should go for C. As I proposed. It's not as >> >difficult as >> >>> it >> >>> > >> seems to group operators in packages and it has numerous >> >advantages. >> >>> The >> >>> > >> nice thing about deprecations - we can do them in the >> >__init__.py of >> >>> the >> >>> > >> packages which causes the noise fairly small (Git will actually >> >>> realise >> >>> > >> that the file has been moved and you can follow that. Maybe not >> >as >> >>> super >> >>> > >> easy to merge changes later to 1.10.4 but it's not a rocket >> >science >> >>> > either. >> >>> > >> >> >>> > >> >> >>> > >>> *Case 7:* Python native solution >> >>> > >>> >> >>> > >> >> >>> > >> Yes. >> >>> > >> >> >>> > >> >> >>> > >>> >> >>> > >>> --- >> >>> > >>> >> >>> > >>> Apart from all, great work! >> >>> > >>> >> >>> > >>> Cheers, Fokko >> >>> > >>> >> >>> > >>> >> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall >> >>> > >>> > >>> > >>>> : >> >>> > >>> >> >>> > >>>> I have no strong "No" against any proposed change of these >> >cases. >> >>> So I >> >>> > >>> go >> >>> > >>>> with +1 (non-binding). >> >>> > >>>> >> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your intense >> >work >> >>> > >>> towards >> >>> > >>>> airflow currently :) and thanks to Kamil for even creating >> >this >> >>> > >>> document. I >> >>> > >>>> like how the code is getting more and more consistent and >> >clean :) >> >>> > >>>> >> >>> > >>>> Kind regards, >> >>> > >>>> Felix >> >>> > >>>> >> >>> > >>>> Sent from ProtonMail mobile >> >>> > >>>> >> >>> > >>>> -------- Original Message -------- >> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote: >> >>> > >>>> >> >>> > >>>>> Hello everyone, >> >>> > >>>>> >> >>> > >>>>> This email is calling a vote on the changes in import paths. >> >It's >> >>> > been >> >>> > >>>>> discussed in >> >>> > >>>>> >> >>> > >>>> >> >>> > >>> >> >>> > >> >>> >> > >> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce46= 8f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E >> >>> > >>>>> >> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm CEST), >> >and >> >>> at >> >>> > >>> least >> >>> > >>>>> three +1 (binding) votes have been cast. >> >>> > >>>>> >> >>> > >>>>> The proposal to vote is based on the document from Kamil >> >>> > >>>>> < >> >>> > >>>> >> >>> > >>> >> >>> > >> >>> >> > >> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9j= X57Ef6A/edit# >> >>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> The proposed solution is: >> >>> > >>>>> >> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well" >> >tested >> >>> and >> >>> > >>>>> rename contrib to "incubating" or similar.* >> >>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator >> >could >> >>> > >>>>> become airflow.operators.foo.FooOperator* >> >>> > >>>>> - *Case 3: C: >> >>> > >>>>> >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >> >>> > could >> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator* >> >>> > >>>>> - *Case 4: B: no namespace introduction* >> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on >> >class >> >>> names* >> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case (and >> >my >> >>> team >> >>> > >>> can >> >>> > >>>>> do the job of GCP-related operators)* >> >>> > >>>>> >> >>> > >>>>> This is my (binding) +1 vote. >> >>> > >>>>> >> >>> > >>>>> Best regards, >> >>> > >>>>> >> >>> > >>>>> J. >> >>> > >>>>> >> >>> > >>>>> -- >> >>> > >>>>> >> >>> > >>>>> Jarek Potiuk >> >>> > >>>>> Polidea | Principal Software >> >Engineer >> >>> > >>>>> >> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129) >> >>> > >>> <[+48660796129](tel:+48660796129)> >> >>> > >>>>> [image: Polidea] >> >>> > >>> >> >>> > >> >> >>> > >> >> >>> > >> -- >> >>> > >> >> >>> > >> Jarek Potiuk >> >>> > >> Polidea | Principal Software >> >Engineer >> >>> > >> >> >>> > >> M: +48 660 796 129 <+48660796129> >> >>> > >> [image: Polidea] >> >>> > >> >> >>> > >> >> >>> > > >> >>> > > -- >> >>> > > >> >>> > > Jarek Potiuk >> >>> > > Polidea | Principal Software Engineer >> >>> > > >> >>> > > M: +48 660 796 129 <+48660796129> >> >>> > > [image: Polidea] >> >>> > >> >>> > >> >>> >> >> >> >> >> >> -- >> >> *Kaxil Naik* >> >> *Big Data Consultant | DevOps Data Engineer* >> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & >> >Neo4j >> >> Developer >> >> *LinkedIn*: https://www.linkedin.com/in/kaxil >> >> >> > >> > >> >-- >> >*Kaxil Naik* >> >*Big Data Consultant | DevOps Data Engineer* >> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & >> >Neo4j >> >Developer >> >*LinkedIn*: https://www.linkedin.com/in/kaxil >> > --0000000000009cc803058ea45065--