airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil Breguła <kamil.breg...@polidea.com>
Subject Re: [VOTE] Changes in import paths
Date Tue, 30 Jul 2019 13:21:23 GMT
Yes. All changes will be backwards compatible. In the case of using
the old path, a message containing a proposal for change will be
reported to the user.

I prepared an example of how to change the name of a class in a case
with the use of a native solution.
Source code: https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename
Preview from IDE: https://imgur.com/a/45NxS5W

On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <ash@apache.org> wrote:
>
> Just cos I'm not sure it's _explicitly_ stated, but all of the moves will have a deprecation of the old name right?
>
> 3+4 case D gets my vote too.
>
> -a
>
> > On 30 Jul 2019, at 09:58, Jarek Potiuk <Jarek.Potiuk@polidea.com> wrote:
> >
> > I went ahead and updated the page (just to speed it up) as I think it
> > really makes sense to join those two cases (and I do not see any drawbacks
> > - I think the options we have cover all possible approaches) and we can
> > always go back if we need to.
> >
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >
> > My vote is *D*.
> >
> >   - airflow/contrib/operators/gcp_bigtable_operator.py  →  airflow/gcp
> >   /operators/bigtable_operator.py
> >   - airflow/contrib/operators/ssh_operator.py ->
> >   airflow/operators/ssh_operator.py
> >
> > I updated the page with my vote.
> > I propose that we vote till Friday on that one (all the rest is already
> > agreed).
> >
> > J.
> >
> >
> > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> > wrote:
> >
> >> I think almost everyone voted and we have almost perfect consensus. We all
> >> agree amongst other on moving all operators out of contrib (Great!).
> >>
> >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
> >> (Using Namespaces).
> >> I think there was actually an overlap between those two. Also Ash's
> >> comment/veto on that is quite clear.
> >> So I have final (I hope!) proposal to join those two cases and have one
> >> voting (till Friday) only on that.
> >>
> >> I will update the doc if you all agree with me that makes more sense as
> >> single case to vote on:
> >>
> >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
> >>
> >> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
> >> hooks) and keep/add prefixes in file names.
> >>
> >>   -
> >> *airflow/contrib/operators/sns_publish_operator.py  becomes
> >>   airflow/operators/**aws_sns_publish_operator.py*
> >>
> >>
> >>   - *airflow/contrib/operators/dataproc_operator.py*
> >>  becomes *airflow/operators/gcp_dataproc_operator.py*
> >>
> >>
> >>   -
> >> *airflow/contrib/hooks/gcp_bigtable_hook.py  *becomes *airflow/*
> >>   *hooks/gcp_bigtable_hook.py*
> >>   -
> >> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
> >>   *operators/ssh_operator.py*
> >>
> >>
> >>
> >> *Option B*. Group operators/sensors/hooks in airflow/operators(sensors,
> >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks), Drop the prefix.
> >>
> >>   -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >>   becomes airflow/operators/**aws/sns_publish_operator.py*
> >>
> >>
> >>   - *airflow/contrib/operators/dataproc_operator.py*
> >>  becomes *airflow/operators/gcp/dataproc_operator.py*
> >>
> >>
> >>   -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
> >>   *operators/gcp/bigtable_operator.py*
> >>   -
> >> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
> >>   *operators/ssh_operator.py*
> >>
> >>
> >> *Option C*. Group operators/sensors/hooks in airflow/operators(sensors,
> >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks), Keep the prefix.
> >>
> >>   -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >>   becomes airflow/operators/**aws/aws_sns_publish_operator.py*
> >>
> >>
> >>   - *airflow/contrib/operators/dataproc_operator.py*
> >>  becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
> >>
> >>
> >>   -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
> >>   *operators/gcp/gcp_bigtable_operator.py*
> >>   -
> >> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
> >>   *operators/ssh_operator.py*
> >>
> >>
> >> *Option D.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
> >> hooks). Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks). Drop the prefix.
> >>
> >>   -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >>   becomes airflow/aws/operators/**sns_publish_operator.py*
> >>
> >>
> >>   - *airflow/contrib/operators/dataproc_operator.py*
> >>  becomes *airflow/gcp/operators/dataproc_operator.py*
> >>
> >>
> >>   -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
> >>   *gcp/operators/bigtable_operator.py*
> >>   -
> >> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
> >>   *operators/ssh_operator.py*
> >>
> >>
> >> *Option E.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
> >> hooks). Non-cloud provider ones are moved to
> >> airflow/operators(sensors/hooks).* Keep the prefix*.
> >>
> >>   -
> >> *airflow/contrib/operators/sns_publish_operator.py
> >>   becomes airflow/aws/operators/aws_**sns_publish_operator.py*
> >>
> >>
> >>   - *airflow/contrib/operators/dataproc_operator.py*
> >>  becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
> >>
> >>
> >>   -
> >> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
> >>   *gcp/operators/gcp_bigtable_operator.py*
> >>   -
> >> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
> >>   *operators/ssh_operator.py*
> >>
> >>
> >> Shall I proceed with updating the page ?
> >>
> >>
> >> J.
> >>
> >>
> >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <kaxilnaik@gmail.com> wrote:
> >>
> >>> Thanks Jarek, adding my vote.
> >>>
> >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <ash@apache.org> wrote:
> >>>
> >>>> Thanks Jarek, much clearer. I have voted there too.
> >>>>
> >>>> -ash
> >>>>
> >>>>
> >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fokko@driesprong.frl>
> >>>> wrote:
> >>>>>
> >>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote
> >>> there
> >>>>> as well.
> >>>>>
> >>>>> You've convinced me on the second case. I've changed my vote on Case 2
> >>>> from
> >>>>> A to B :-)
> >>>>>
> >>>>> Maybe we should leave the vote open a bit longer since it involves
> >>> quite
> >>>>> some reading, and I would like to get some proper feedback and
> >>> opinions
> >>>> on
> >>>>> this. Plus I only see two votes right now. If we don't think this
> >>>> through,
> >>>>> it might be that we're having this vote again in the near future :-)
> >>>>>
> >>>>> Cheers, Fokko
> >>>>>
> >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> >>>> Jarek.Potiuk@polidea.com>:
> >>>>>
> >>>>>> I updated the AIP-21 proposal in the way that should answer all the
> >>>>>> concerns in this thread.
> >>>>>>
> >>>>>> NOTE! There is an action for all of those who are interested: Please
> >>>>>> fill-in your voting in the voting table till Tuesday 30th of July 6pm
> >>>> CEST
> >>>>>> (5pm BST, 9 am PST).
> >>>>>>
> >>>>>> I created a summary of the proposal
> >>>>>> <
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> >>>>>>>
> >>>>>> in table form - which contains also examples for each case.
> >>>>>>
> >>>>>> I added a voting table
> >>>>>> <
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> >>>>>>>
> >>>>>> where you should add your votes:
> >>>>>>
> >>>>>> I propose this voting mechanism:
> >>>>>>
> >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST  (5 pm
> >>> BST,
> >>>> 9am
> >>>>>> PST) .
> >>>>>>
> >>>>>> After the choice, the final consistent set of choices will be
> >>> announced
> >>>>>> (taking into account majority of binding vote, also including
> >>> potential
> >>>>>> vetos and consistency between the choices. Non-binding votes will be
> >>>> taken
> >>>>>> into account in case there is a draw. The final set of choices will
> >>> be
> >>>>>> announced at devlist thread
> >>>>>> <
> >>>>>>
> >>>>
> >>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> >>>>>>>
> >>>>>> after
> >>>>>> the voting completes.
> >>>>>>
> >>>>>> Unless there is a veto raised to the final proposal, the final
> >>> proposal
> >>>> is
> >>>>>> accepted by Lazy Consensus
> >>>>>> <https://community.apache.org/committers/lazyConsensus.html>  on
> >>>> *Friday*
> >>>>>> 02
> >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> >>>>>>
> >>>>>> Please let me know if what I proposed can be further improved to
> >>> avoid
> >>>>>> chaos.
> >>>>>>
> >>>>>> J.
> >>>>>>
> >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> >>> Jarek.Potiuk@polidea.com
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Ok. I see that indeed voting could have been organised a bit better
> >>> -
> >>>> dev
> >>>>>>> list does not seem to cope well with such a complex case :).
> >>>>>>>
> >>>>>>> As Kamil noticed - we already have a formal AIP-21
> >>>>>>> <
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>>>>>
> >>>>>>> in the Wiki - which I should have mentioned before. I have not much
> >>>> time
> >>>>>>> today - but tomorrow I will try to summarise the options - based on
> >>>> some
> >>>>>>> real code examples (to make it easier to digest). I think the best
> >>>>>> approach
> >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where people
> >>>> will
> >>>>>>> be able to state their preferences for each case (by adding a row to
> >>>> the
> >>>>>>> table). I think that might provide much better clarity and a single
> >>>> place
> >>>>>>> which will show the current aggregated state of voting.
> >>>>>>>
> >>>>>>> However - as Ash also stated - some of the options are
> >>> inter-dependent
> >>>> so
> >>>>>>> at the end we will have to adjust the results in case they are not
> >>>>>>> consistent  - and make final voting on aggregated set of cases -
> >>> but I
> >>>>>>> think in this case following "lasy consensus" - i,e. if noone
> >>> objects
> >>>> to
> >>>>>>> final set of cases, we will proceed with it..
> >>>>>>>
> >>>>>>> Do you think that will work better ?
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>> Principal Software Engineer
> >>>>>>> Phone: +48660796129
> >>>>>>>
> >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> >>>>>>> kamil.bregula@polidea.com> napisał:
> >>>>>>>
> >>>>>>>> Document is also available as AIP on wiki:
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> >>>>>>>>
> >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> >>> <fokko@driesprong.frl
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of clarity,
> >>> I
> >>>>>>>> would
> >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
> >>>>>> structural
> >>>>>>>>> code improvement AIP and then and add a matrix of users/cases
> >>> where
> >>>>>>>>> everybody can add his vote per case. This gives also the
> >>> opportunity
> >>>>>> for
> >>>>>>>>> changing your vote on a specific case. WDYT?
> >>>>>>>>>
> >>>>>>>>> Cheers, Fokko
> >>>>>>>>>
> >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <cixuuz@gmail.com>:
> >>>>>>>>>
> >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than
> >>> all
> >>>>>>>>>> together.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
> >>> ash@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> A number of proposals overlap or add on to each other, so I
> >>> think
> >>>>>>>> (?) a
> >>>>>>>>>>> single vote makes sense.
> >>>>>>>>>>>
> >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
> >>>>>>>> voting
> >>>>>>>>> on.
> >>>>>>>>>>>
> >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <kaxilnaik@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
> >>>>>> how
> >>>>>>>>>>>> about
> >>>>>>>>>>>> separate vote emails for all the cases? Each email can have the
> >>>>>>>>>>>> description
> >>>>>>>>>>>> copied over from the doc.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It would probably be a bit easier to track a decision in the
> >>>>>> future
> >>>>>>>> as
> >>>>>>>>>>>> well.
> >>>>>>>>>>>>
> >>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
> >>> Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>> @Driesprong,
> >>>>>>>>>>>> Fokko <fokko@driesprong.frl>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
> >>> kaxilnaik@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #4* Separate namespace for resources :
> >>>>>>>>>>>>> *Solution A*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *Case #7* : *Solution A*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> >>>>>>>>>>>> <dpstandish@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary,
> >>> but
> >>>>>> I
> >>>>>>>> am
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
> >>>>>> can
> >>>>>>>>>>>> further
> >>>>>>>>>>>>>> clarify.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
> >>>>>>>> either
> >>>>>>>>>>>>>> proposal
> >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
> >>>>>>>>>>>> focused on
> >>>>>>>>>>>>>> cases 1,2,3 and 7.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> *Motion is to adopt the following:*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> >>>>>>>>>>>>>> All objects will be moved out of contrib folder and into
> >>>>>>>> respective
> >>>>>>>>>>>>>> non-contrib folder.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
> >>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> >>>>>>>>>>>>>> airflow.operators.foo.FooOperator
> >>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> >>>>>>>> airflow.hooks.foo.FooHook
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
> >>>>>>>>>>>>>> Example case is
> >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
> >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is
> >>> now
> >>>>>>>> named
> >>>>>>>>>>>> like
> >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> >>>>>>>>>>>>>> So what is proposed for this case?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 4 - Solution B - *no change*
> >>>>>>>>>>>>>> This proposal was about namespaces but the motion is to
> >>> reject
> >>>>>>>> the
> >>>>>>>>>>>>>> proposal.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 5 - Solution B - *no change*
> >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
> >>>>>>>>>>>> Operator in
> >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal --
> >>> i.e.
> >>>>>>>>>>>> motion is
> >>>>>>>>>>>>>> no
> >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
> >>>>>>>>>>>> "Sensor")
> >>>>>>>>>>>>>> suffixes on class names
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 6 - *No change*
> >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but motion
> >>>>>> does
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>> enact
> >>>>>>>>>>>>>> any specific proposal.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> >>>>>> deprecation
> >>>>>>>>>>>> warnings
> >>>>>>>>>>>>>> (instead of zope)
> >>>>>>>>>>>>>> Example:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> # in old_package/old_mod.py
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> import warnings
> >>>>>>>>>>>>>> from new_package.new_mod import *
> >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> >>>>>>>> new_package.new_mod.
> >>>>>>>>>>>>>> Import
> >>>>>>>>>>>>>> of "
> >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
> >>>>>>>>>>>>>> DeprecationWarning, 2)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Reference implementation here:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib.
> >>>>>> Thank
> >>>>>>>> you
> >>>>>>>>>>>>>> Sergio
> >>>>>>>>>>>>>> for turning the tide with your effective argumentation ;)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> >>>>>>>> ash@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Could someone summarise what the proposed name changes are,
> >>>>>>>> here,
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial
> >>>>>>>> examples
> >>>>>>>>>>>> is 1)
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
> >>>>>> 2)
> >>>>>>>>>>>> using a
> >>>>>>>>>>>>>>> google doc isn't great for a longer term record.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Ash
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> >>>>>>>>>>>> <Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
> >>>>>>>> would
> >>>>>>>>>>>> like
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
> >>>>>>>>>>>> (binding)
> >>>>>>>>>>>>>> votes
> >>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the
> >>>>>>>>>>>> previous
> >>>>>>>>>>>>>>> proposal.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Here is the proposal (details here
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
> >>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> >>>>>>>> could
> >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> >>>>>>>> class
> >>>>>>>>>>>>>> names*
> >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> >>>>>>>> (and
> >>>>>>>>>>>> my team
> >>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
> >>>>>>>>>>>> integration)*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is my binding (+1) vote.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> >>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey Felix,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
> >>>>>> (I
> >>>>>>>>>>>> use IDE
> >>>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>>> often ;) )
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> >>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
> >>>>>>>> able
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> respond
> >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
> >>>>>>>> Europe
> >>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
> >>>>>> into
> >>>>>>>> a
> >>>>>>>>>>>> single
> >>>>>>>>>>>>>>>>>> package.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
> >>>>>>>> to
> >>>>>>>>>>>> get
> >>>>>>>>>>>>>> rid of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
> >>>>>>>>>>>> tricky
> >>>>>>>>>>>>>>> question,
> >>>>>>>>>>>>>>>>>> I think this is already reflected in the document since
> >>>>>>>>>>>> "maturity"
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
> >>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> core
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> >>>>>> incubating
> >>>>>>>>>>>> will
> >>>>>>>>>>>>>> confuse
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with
> >>>>>>>> Airflow it
> >>>>>>>>>>>> isn't
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if you
> >>>>>>>>>>>> don't use
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>> IDE
> >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
> >>>>>>>> changing
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> public
> >>>>>>>>>>>>>>>>>> API just for a different name.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
> >>>>>>>> will
> >>>>>>>>>>>> make
> >>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>> as we will not have to define other rules as those that we
> >>>>>>>>>>>> already
> >>>>>>>>>>>>>> have.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
> >>>>>> +0)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for
> >>>>>>>>>>>> BashOperator.
> >>>>>>>>>>>>>>> After
> >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
> >>>>>> mail
> >>>>>>>> you
> >>>>>>>>>>>>>> state: -
> >>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
> >>>>>>>> could
> >>>>>>>>>>>> become
> >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
> >>>>>>>>>>>> keeping it
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
> >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
> >>>>>>>>>>>> confusion
> >>>>>>>>>>>>>>> here.
> >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
> >>>>>> for
> >>>>>>>>>>>> changing
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> *module* name not the *class* name. So
> >>>>>>>>>>>> bash_operator.BashOperator ->
> >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> >>>>>>>> BashOperator
> >>>>>>>>>>>> in this
> >>>>>>>>>>>>>>> case.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
> >>>>>>>> its
> >>>>>>>>>>>> own
> >>>>>>>>>>>>>>> package
> >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
> >>>>>>>> like
> >>>>>>>>>>>>>>> Databricks
> >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own package
> >>>>>>>>>>>> because
> >>>>>>>>>>>>>> their
> >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
> >>>>>>>> bit
> >>>>>>>>>>>>>> painful
> >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
> >>>>>>>> everything
> >>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>> introduces noise into version control etc.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
> >>>>>>>>>>>> difficult as
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> seems to group operators in packages and it has numerous
> >>>>>>>>>>>> advantages.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the
> >>>>>>>>>>>> __init__.py of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will
> >>>>>>>> actually
> >>>>>>>>>>>>>> realise
> >>>>>>>>>>>>>>>>> that the file has been moved and you can follow that.
> >>>>>> Maybe
> >>>>>>>> not
> >>>>>>>>>>>> as
> >>>>>>>>>>>>>> super
> >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4  but it's not a
> >>>>>> rocket
> >>>>>>>>>>>> science
> >>>>>>>>>>>>>>> either.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yes.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Apart from all, great work!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> >>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
> >>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
> >>>>>> these
> >>>>>>>>>>>> cases.
> >>>>>>>>>>>>>> So I
> >>>>>>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>> with +1 (non-binding).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
> >>>>>>>> intense
> >>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>> towards
> >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> >>>>>> creating
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> document. I
> >>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent
> >>>>>> and
> >>>>>>>>>>>> clean :)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>> Felix
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -------- Original Message --------
> >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
> >>>>>>>> paths.
> >>>>>>>>>>>> It's
> >>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>> discussed in
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
> >>>>>>>> CEST),
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>> least
> >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from
> >>>>>> Kamil
> >>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The proposed solution is:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
> >>>>>> "well"
> >>>>>>>>>>>> tested
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> >>>>>>>>>>>>>>>>>>>> - *Case 2: B:
> >>>>>> Airflow.operators.foo_operator.FooOperator
> >>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> >>>>>>>>>>>>>>>>>>>> - *Case 3: C:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> >>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
> >>>>>> on
> >>>>>>>>>>>> class
> >>>>>>>>>>>>>> names*
> >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> >>>>>>>> (and
> >>>>>>>>>>>> my
> >>>>>>>>>>>>>> team
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> >>>>>> Software
> >>>>>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >>>>>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jarek Potiuk
> >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> >>>>>>>>> Engineer
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> *Kaxil Naik*
> >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> >>>>>> Spark
> >>>>>>>> &
> >>>>>>>>>>>> Neo4j
> >>>>>>>>>>>>> Developer
> >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> *Kaxil Naik*
> >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> >>> Spark
> >>>>>> &
> >>>>>>>>>>>> Neo4j
> >>>>>>>>>>>> Developer
> >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Jarek Potiuk
> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>>
> >>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> *Kaxil Naik*
> >>> *Big Data Consultant | DevOps Data Engineer*
> >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> >>> Developer
> >>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>

Mime
View raw message