airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [VOTE] Changes in import paths
Date Tue, 30 Jul 2019 12:28:26 GMT
Just cos I'm not sure it's _explicitly_ stated, but all of the moves will have a deprecation of the old name right?

3+4 case D gets my vote too.

-a

> On 30 Jul 2019, at 09:58, Jarek Potiuk <Jarek.Potiuk@polidea.com> wrote:
> 
> I went ahead and updated the page (just to speed it up) as I think it
> really makes sense to join those two cases (and I do not see any drawbacks
> - I think the options we have cover all possible approaches) and we can
> always go back if we need to.
> 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> 
> My vote is *D*.
> 
>   - airflow/contrib/operators/gcp_bigtable_operator.py  →  airflow/gcp
>   /operators/bigtable_operator.py
>   - airflow/contrib/operators/ssh_operator.py ->
>   airflow/operators/ssh_operator.py
> 
> I updated the page with my vote.
> I propose that we vote till Friday on that one (all the rest is already
> agreed).
> 
> J.
> 
> 
> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> wrote:
> 
>> I think almost everyone voted and we have almost perfect consensus. We all
>> agree amongst other on moving all operators out of contrib (Great!).
>> 
>> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
>> (Using Namespaces).
>> I think there was actually an overlap between those two. Also Ash's
>> comment/veto on that is quite clear.
>> So I have final (I hope!) proposal to join those two cases and have one
>> voting (till Friday) only on that.
>> 
>> I will update the doc if you all agree with me that makes more sense as
>> single case to vote on:
>> 
>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.*
>> 
>> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
>> hooks) and keep/add prefixes in file names.
>> 
>>   -
>> *airflow/contrib/operators/sns_publish_operator.py  becomes
>>   airflow/operators/**aws_sns_publish_operator.py*
>> 
>> 
>>   - *airflow/contrib/operators/dataproc_operator.py*
>>  becomes *airflow/operators/gcp_dataproc_operator.py*
>> 
>> 
>>   -
>> *airflow/contrib/hooks/gcp_bigtable_hook.py  *becomes *airflow/*
>>   *hooks/gcp_bigtable_hook.py*
>>   -
>> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
>>   *operators/ssh_operator.py*
>> 
>> 
>> 
>> *Option B*. Group operators/sensors/hooks in airflow/operators(sensors,
>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks), Drop the prefix.
>> 
>>   -
>> *airflow/contrib/operators/sns_publish_operator.py
>>   becomes airflow/operators/**aws/sns_publish_operator.py*
>> 
>> 
>>   - *airflow/contrib/operators/dataproc_operator.py*
>>  becomes *airflow/operators/gcp/dataproc_operator.py*
>> 
>> 
>>   -
>> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
>>   *operators/gcp/bigtable_operator.py*
>>   -
>> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
>>   *operators/ssh_operator.py*
>> 
>> 
>> *Option C*. Group operators/sensors/hooks in airflow/operators(sensors,
>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks), Keep the prefix.
>> 
>>   -
>> *airflow/contrib/operators/sns_publish_operator.py
>>   becomes airflow/operators/**aws/aws_sns_publish_operator.py*
>> 
>> 
>>   - *airflow/contrib/operators/dataproc_operator.py*
>>  becomes *airflow/operators/gcp/gcp_dataproc_operator.py*
>> 
>> 
>>   -
>> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
>>   *operators/gcp/gcp_bigtable_operator.py*
>>   -
>> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
>>   *operators/ssh_operator.py*
>> 
>> 
>> *Option D.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
>> hooks). Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks). Drop the prefix.
>> 
>>   -
>> *airflow/contrib/operators/sns_publish_operator.py
>>   becomes airflow/aws/operators/**sns_publish_operator.py*
>> 
>> 
>>   - *airflow/contrib/operators/dataproc_operator.py*
>>  becomes *airflow/gcp/operators/dataproc_operator.py*
>> 
>> 
>>   -
>> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
>>   *gcp/operators/bigtable_operator.py*
>>   -
>> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
>>   *operators/ssh_operator.py*
>> 
>> 
>> *Option E.* Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors,
>> hooks). Non-cloud provider ones are moved to
>> airflow/operators(sensors/hooks).* Keep the prefix*.
>> 
>>   -
>> *airflow/contrib/operators/sns_publish_operator.py
>>   becomes airflow/aws/operators/aws_**sns_publish_operator.py*
>> 
>> 
>>   - *airflow/contrib/operators/dataproc_operator.py*
>>  becomes *airflow/gcp/operators/gcp_dataproc_operator.py*
>> 
>> 
>>   -
>> *airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
>>   *gcp/operators/gcp_bigtable_operator.py*
>>   -
>> *airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
>>   *operators/ssh_operator.py*
>> 
>> 
>> Shall I proceed with updating the page ?
>> 
>> 
>> J.
>> 
>> 
>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <kaxilnaik@gmail.com> wrote:
>> 
>>> Thanks Jarek, adding my vote.
>>> 
>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <ash@apache.org> wrote:
>>> 
>>>> Thanks Jarek, much clearer. I have voted there too.
>>>> 
>>>> -ash
>>>> 
>>>> 
>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fokko@driesprong.frl>
>>>> wrote:
>>>>> 
>>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote
>>> there
>>>>> as well.
>>>>> 
>>>>> You've convinced me on the second case. I've changed my vote on Case 2
>>>> from
>>>>> A to B :-)
>>>>> 
>>>>> Maybe we should leave the vote open a bit longer since it involves
>>> quite
>>>>> some reading, and I would like to get some proper feedback and
>>> opinions
>>>> on
>>>>> this. Plus I only see two votes right now. If we don't think this
>>>> through,
>>>>> it might be that we're having this vote again in the near future :-)
>>>>> 
>>>>> Cheers, Fokko
>>>>> 
>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
>>>> Jarek.Potiuk@polidea.com>:
>>>>> 
>>>>>> I updated the AIP-21 proposal in the way that should answer all the
>>>>>> concerns in this thread.
>>>>>> 
>>>>>> NOTE! There is an action for all of those who are interested: Please
>>>>>> fill-in your voting in the voting table till Tuesday 30th of July 6pm
>>>> CEST
>>>>>> (5pm BST, 9 am PST).
>>>>>> 
>>>>>> I created a summary of the proposal
>>>>>> <
>>>>>> 
>>>> 
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
>>>>>>> 
>>>>>> in table form - which contains also examples for each case.
>>>>>> 
>>>>>> I added a voting table
>>>>>> <
>>>>>> 
>>>> 
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
>>>>>>> 
>>>>>> where you should add your votes:
>>>>>> 
>>>>>> I propose this voting mechanism:
>>>>>> 
>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST  (5 pm
>>> BST,
>>>> 9am
>>>>>> PST) .
>>>>>> 
>>>>>> After the choice, the final consistent set of choices will be
>>> announced
>>>>>> (taking into account majority of binding vote, also including
>>> potential
>>>>>> vetos and consistency between the choices. Non-binding votes will be
>>>> taken
>>>>>> into account in case there is a draw. The final set of choices will
>>> be
>>>>>> announced at devlist thread
>>>>>> <
>>>>>> 
>>>> 
>>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
>>>>>>> 
>>>>>> after
>>>>>> the voting completes.
>>>>>> 
>>>>>> Unless there is a veto raised to the final proposal, the final
>>> proposal
>>>> is
>>>>>> accepted by Lazy Consensus
>>>>>> <https://community.apache.org/committers/lazyConsensus.html>  on
>>>> *Friday*
>>>>>> 02
>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
>>>>>> 
>>>>>> Please let me know if what I proposed can be further improved to
>>> avoid
>>>>>> chaos.
>>>>>> 
>>>>>> J.
>>>>>> 
>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com
>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Ok. I see that indeed voting could have been organised a bit better
>>> -
>>>> dev
>>>>>>> list does not seem to cope well with such a complex case :).
>>>>>>> 
>>>>>>> As Kamil noticed - we already have a formal AIP-21
>>>>>>> <
>>>>>> 
>>>> 
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>>>>> 
>>>>>>> in the Wiki - which I should have mentioned before. I have not much
>>>> time
>>>>>>> today - but tomorrow I will try to summarise the options - based on
>>>> some
>>>>>>> real code examples (to make it easier to digest). I think the best
>>>>>> approach
>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where people
>>>> will
>>>>>>> be able to state their preferences for each case (by adding a row to
>>>> the
>>>>>>> table). I think that might provide much better clarity and a single
>>>> place
>>>>>>> which will show the current aggregated state of voting.
>>>>>>> 
>>>>>>> However - as Ash also stated - some of the options are
>>> inter-dependent
>>>> so
>>>>>>> at the end we will have to adjust the results in case they are not
>>>>>>> consistent  - and make final voting on aggregated set of cases -
>>> but I
>>>>>>> think in this case following "lasy consensus" - i,e. if noone
>>> objects
>>>> to
>>>>>>> final set of cases, we will proceed with it..
>>>>>>> 
>>>>>>> Do you think that will work better ?
>>>>>>> 
>>>>>>> J.
>>>>>>> 
>>>>>>> Principal Software Engineer
>>>>>>> Phone: +48660796129
>>>>>>> 
>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
>>>>>>> kamil.bregula@polidea.com> napisał:
>>>>>>> 
>>>>>>>> Document is also available as AIP on wiki:
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>>>>>>>> 
>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
>>> <fokko@driesprong.frl
>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of clarity,
>>> I
>>>>>>>> would
>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
>>>>>> structural
>>>>>>>>> code improvement AIP and then and add a matrix of users/cases
>>> where
>>>>>>>>> everybody can add his vote per case. This gives also the
>>> opportunity
>>>>>> for
>>>>>>>>> changing your vote on a specific case. WDYT?
>>>>>>>>> 
>>>>>>>>> Cheers, Fokko
>>>>>>>>> 
>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <cixuuz@gmail.com>:
>>>>>>>>> 
>>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than
>>> all
>>>>>>>>>> together.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <
>>> ash@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> A number of proposals overlap or add on to each other, so I
>>> think
>>>>>>>> (?) a
>>>>>>>>>>> single vote makes sense.
>>>>>>>>>>> 
>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
>>>>>>>> voting
>>>>>>>>> on.
>>>>>>>>>>> 
>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <kaxilnaik@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
>>>>>> how
>>>>>>>>>>>> about
>>>>>>>>>>>> separate vote emails for all the cases? Each email can have the
>>>>>>>>>>>> description
>>>>>>>>>>>> copied over from the doc.
>>>>>>>>>>>> 
>>>>>>>>>>>> It would probably be a bit easier to track a decision in the
>>>>>> future
>>>>>>>> as
>>>>>>>>>>>> well.
>>>>>>>>>>>> 
>>>>>>>>>>>> What do you guys think? @Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>> @Driesprong,
>>>>>>>>>>>> Fokko <fokko@driesprong.frl>
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <
>>> kaxilnaik@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
>>>>>>>>>>>>> 
>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
>>>>>>>>>>>>> 
>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
>>>>>>>>>>>>> 
>>>>>>>>>>>>> *Case #4* Separate namespace for resources :
>>>>>>>>>>>>> *Solution A*
>>>>>>>>>>>>> 
>>>>>>>>>>>>> *Case #5* *Operator : *Solution A*
>>>>>>>>>>>>> 
>>>>>>>>>>>>> *Case #7* : *Solution A*
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
>>>>>>>>>>>> <dpstandish@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary,
>>> but
>>>>>> I
>>>>>>>> am
>>>>>>>>>>>> a
>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
>>>>>> can
>>>>>>>>>>>> further
>>>>>>>>>>>>>> clarify.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
>>>>>>>> either
>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
>>>>>>>>>>>> focused on
>>>>>>>>>>>>>> cases 1,2,3 and 7.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> *Motion is to adopt the following:*
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib
>>>>>>>>>>>>>> All objects will be moved out of contrib folder and into
>>>>>>>> respective
>>>>>>>>>>>>>> non-contrib folder.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
>>>>>>>>>>>>>> airflow.operators.foo.FooOperator
>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
>>>>>>>> airflow.hooks.foo.FooHook
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.*
>>>>>>>>>>>>>> Example case is
>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py
>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is
>>> now
>>>>>>>> named
>>>>>>>>>>>> like
>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
>>>>>>>>>>>>>> So what is proposed for this case?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Case 4 - Solution B - *no change*
>>>>>>>>>>>>>> This proposal was about namespaces but the motion is to
>>> reject
>>>>>>>> the
>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Case 5 - Solution B - *no change*
>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
>>>>>>>>>>>> Operator in
>>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal --
>>> i.e.
>>>>>>>>>>>> motion is
>>>>>>>>>>>>>> no
>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
>>>>>>>>>>>> "Sensor")
>>>>>>>>>>>>>> suffixes on class names
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Case 6 - *No change*
>>>>>>>>>>>>>> This case concerns object naming inconsistencies, but motion
>>>>>> does
>>>>>>>>>>>> not
>>>>>>>>>>>>>> enact
>>>>>>>>>>>>>> any specific proposal.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
>>>>>> deprecation
>>>>>>>>>>>> warnings
>>>>>>>>>>>>>> (instead of zope)
>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> # in old_package/old_mod.py
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> import warnings
>>>>>>>>>>>>>> from new_package.new_mod import *
>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to
>>>>>>>> new_package.new_mod.
>>>>>>>>>>>>>> Import
>>>>>>>>>>>>>> of "
>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
>>>>>>>>>>>>>> DeprecationWarning, 2)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Reference implementation here:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib.
>>>>>> Thank
>>>>>>>> you
>>>>>>>>>>>>>> Sergio
>>>>>>>>>>>>>> for turning the tide with your effective argumentation ;)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
>>>>>>>> ash@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Could someone summarise what the proposed name changes are,
>>>>>>>> here,
>>>>>>>>>>>> in
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial
>>>>>>>> examples
>>>>>>>>>>>> is 1)
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
>>>>>> 2)
>>>>>>>>>>>> using a
>>>>>>>>>>>>>>> google doc isn't great for a longer term record.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Ash
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
>>>>>>>>>>>> <Jarek.Potiuk@polidea.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
>>>>>>>> would
>>>>>>>>>>>> like
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
>>>>>>>>>>>> (binding)
>>>>>>>>>>>>>> votes
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the
>>>>>>>>>>>> previous
>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Here is the proposal (details here
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,*
>>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator
>>>>>>>> could
>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
>>>>>>>> class
>>>>>>>>>>>>>> names*
>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>>>>>>>> (and
>>>>>>>>>>>> my team
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE
>>>>>>>>>>>> integration)*
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This is my binding (+1) vote.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
>>>>>>>>>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hey Felix,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
>>>>>> (I
>>>>>>>>>>>> use IDE
>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>> often ;) )
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
>>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
>>>>>>>> able
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> respond
>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
>>>>>>>> Europe
>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
>>>>>> into
>>>>>>>> a
>>>>>>>>>>>> single
>>>>>>>>>>>>>>>>>> package.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
>>>>>>>> to
>>>>>>>>>>>> get
>>>>>>>>>>>>>> rid of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
>>>>>>>>>>>> tricky
>>>>>>>>>>>>>>> question,
>>>>>>>>>>>>>>>>>> I think this is already reflected in the document since
>>>>>>>>>>>> "maturity"
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>> core
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
>>>>>> incubating
>>>>>>>>>>>> will
>>>>>>>>>>>>>> confuse
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with
>>>>>>>> Airflow it
>>>>>>>>>>>> isn't
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if you
>>>>>>>>>>>> don't use
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>> IDE
>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
>>>>>>>> changing
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>> API just for a different name.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
>>>>>>>> will
>>>>>>>>>>>> make
>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>> as we will not have to define other rules as those that we
>>>>>>>>>>>> already
>>>>>>>>>>>>>> have.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
>>>>>> +0)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for
>>>>>>>>>>>> BashOperator.
>>>>>>>>>>>>>>> After
>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
>>>>>> mail
>>>>>>>> you
>>>>>>>>>>>>>> state: -
>>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
>>>>>>>> could
>>>>>>>>>>>> become
>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
>>>>>>>>>>>> keeping it
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
>>>>>>>>>>>> confusion
>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
>>>>>> for
>>>>>>>>>>>> changing
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> *module* name not the *class* name. So
>>>>>>>>>>>> bash_operator.BashOperator ->
>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for
>>>>>>>> BashOperator
>>>>>>>>>>>> in this
>>>>>>>>>>>>>>> case.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
>>>>>>>> its
>>>>>>>>>>>> own
>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
>>>>>>>> like
>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own package
>>>>>>>>>>>> because
>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
>>>>>>>> bit
>>>>>>>>>>>>>> painful
>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move
>>>>>>>> everything
>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> introduces noise into version control etc.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
>>>>>>>>>>>> difficult as
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> seems to group operators in packages and it has numerous
>>>>>>>>>>>> advantages.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the
>>>>>>>>>>>> __init__.py of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will
>>>>>>>> actually
>>>>>>>>>>>>>> realise
>>>>>>>>>>>>>>>>> that the file has been moved and you can follow that.
>>>>>> Maybe
>>>>>>>> not
>>>>>>>>>>>> as
>>>>>>>>>>>>>> super
>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4  but it's not a
>>>>>> rocket
>>>>>>>>>>>> science
>>>>>>>>>>>>>>> either.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Apart from all, great work!
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
>>>>>>>>>>>>>>>>>> <feluelle@pm.me.invalid
>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
>>>>>> these
>>>>>>>>>>>> cases.
>>>>>>>>>>>>>> So I
>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>> with +1 (non-binding).
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
>>>>>>>> intense
>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
>>>>>> creating
>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> document. I
>>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent
>>>>>> and
>>>>>>>>>>>> clean :)
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>> Felix
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -------- Original Message --------
>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
>>>>>>>> paths.
>>>>>>>>>>>> It's
>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>> discussed in
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
>>>>>>>> CEST),
>>>>>>>>>>>> and
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> least
>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from
>>>>>> Kamil
>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The proposed solution is:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
>>>>>> "well"
>>>>>>>>>>>> tested
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
>>>>>>>>>>>>>>>>>>>> - *Case 2: B:
>>>>>> Airflow.operators.foo_operator.FooOperator
>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
>>>>>>>>>>>>>>>>>>>> - *Case 3: C:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
>>>>>> on
>>>>>>>>>>>> class
>>>>>>>>>>>>>> names*
>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
>>>>>>>> (and
>>>>>>>>>>>> my
>>>>>>>>>>>>>> team
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)*
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
>>>>>> Software
>>>>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>>>>>>>>>>>> Engineer
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
>>>>>>>>> Engineer
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *Kaxil Naik*
>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>>>>>> Spark
>>>>>>>> &
>>>>>>>>>>>> Neo4j
>>>>>>>>>>>>> Developer
>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> *Kaxil Naik*
>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
>>> Spark
>>>>>> &
>>>>>>>>>>>> Neo4j
>>>>>>>>>>>> Developer
>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>> 
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>> 
>>>> 
>>>> 
>>> 
>>> --
>>> *Kaxil Naik*
>>> *Big Data Consultant | DevOps Data Engineer*
>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
>>> Developer
>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>> 
>> 
>> 
>> --
>> 
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> 
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>> 
>> 
> 
> -- 
> 
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
> 
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>


Mime
View raw message