airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: [VOTE] Changes in import paths
Date Tue, 30 Jul 2019 07:42:52 GMT
I think almost everyone voted and we have almost perfect consensus. We all
agree amongst other on moving all operators out of contrib (Great!).

The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4*
(Using Namespaces).
I think there was actually an overlap between those two. Also Ash's
comment/veto on that is quite clear.
So I have final (I hope!) proposal to join those two cases and have one
voting (till Friday) only on that.

I will update the doc if you all agree with me that makes more sense as
single case to vote on:

*[Case 3 + Case 4]: Grouping Cloud Provider operators.*

*Option A*. Keep operators/sensors/hooks in airflow/operators(sensors,
hooks) and keep/add prefixes in file names.

   -
*airflow/contrib/operators/sns_publish_operator.py  becomes
   airflow/operators/**aws_sns_publish_operator.py*


   - *airflow/contrib/operators/dataproc_operator.py*
  becomes *airflow/operators/gcp_dataproc_operator.py*


   -
*airflow/contrib/hooks/gcp_bigtable_hook.py  *becomes *airflow/*
   *hooks/gcp_bigtable_hook.py*
   -
*airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
   *operators/ssh_operator.py*



*Option B*. Group operators/sensors/hooks in airflow/operators(sensors,
hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to
airflow/operators(sensors/hooks), Drop the prefix.

   -
*airflow/contrib/operators/sns_publish_operator.py
   becomes airflow/operators/**aws/sns_publish_operator.py*


   - *airflow/contrib/operators/dataproc_operator.py*
  becomes *airflow/operators/gcp/dataproc_operator.py*


   -
*airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
   *operators/gcp/bigtable_operator.py*
   -
*airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
   *operators/ssh_operator.py*


*Option C*. Group operators/sensors/hooks in airflow/operators(sensors,
hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to
airflow/operators(sensors/hooks), Keep the prefix.

   -
*airflow/contrib/operators/sns_publish_operator.py
   becomes airflow/operators/**aws/aws_sns_publish_operator.py*


   - *airflow/contrib/operators/dataproc_operator.py*
  becomes *airflow/operators/gcp/gcp_dataproc_operator.py*


   -
*airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
   *operators/gcp/gcp_bigtable_operator.py*
   -
*airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
   *operators/ssh_operator.py*


*Option D.* Group operators/sensors/hooks in
*airflow/<PROVIDER>*/operators(sensors,
hooks). Non-cloud provider ones are moved to
airflow/operators(sensors/hooks). Drop the prefix.

   -
*airflow/contrib/operators/sns_publish_operator.py
   becomes airflow/aws/operators/**sns_publish_operator.py*


   - *airflow/contrib/operators/dataproc_operator.py*
  becomes *airflow/gcp/operators/dataproc_operator.py*


   -
*airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
   *gcp/operators/bigtable_operator.py*
   -
*airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
   *operators/ssh_operator.py*


*Option E.* Group operators/sensors/hooks in
*airflow/<PROVIDER>*/operators(sensors,
hooks). Non-cloud provider ones are moved to
airflow/operators(sensors/hooks).* Keep the prefix*.

   -
*airflow/contrib/operators/sns_publish_operator.py
   becomes airflow/aws/operators/aws_**sns_publish_operator.py*


   - *airflow/contrib/operators/dataproc_operator.py*
  becomes *airflow/gcp/operators/gcp_dataproc_operator.py*


   -
*airflow/contrib/operators/gcp_bigtable_operator.py  *becomes *airflow/*
   *gcp/operators/gcp_bigtable_operator.py*
   -
*airflow/contrib/operators/ssh_operator.py  *becomes *airflow/*
   *operators/ssh_operator.py*


Shall I proceed with updating the page ?


J.


On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <kaxilnaik@gmail.com> wrote:

> Thanks Jarek, adding my vote.
>
> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <ash@apache.org> wrote:
>
> > Thanks Jarek, much clearer. I have voted there too.
> >
> > -ash
> >
> >
> > > On 29 Jul 2019, at 08:13, Driesprong, Fokko <fokko@driesprong.frl>
> > wrote:
> > >
> > > Thanks Jarek, the Wiki works much better for me. I've added my vote
> there
> > > as well.
> > >
> > > You've convinced me on the second case. I've changed my vote on Case 2
> > from
> > > A to B :-)
> > >
> > > Maybe we should leave the vote open a bit longer since it involves
> quite
> > > some reading, and I would like to get some proper feedback and opinions
> > on
> > > this. Plus I only see two votes right now. If we don't think this
> > through,
> > > it might be that we're having this vote again in the near future :-)
> > >
> > > Cheers, Fokko
> > >
> > > Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>:
> > >
> > >> I updated the AIP-21 proposal in the way that should answer all the
> > >> concerns in this thread.
> > >>
> > >> NOTE! There is an action for all of those who are interested: Please
> > >> fill-in your voting in the voting table till Tuesday 30th of July 6pm
> > CEST
> > >> (5pm BST, 9 am PST).
> > >>
> > >> I created a summary of the proposal
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal
> > >>>
> > >> in table form - which contains also examples for each case.
> > >>
> > >> I added a voting table
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting
> > >>>
> > >> where you should add your votes:
> > >>
> > >> I propose this voting mechanism:
> > >>
> > >> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST  (5 pm BST,
> > 9am
> > >> PST) .
> > >>
> > >> After the choice, the final consistent set of choices will be
> announced
> > >> (taking into account majority of binding vote, also including
> potential
> > >> vetos and consistency between the choices. Non-binding votes will be
> > taken
> > >> into account in case there is a draw. The final set of choices will be
> > >> announced at devlist thread
> > >> <
> > >>
> >
> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E
> > >>>
> > >> after
> > >> the voting completes.
> > >>
> > >> Unless there is a veto raised to the final proposal, the final
> proposal
> > is
> > >> accepted by Lazy Consensus
> > >> <https://community.apache.org/committers/lazyConsensus.html>  on
> > *Friday*
> > >> 02
> > >> Aug 2019 at 6pm CEST (5 pm BST, 9am PST).
> > >>
> > >> Please let me know if what I proposed can be further improved to avoid
> > >> chaos.
> > >>
> > >> J.
> > >>
> > >> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > >> wrote:
> > >>
> > >>> Ok. I see that indeed voting could have been organised a bit better -
> > dev
> > >>> list does not seem to cope well with such a complex case :).
> > >>>
> > >>> As Kamil noticed - we already have a formal AIP-21
> > >>> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >>>
> > >>> in the Wiki - which I should have mentioned before. I have not much
> > time
> > >>> today - but tomorrow I will try to summarise the options - based on
> > some
> > >>> real code examples (to make it easier to digest). I think the best
> > >> approach
> > >>> will be to make a voting matrix in the AIP-21 Wiki page where people
> > will
> > >>> be able to state their preferences for each case (by adding a row to
> > the
> > >>> table). I think that might provide much better clarity and a single
> > place
> > >>> which will show the current aggregated state of voting.
> > >>>
> > >>> However - as Ash also stated - some of the options are
> inter-dependent
> > so
> > >>> at the end we will have to adjust the results in case they are not
> > >>> consistent  - and make final voting on aggregated set of cases - but
> I
> > >>> think in this case following "lasy consensus" - i,e. if noone objects
> > to
> > >>> final set of cases, we will proceed with it..
> > >>>
> > >>> Do you think that will work better ?
> > >>>
> > >>> J.
> > >>>
> > >>> Principal Software Engineer
> > >>> Phone: +48660796129
> > >>>
> > >>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła <
> > >>> kamil.bregula@polidea.com> napisał:
> > >>>
> > >>>> Document is also available as AIP on wiki:
> > >>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
> > >>>>
> > >>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko
> <fokko@driesprong.frl
> > >
> > >>>> wrote:
> > >>>>
> > >>>>> I agree with all, this is a bit chaotic. For the sake of clarity, I
> > >>>> would
> > >>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a
> > >> structural
> > >>>>> code improvement AIP and then and add a matrix of users/cases where
> > >>>>> everybody can add his vote per case. This gives also the
> opportunity
> > >> for
> > >>>>> changing your vote on a specific case. WDYT?
> > >>>>>
> > >>>>> Cheers, Fokko
> > >>>>>
> > >>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <cixuuz@gmail.com>:
> > >>>>>
> > >>>>>> I agree with Ash. It is clear to vote on each case rather than all
> > >>>>>> together.
> > >>>>>>
> > >>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <ash@apache.org
> >
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> A number of proposals overlap or add on to each other, so I think
> > >>>> (?) a
> > >>>>>>> single vote makes sense.
> > >>>>>>>
> > >>>>>>> To be clear, my vote is -1 until it's clear exactly what we are
> > >>>> voting
> > >>>>> on.
> > >>>>>>>
> > >>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <kaxilnaik@gmail.com>
> > >>>> wrote:
> > >>>>>>>> I agree with Ash and instead of voting on all 7 Cases together,
> > >> how
> > >>>>>>>> about
> > >>>>>>>> separate vote emails for all the cases? Each email can have the
> > >>>>>>>> description
> > >>>>>>>> copied over from the doc.
> > >>>>>>>>
> > >>>>>>>> It would probably be a bit easier to track a decision in the
> > >> future
> > >>>> as
> > >>>>>>>> well.
> > >>>>>>>>
> > >>>>>>>> What do you guys think? @Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > >>>>>>>> @Driesprong,
> > >>>>>>>> Fokko <fokko@driesprong.frl>
> > >>>>>>>>
> > >>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <kaxilnaik@gmail.com
> >
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A *
> > >>>>>>>>>
> > >>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A *
> > >>>>>>>>>
> > >>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C*
> > >>>>>>>>>
> > >>>>>>>>> *Case #4* Separate namespace for resources :
> > >>>>>>>>> *Solution A*
> > >>>>>>>>>
> > >>>>>>>>> *Case #5* *Operator : *Solution A*
> > >>>>>>>>>
> > >>>>>>>>> *Case #7* : *Solution A*
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish
> > >>>>>>>> <dpstandish@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I have tried to add some clarification to Jarek's summary, but
> > >> I
> > >>>> am
> > >>>>>>>> a
> > >>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek
> > >> can
> > >>>>>>>> further
> > >>>>>>>>>> clarify.
> > >>>>>>>>>>
> > >>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all
> > >>>> either
> > >>>>>>>>>> proposal
> > >>>>>>>>>> *rejections* or otherwise have no effect, so attention can be
> > >>>>>>>> focused on
> > >>>>>>>>>> cases 1,2,3 and 7.
> > >>>>>>>>>>
> > >>>>>>>>>> *Motion is to adopt the following:*
> > >>>>>>>>>>
> > >>>>>>>>>> Case 1 - Solution A - Get rid of contrib
> > >>>>>>>>>> All objects will be moved out of contrib folder and into
> > >>>> respective
> > >>>>>>>>>> non-contrib folder.
> > >>>>>>>>>>
> > >>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules
> > >>>>>>>>>> Example:
> > >>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes
> > >>>>>>>>>> airflow.operators.foo.FooOperator
> > >>>>>>>>>> Example:
> > >>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes
> > >>>> airflow.hooks.foo.FooHook
> > >>>>>>>>>>
> > >>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes
> > >>>>>>>>>> *I am not sure exactly what the proposal is.*
> > >>>>>>>>>> Example case is
> > >>>> airflow/contrib/operators/gcp_bigtable_operator.py
> > >>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is now
> > >>>> named
> > >>>>>>>> like
> > >>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py
> > >>>>>>>>>> So what is proposed for this case?
> > >>>>>>>>>>
> > >>>>>>>>>> Case 4 - Solution B - *no change*
> > >>>>>>>>>> This proposal was about namespaces but the motion is to reject
> > >>>> the
> > >>>>>>>>>> proposal.
> > >>>>>>>>>>
> > >>>>>>>>>> Case 5 - Solution B - *no change*
> > >>>>>>>>>> The proposal was to remove class type suffix e.g. remove the
> > >>>>>>>> Operator in
> > >>>>>>>>>> FooOperator, but the motion is to reject this proposal -- i.e.
> > >>>>>>>> motion is
> > >>>>>>>>>> no
> > >>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and
> > >>>>>>>> "Sensor")
> > >>>>>>>>>> suffixes on class names
> > >>>>>>>>>>
> > >>>>>>>>>> Case 6 - *No change*
> > >>>>>>>>>> This case concerns object naming inconsistencies, but motion
> > >> does
> > >>>>>>>> not
> > >>>>>>>>>> enact
> > >>>>>>>>>> any specific proposal.
> > >>>>>>>>>>
> > >>>>>>>>>> Case 7 - Solution A - use warnings.warn template for
> > >> deprecation
> > >>>>>>>> warnings
> > >>>>>>>>>> (instead of zope)
> > >>>>>>>>>> Example:
> > >>>>>>>>>>
> > >>>>>>>>>> # in old_package/old_mod.py
> > >>>>>>>>>>
> > >>>>>>>>>> import warnings
> > >>>>>>>>>> from new_package.new_mod import *
> > >>>>>>>>>> warnings.warn("old_package.old_mod has moved to
> > >>>> new_package.new_mod.
> > >>>>>>>>>> Import
> > >>>>>>>>>> of "
> > >>>>>>>>>> "old_package.old_mod will become unsupported in version 2",
> > >>>>>>>>>> DeprecationWarning, 2)
> > >>>>>>>>>>
> > >>>>>>>>>> Reference implementation here:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3.
> > >>>>>>>>>>
> > >>>>>>>>>> I am very happy that this motion now gets rid of contrib.
> > >> Thank
> > >>>> you
> > >>>>>>>>>> Sergio
> > >>>>>>>>>> for turning the tide with your effective argumentation ;)
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor <
> > >>>> ash@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Could someone summarise what the proposed name changes are,
> > >>>> here,
> > >>>>>>>> in
> > >>>>>>>>>> this
> > >>>>>>>>>>> thread? Pointing at a google doc and only giving partial
> > >>>> examples
> > >>>>>>>> is 1)
> > >>>>>>>>>> a
> > >>>>>>>>>>> bit difficult to quickly work out what we are voting on, and
> > >> 2)
> > >>>>>>>> using a
> > >>>>>>>>>>> google doc isn't great for a longer term record.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> Ash
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk
> > >>>>>>>> <Jarek.Potiuk@polidea.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I
> > >>>> would
> > >>>>>>>> like
> > >>>>>>>>>> to
> > >>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1
> > >>>>>>>> (binding)
> > >>>>>>>>>> votes
> > >>>>>>>>>>> are
> > >>>>>>>>>>>> needed to pass. I marked in red the modifications to the
> > >>>>>>>> previous
> > >>>>>>>>>>> proposal.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Here is the proposal (details here
> > >>>>>>>>>>>> <
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> )
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  - *Case 1: A: Get rid of contrib,*
> > >>>>>>>>>>>>  - *Case 2: A: Airflow.operators.foo_operator.FooOperator
> > >>>> could
> > >>>>>>>>>>>>  become airflow.operators.foo.FooOperator*
> > >>>>>>>>>>>>  - *Case 3: C:
> > >>>>>>>>>>>>
> > >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >>>>>>>>>> could
> > >>>>>>>>>>>>  become airflow.gcp.operators.bigtable.BigTableOperator*
> > >>>>>>>>>>>>  - *Case 4: B: no namespace introduction*
> > >>>>>>>>>>>>  - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on
> > >>>> class
> > >>>>>>>>>> names*
> > >>>>>>>>>>>>  - *Case 6: We will treat isolated cases on case-by-case
> > >>>> (and
> > >>>>>>>> my team
> > >>>>>>>>>>> can
> > >>>>>>>>>>>>  do the job of GCP-related operators)*
> > >>>>>>>>>>>>  - *Case 7: Native python solution (with better IDE
> > >>>>>>>> integration)*
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This is my binding (+1) vote.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk <
> > >>>>>>>>>> Jarek.Potiuk@polidea.com>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hey Felix,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well
> > >> (I
> > >>>>>>>> use IDE
> > >>>>>>>>>>> quite
> > >>>>>>>>>>>>> often ;) )
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> J.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko
> > >>>>>>>>>> <fokko@driesprong.frl
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't
> > >>>> able
> > >>>>>>>> to
> > >>>>>>>>>>> respond
> > >>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout
> > >>>> Europe
> > >>>>>>>> :-)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib
> > >> into
> > >>>> a
> > >>>>>>>> single
> > >>>>>>>>>>>>>> package.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> To make it easier to the end-user, my preference would be
> > >>>> to
> > >>>>>>>> get
> > >>>>>>>>>> rid of
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a
> > >>>>>>>> tricky
> > >>>>>>>>>>> question,
> > >>>>>>>>>>>>>> I think this is already reflected in the document since
> > >>>>>>>> "maturity"
> > >>>>>>>>>> is
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator
> > >>>> to
> > >>>>>>>> the
> > >>>>>>>>>> core
> > >>>>>>>>>>> or
> > >>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to
> > >> incubating
> > >>>>>>>> will
> > >>>>>>>>>> confuse
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> user even more. For people who aren't as known with
> > >>>> Airflow it
> > >>>>>>>> isn't
> > >>>>>>>>>>> that
> > >>>>>>>>>>>>>> transparent which operator lives where, especially if you
> > >>>>>>>> don't use
> > >>>>>>>>>> an
> > >>>>>>>>>>> IDE
> > >>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth
> > >>>> changing
> > >>>>>>>> the
> > >>>>>>>>>>> public
> > >>>>>>>>>>>>>> API just for a different name.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It
> > >>>> will
> > >>>>>>>> make
> > >>>>>>>>>>> easier
> > >>>>>>>>>>>>> as we will not have to define other rules as those that we
> > >>>>>>>> already
> > >>>>>>>>>> have.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a
> > >> +0)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I like to search on Github with t, and then search for
> > >>>>>>>> BashOperator.
> > >>>>>>>>>>> After
> > >>>>>>>>>>>>>> the change, you should search for Operator/Bash.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your
> > >> mail
> > >>>> you
> > >>>>>>>>>> state: -
> > >>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator
> > >>>> could
> > >>>>>>>> become
> > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is
> > >>>>>>>> keeping it
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>> same, so that would mean - *Case 2: B:
> > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay
> > >>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator*
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little
> > >>>>>>>> confusion
> > >>>>>>>>>>> here.
> > >>>>>>>>>>>>> This case is not for changing BashOperator into Bash but
> > >> for
> > >>>>>>>> changing
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>> *module* name not the *class* name. So
> > >>>>>>>> bash_operator.BashOperator ->
> > >>>>>>>>>>>>> bash.BashOperator :) . you will still search for
> > >>>> BashOperator
> > >>>>>>>> in this
> > >>>>>>>>>>> case.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 3:* I'm leaning towards B
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets
> > >>>> its
> > >>>>>>>> own
> > >>>>>>>>>>> package
> > >>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies
> > >>>> like
> > >>>>>>>>>>> Databricks
> > >>>>>>>>>>>>>> and Qubole which might also end up with their own package
> > >>>>>>>> because
> > >>>>>>>>>> their
> > >>>>>>>>>>>>>> integration is getting richer over time. Moving this is a
> > >>>> bit
> > >>>>>>>>>> painful
> > >>>>>>>>>>>>>> because we need to deprecate the old path and move
> > >>>> everything
> > >>>>>>>> which
> > >>>>>>>>>>>>>> introduces noise into version control etc.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as
> > >>>>>>>> difficult as
> > >>>>>>>>>> it
> > >>>>>>>>>>>>> seems to group operators in packages and it has numerous
> > >>>>>>>> advantages.
> > >>>>>>>>>> The
> > >>>>>>>>>>>>> nice thing about deprecations - we can do them in the
> > >>>>>>>> __init__.py of
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> packages which causes the noise fairly small (Git will
> > >>>> actually
> > >>>>>>>>>> realise
> > >>>>>>>>>>>>> that the file has been moved and you can follow that.
> > >> Maybe
> > >>>> not
> > >>>>>>>> as
> > >>>>>>>>>> super
> > >>>>>>>>>>>>> easy to merge changes later to 1.10.4  but it's not a
> > >> rocket
> > >>>>>>>> science
> > >>>>>>>>>>> either.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> *Case 7:* Python native solution
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Apart from all, great work!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cheers, Fokko
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall
> > >>>>>>>>>>>>>> <feluelle@pm.me.invalid
> > >>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I have no strong "No" against any proposed change of
> > >> these
> > >>>>>>>> cases.
> > >>>>>>>>>> So I
> > >>>>>>>>>>>>>> go
> > >>>>>>>>>>>>>>> with +1 (non-binding).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your
> > >>>> intense
> > >>>>>>>> work
> > >>>>>>>>>>>>>> towards
> > >>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even
> > >> creating
> > >>>>>>>> this
> > >>>>>>>>>>>>>> document. I
> > >>>>>>>>>>>>>>> like how the code is getting more and more consistent
> > >> and
> > >>>>>>>> clean :)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Kind regards,
> > >>>>>>>>>>>>>>> Felix
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Sent from ProtonMail mobile
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> -------- Original Message --------
> > >>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This email is calling a vote on the changes in import
> > >>>> paths.
> > >>>>>>>> It's
> > >>>>>>>>>>> been
> > >>>>>>>>>>>>>>>> discussed in
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm
> > >>>> CEST),
> > >>>>>>>> and
> > >>>>>>>>>> at
> > >>>>>>>>>>>>>> least
> > >>>>>>>>>>>>>>>> three +1 (binding) votes have been cast.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The proposal to vote is based on the document from
> > >> Kamil
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The proposed solution is:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are
> > >> "well"
> > >>>>>>>> tested
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.*
> > >>>>>>>>>>>>>>>> - *Case 2: B:
> > >> Airflow.operators.foo_operator.FooOperator
> > >>>>>>>> could
> > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator*
> > >>>>>>>>>>>>>>>> - *Case 3: C:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator
> > >>>>>>>>>>> could
> > >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator*
> > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction*
> > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes
> > >> on
> > >>>>>>>> class
> > >>>>>>>>>> names*
> > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case
> > >>>> (and
> > >>>>>>>> my
> > >>>>>>>>>> team
> > >>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>> do the job of GCP-related operators)*
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This is my (binding) +1 vote.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> J.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
> > >> Software
> > >>>>>>>> Engineer
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129)
> > >>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)>
> > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > >>>>>>>> Engineer
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jarek Potiuk
> > >>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
> > >>>>> Engineer
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> *Kaxil Naik*
> > >>>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> > >> Spark
> > >>>> &
> > >>>>>>>> Neo4j
> > >>>>>>>>> Developer
> > >>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> *Kaxil Naik*
> > >>>>>>>> *Big Data Consultant | DevOps Data Engineer*
> > >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache
> Spark
> > >> &
> > >>>>>>>> Neo4j
> > >>>>>>>> Developer
> > >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> >
> >
>
> --
> *Kaxil Naik*
> *Big Data Consultant | DevOps Data Engineer*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *LinkedIn*: https://www.linkedin.com/in/kaxil
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message