airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomasz Urbaszek <tomasz.urbas...@polidea.com>
Subject Re: AIP-21 - grouping google operators
Date Fri, 04 Oct 2019 19:35:33 GMT
I agree with Jarek. Using the structure proposed by Kamil sounds really
sensible to me.

Tomek

On Fri, 4 Oct 2019 at 21:27, Jarek Potiuk <Jarek.Potiuk@polidea.com> wrote:

> Side comment:
>
> In the near future (AIP-26 ??) we can also adopt the structure reflected in
> the current documentation restructuring by Kamil
>
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#
>
>
>    -
>
>    Fundamentals
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#fundamentals
> >
>    -
>
>    ASF: Apache Software Foundation
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#asf-apache-software-foundation
> >
>    -
>
>    Service integrations
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#service-integrations
> >
>    -
>
>    Software integrations
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#software-integrations
> >
>    -
>
>    Protocol integrations
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#protocol-integrations
> >
>
> Providers:
>
>    - Azure: Microsoft Azure
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#azure-microsoft-azure
> >
>    - AWS: Amazon Web Services
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#aws-amazon-web-services
> >
>    - GCP: Google Cloud Platform
>    <
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#gcp-google-cloud-platform
> >
>
> This will make it much cleaner on the airflow.* package level.
>
> J.
>
> On Fri, Oct 4, 2019 at 9:05 PM Driesprong, Fokko <fokko@driesprong.frl>
> wrote:
>
> > I agree with Jarek's suggestion to add another module. Personally I don't
> > like having too many on the airflow._ level, there are quite a few
> already.
> >
> > Also, I would also go for providers. Mostly because we have hook*s* and
> > operator*s*. When there are a lot of {operators,hooks,sensors} it makes
> > sense to package these in a separate module. I could imagine that other
> > companies that provide managed services
> > <
> >
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#service-integrations
> > >
> > might
> > evolve to a separate package, but maybe we should come up with some
> > criteria.
> >
> > Cheers, Fokko
> >
> > Op vr 4 okt. 2019 om 19:00 schreef Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >:
> >
> > > Not that I want to open Pandora's box :). But I think the longer name -
> > the
> > > worse. Provider is quite nice and short.
> > >
> > > I agree with plural 'providers' (as we have hooks and operators). So
> for
> > > consistency it should be indeed plural.
> > >
> > > J.
> > >
> > > pt., 4 paź 2019, 18:48 użytkownik Chris Palmer <chris@crpalmer.com>
> > > napisał:
> > >
> > > > My question about Oracle/MySql wasn't a serious one, but I forget
> > > sometimes
> > > > that sarcasm doesn't come across well on email.
> > > >
> > > > I guess my objection is that I don't think that 'provider' adds
> > anything
> > > of
> > > > value. I'm not convinced that there needs to be a level between
> > 'airflow'
> > > > and 'google' but if going that route I would advocate for at least
> the
> > > > plural form 'providers' or the more descriptive 'cloud_providers'.
> > > >
> > > > Chris
> > > >
> > > > On Fri, Oct 4, 2019 at 11:57 AM Kamil Breguła <
> > kamil.bregula@polidea.com
> > > >
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > We think that only cloud providers should be separated from others,
> > > > > because these services are integrated with each other. Very often,
> > > > > when you use one cloud provider, you use many services of a given
> > > > > provider. Using a single provider solution provides a uniform way
> of
> > > > > authorization, etc. A large number of problems and mechanisms are
> > > > > common to one provider. You can see the amount of integration from
> > > > > Microsoft, Azure, Google on the reference list.
> > > > >
> > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
> > > > > In this list, cloud reference providers have been placed in
> separate
> > > > > tables because they have a very large number of services. If Oracle
> > > > > will have many integrations, it is worth emphasizing this fact and
> > > > > moving these integrations to a separate place, so that it is easier
> > to
> > > > > find them. and use. Keeping all possible files in one place makes
> it
> > > > > very difficult to use them.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > On Fri, Oct 4, 2019 at 5:32 PM Chris Palmer <chris@crpalmer.com>
> > > wrote:
> > > > > >
> > > > > > This seems unnecessary to me.
> > > > > >
> > > > > > Is everything going to be under some 'provider' or just certain
> > sets
> > > of
> > > > > > operators, and if so what differentiates when something should
be
> > > > under a
> > > > > > provider or not? For example, are the mysql operators going
to go
> > > under
> > > > > > 'provider/oracle/'?
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, Oct 4, 2019 at 9:21 AM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Agree with Ash.
> > > > > > >
> > > > > > > After doing the gcp move and seeing the result we agreed
that
> > > > > 'provider' is
> > > > > > > better as additional prefix.
> > > > > > >
> > > > > > > If no-one objects (Lazy Consensus
> > > > > > > <https://community.apache.org/committers/lazyConsensus.html>)
> > till
> > > > > Monday
> > > > > > > 3.20 CEST, we will update AIP-21 and move the gcp operators
to
> > > > > > > *provider/google/[gcp,gsuite]*.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
-- 

Tomasz Urbaszek
Polidea <https://www.polidea.com/> | Junior Software Engineer

M: +48 505 628 493 <+48505628493>
E: tomasz.urbaszek@polidea.com <tomasz.urbaszeki@polidea.com>

Unique Tech
Check out our projects! <https://www.polidea.com/our-work>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message