airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Standish <dpstand...@gmail.com>
Subject Re: AIP-21 (Move operators to Core) - "cross_transfer" packages
Date Sat, 21 Sep 2019 20:00:22 GMT
I like putting them with source.

On Sat, Sep 21, 2019, 12:57 PM Philippe Gagnon <philgagnon1@gmail.com>
wrote:

> Another way to resolve this could be to define a convention such that a
> "cross transfer" operator should belong to either the source or destination
> cloud provider's package.
>
> Personally though I do not see any technical issues with having a
> "cross_transfer" package, but I don't find the name to roll off the tongue
> very well. ;-)
>
> On Sat, Sep 21, 2019 at 3:53 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> wrote:
>
> > I have a question: Should we put all transfer operators between into
> > separate "cross_transfer" package ?
> >
> > *Context:*
> >
> > We had one unresolved point when we decided about AIP-21 - where to put
> > transfer operators between service providers. In the middle of
> implementing
> > it, it turned out that we need to make some decisions as it has some
> > undesirable side effects if we just move the transfer operators to core
> > without any structure. Detailed discussion in this PR:
> > https://github.com/apache/airflow/pull/6147
> >
> > We can solve it easily by choosing "cross_transfer" package for all
> > transfer operators that are crossing "service provider" boundary.
> >
> > This way we will have "gcp" (or maybe even "alphabet" soon), "aws",
> "azure"
> > etc. and "cross_transfer" for all the S3->GCP, AWS->S3 etc.
> >
> > What do you think? Anyone strongly against this? Or maybe we can follow
> > lazy consensus rule for this? Or maybe someone can come up with a better
> > name :) ?
> >
> > J.
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message