airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject AIP-21 (Move operators to Core) - "cross_transfer" packages
Date Sat, 21 Sep 2019 19:52:44 GMT
I have a question: Should we put all transfer operators between into
separate "cross_transfer" package ?

*Context:*

We had one unresolved point when we decided about AIP-21 - where to put
transfer operators between service providers. In the middle of implementing
it, it turned out that we need to make some decisions as it has some
undesirable side effects if we just move the transfer operators to core
without any structure. Detailed discussion in this PR:
https://github.com/apache/airflow/pull/6147

We can solve it easily by choosing "cross_transfer" package for all
transfer operators that are crossing "service provider" boundary.

This way we will have "gcp" (or maybe even "alphabet" soon), "aws", "azure"
etc. and "cross_transfer" for all the S3->GCP, AWS->S3 etc.

What do you think? Anyone strongly against this? Or maybe we can follow
lazy consensus rule for this? Or maybe someone can come up with a better
name :) ?

J.

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message