airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil Breguła <kamil.breg...@polidea.com>
Subject Re: [2.0 spring cleaning] Changes in import paths
Date Thu, 18 Apr 2019 18:10:02 GMT
On Thu, Apr 18, 2019 at 7:30 PM Arthur Wiedmer <arthur.wiedmer@gmail.com>
wrote:

> Hi Kamil,
>
> Thank you for this doc, it was a lot of good work to put all of this
> together. Could you open the doc for comment?
> May I recommend that once comments have been added to eventually port the
> doc to the wiki and keep trace of the history there as well.
>
> I would prefer not to turn on comments in this document. This will make it
difficult to manage this document. Turned on comments are also the ability
to add your own suggestions. History of comments is the second reason why I
did turn on it. Saving the discussion history in the comments will be very
hard. The official communication channel is a mailing list.


> Case #2 A, but I do not think it is backwards compatible as is claimed at
> the top of the document. People will have to update their DAGs.
>
> It is possible to move/rename classes and keep backward compatibility.
Proff:
https://github.com/apache/airflow/pull/4667
https://github.com/apache/airflow/blob/master/airflow/contrib/operators/gcs_to_gcs_transfer_operator.py
https://github.com/apache/airflow/blob/master/airflow/contrib/operators/gcp_transfer_operator.py
Do you have a specific case for which this is not possible?
I started working on this document when I found a way to preserve backwards
compatibility. A detailed description of my observations is included in the
mentioned PR.

Case #3 I agree about some form of submodules for large cloud providers. It
> sounds helpful. One question: where would I contribute an
> S3_to_GCS_operator in some of the new models?
>
>
"Operators that integrate with two services will not change."
They will be in the package with the remaining operators. The specific
place depends on the adoption of other solutions.


> Case #5 Are we talking about the class name or the file name? The class
> name is fine to me, but we could remove _operator from the file name.
>

Case #2 describes the change of file names
Case #5 describes the change of class names

I added examples to two cases to better describe the changes.

On Thu, Apr 18, 2019 at 6:58 AM Kamil Breguła <kamil.bregula@polidea.com>
> wrote:
>
> > Hello all,
> >
> > During the 4-year project development, the community made many
> > decisions about the structure and principles of module creation. Some
> > decisions have been made by Airbnb and they no longer apply. Other
> > recommendations were rejected because of the low recognition of the
> > community.
> >
> > I have created a document that collects several issues related to
> > import paths, because I see that this appears in the discussion many
> > times.
> >
> >
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> > However, it is not possible to solve this problem at the level of
> > individual contributions, but it is necessary to organize at the
> > project level
> >
> > I invite everyone to discuss the topic presented in this document.
> >
> > Greetings,
> >
> > --
> > Kamil Breguła
> > Polidea | Software Engineer
> >
> > M: +48 505 458 451
> > E: kamil.bregula@polidea.com
> >
> > We create human & business stories through technology.
> > Check out our projects!
> >
>


-- 

Kamil Breguła
Polidea <https://www.polidea.com/> | Software Engineer

M: +48 505 458 451 <+48505458451>
E: kamil.bregula@polidea.com
[image: Polidea] <https://www.polidea.com/>

We create human & business stories through technology.
Check out our projects! <https://www.polidea.com/our-work>
[image: Github] <https://github.com/Polidea> [image: Facebook]
<https://www.facebook.com/Polidea.Software> [image: Twitter]
<https://twitter.com/polidea> [image: Linkedin]
<https://www.linkedin.com/company/polidea> [image: Instagram]
<https://instagram.com/polidea> [image: Behance]
<https://www.behance.net/polidea>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message