airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arthur Wiedmer <arthur.wied...@gmail.com>
Subject Re: [2.0 spring cleaning] Changes in import paths
Date Thu, 18 Apr 2019 17:29:32 GMT
Hi Kamil,

Thank you for this doc, it was a lot of good work to put all of this
together. Could you open the doc for comment?
May I recommend that once comments have been added to eventually port the
doc to the wiki and keep trace of the history there as well.

Here are my choices
Case #1 B or A with some mechanism to "certify" the level of care on an
operator (code coverage, whatever). I think that when people come to
airflow, they need to know which parts are battle tested versus which parts
are more beta status.

Case #2 A, but I do not think it is backwards compatible as is claimed at
the top of the document. People will have to update their DAGs.

Case #3 I agree about some form of submodules for large cloud providers. It
sounds helpful. One question: where would I contribute an
S3_to_GCS_operator in some of the new models?

Case #4 A sounds good, but the transition will need to be done carefully.
We have had a hard time splitting up models.py, there will be some
interesting pieces there as well. Finally, same question with my
S3_to_GCS_operator, where does it fit?

Case #5 Are we talking about the class name or the file name? The class
name is fine to me, but we could remove _operator from the file name.

Case #6 Sure, those look fine to me

Case #7 yes to the python native solution. The fewer the number of external
dependencies, the best.



Another minor point that I would like to correct, the airflow vs
airflow/contrib structure was always meant to be based on the amount of
testing that the operators had received. It was the case when Airbnb open
sourced it. Some Airbnb contributed operators are in airflow/contrib, and
some externally contributed operators are in the main airflow folder.

The main reason why in the end, operators and hooks did not move was
backwards compatibility. We have not had a major version release since the
project has been open sourced, as such we were trying (and sometimes
inadvertently failing) not to break backwards compatibility.

Best,
Arthur

On Thu, Apr 18, 2019 at 6:58 AM Kamil Breguła <kamil.bregula@polidea.com>
wrote:

> Hello all,
>
> During the 4-year project development, the community made many
> decisions about the structure and principles of module creation. Some
> decisions have been made by Airbnb and they no longer apply. Other
> recommendations were rejected because of the low recognition of the
> community.
>
> I have created a document that collects several issues related to
> import paths, because I see that this appears in the discussion many
> times.
>
> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#
> However, it is not possible to solve this problem at the level of
> individual contributions, but it is necessary to organize at the
> project level
>
> I invite everyone to discuss the topic presented in this document.
>
> Greetings,
>
> --
> Kamil Breguła
> Polidea | Software Engineer
>
> M: +48 505 458 451
> E: kamil.bregula@polidea.com
>
> We create human & business stories through technology.
> Check out our projects!
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message