airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil Breguła <kamil.breg...@polidea.com>
Subject Re: AIP-8 Split Hooks/Operators into Separate Packages
Date Mon, 07 Jan 2019 23:14:57 GMT
I would like to mention that there is a 2 pull requests that introduces
plugin support installed by pip
https://github.com/apache/airflow/pull/4412
https://github.com/apache/airflow/pull/730
These PRs should be analyzed to meet all our requirements

It is worth mentioning that this change not only affects the plugin, but
also can affect the core.. Some changes can be developed independently of
the  core and tested in practice. After some time, however, they can become
part of the core if you notice the need. I observed a similar phenomenon
during the development of the Javascript programming language. Additional
parts of the language specification are developed (as plugins for the Babel
transpilator) independently, and after the standardization and transition
stage become part of the core specification. This way of developing
language specifications will make it a lot more stable and better reflect
the needs of programmers. All changes can start their existence at the
bottom of the software development cycle instead of being top-controlled by
the core.
An example of an idea that is difficult to test in the core:
https://github.com/apache/airflow/pull/4363#issuecomment-449889406

Regards,


On Mon, Jan 7, 2019 at 6:05 PM Tim Swast <swast@google.com.invalid> wrote:

> I've created AIP-8:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
>
> To follow-up from the discussion about splitting hooks/operators out of the
> core Airflow package at
>
> http://mail-archives.apache.org/mod_mbox/airflow-dev/201809.mbox/%3C308670DB-BD2A-4738-81B1-3F6FB312C0C8@apache.org%3E
>
> I propose packaging based on the target system, informed by the existing
> hooks in both core and contrib. This will allow those with the relevant
> expertise in each target system to respond to contributions / issues
> without having to follow the flood of everything Airflow-related. It will
> also decrease the surface area of the core package, helping with
> testability and long-term maintenance.
>
> *  •  **Tim Swast*
> *  •  *Software Friendliness Engineer
> *  •  *Google Cloud Developer Relations
> *  •  *Seattle, WA, USA
>


-- 

Kamil Breguła
Polidea <https://www.polidea.com/> | Software Engineer

M: +48 505 458 451 <+48505458451>
E: kamil.bregula@polidea.com
[image: Polidea] <https://www.polidea.com/>

We create human & business stories through technology.
Check out our projects! <https://www.polidea.com/our-work>
[image: Github] <https://github.com/Polidea> [image: Facebook]
<https://www.facebook.com/Polidea.Software> [image: Twitter]
<https://twitter.com/polidea> [image: Linkedin]
<https://www.linkedin.com/company/polidea> [image: Instagram]
<https://instagram.com/polidea> [image: Behance]
<https://www.behance.net/polidea>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message