airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: AIP-8 Split Hooks/Operators into Separate Packages
Date Mon, 07 Jan 2019 20:16:55 GMT
Something to think about is how data transfer operators like the
MysqlToHiveOperator usually rely on 2 hooks. With a package-specific
approach that may mean something like an `airflow-hive`, `airflow-mysql`
and `airflow-mysql-hive` packages, where the `airflow-mysql-hive` package
depends on the two other packages.

It's just a matter of having a clear strategy, good naming conventions and
a nice central place in the docs that centralize a list of approved
packages.

Max

On Mon, Jan 7, 2019 at 9:05 AM Tim Swast <swast@google.com.invalid> wrote:

> I've created AIP-8:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
>
> To follow-up from the discussion about splitting hooks/operators out of the
> core Airflow package at
>
> http://mail-archives.apache.org/mod_mbox/airflow-dev/201809.mbox/%3C308670DB-BD2A-4738-81B1-3F6FB312C0C8@apache.org%3E
>
> I propose packaging based on the target system, informed by the existing
> hooks in both core and contrib. This will allow those with the relevant
> expertise in each target system to respond to contributions / issues
> without having to follow the flood of everything Airflow-related. It will
> also decrease the surface area of the core package, helping with
> testability and long-term maintenance.
>
> *  •  **Tim Swast*
> *  •  *Software Friendliness Engineer
> *  •  *Google Cloud Developer Relations
> *  •  *Seattle, WA, USA
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message