airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Swast <sw...@google.com.INVALID>
Subject Re: AIP-8 Split Hooks/Operators into Separate Packages
Date Mon, 07 Jan 2019 23:44:38 GMT
In general it’s easier for cloud providers to fund development of operators
that bring data in. I’d say if there is overlap, put the operator in the
target system’s repo.

On Mon, Jan 7, 2019 at 2:17 PM Maxime Beauchemin <maximebeauchemin@gmail.com>
wrote:

> Something to think about is how data transfer operators like the
> MysqlToHiveOperator usually rely on 2 hooks. With a package-specific
> approach that may mean something like an `airflow-hive`, `airflow-mysql`
> and `airflow-mysql-hive` packages, where the `airflow-mysql-hive` package
> depends on the two other packages.
>
> It's just a matter of having a clear strategy, good naming conventions and
> a nice central place in the docs that centralize a list of approved
> packages.
>
> Max
>
> On Mon, Jan 7, 2019 at 9:05 AM Tim Swast <swast@google.com.invalid> wrote:
>
> > I've created AIP-8:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
> >
> > To follow-up from the discussion about splitting hooks/operators out of
> the
> > core Airflow package at
> >
> >
> http://mail-archives.apache.org/mod_mbox/airflow-dev/201809.mbox/%3C308670DB-BD2A-4738-81B1-3F6FB312C0C8@apache.org%3E
> >
> > I propose packaging based on the target system, informed by the existing
> > hooks in both core and contrib. This will allow those with the relevant
> > expertise in each target system to respond to contributions / issues
> > without having to follow the flood of everything Airflow-related. It will
> > also decrease the surface area of the core package, helping with
> > testability and long-term maintenance.
> >
> > *  •  **Tim Swast*
> > *  •  *Software Friendliness Engineer
> > *  •  *Google Cloud Developer Relations
> > *  •  *Seattle, WA, USA
> >
>
-- 
*  •  **Tim Swast*
*  •  *Software Friendliness Engineer
*  •  *Google Cloud Developer Relations
*  •  *Seattle, WA, USA

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message