airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <>
Subject Re: Cloud Provider grouping into Plugins
Date Fri, 14 Oct 2016 15:24:51 GMT
Ya, if we go the deprecation route, and let them float around for a
release or two, I'm OK with that (or until we bump major to 2.0).

Other than that, it sounds like a good opportunity to clean things up.
:) I do notice a lot of AWS/GCP code (e.g. the S3 Redshift operator).

On Fri, Oct 14, 2016 at 8:16 AM, Alex Van Boxel <> wrote:
> Well, I wouldn't touch the on that exist (maybe we could mark them
> deprecated, but that's all). But I would move (copy) them together and make
> them consistent (example, let them all use the same default connection_id,
> ...). For a new user it's quite confusing I think due to different reasons
> (style, etc...) you know we have an old ticket: making gcp consistent (I
> just don't want to start on this on, on fear of breaking something).
> On Fri, Oct 14, 2016 at 4:59 PM Chris Riccomini <>
> wrote:
> Hmm. What advantages would this provide? I'm a little nervous about
> breaking compatibility. We have a bunch of DAGs which import all kinds
> of GCP hooks and operators. Wouldn't want those to move.
> On Fri, Oct 14, 2016 at 7:54 AM, Alex Van Boxel <> wrote:
>> Hi all,
>> I'm starting to write some very exotic Operators that are a bit strange
>> adding to contrib. Examples of this are:
>> + See if a Compute snapshot of a disc is created
>> + See if a string appears on the serial port of Compute instance
>> but they would be a nice addition if we had a Google Compute plugin (or
> any
>> other cloud provider, AWS, Azure, ...). I'm not talking about getting
> cloud
>> support out of the main source tree. No, I'm talking about grouping them
>> together in a consistent part. We can even start adding macro's etc. This
>> would be a good opportunity to move all the GCP operators together, making
>> them consistent without braking the existing operators that exist in
>> *contrib*.
>> Here are a few requirements that I think of:
>>    - separate folder ( example  <airflow>/integration/googlecloud ,
>> <airflow>/integration/aws
>>    ,  <airflow>/integration/azure )
>>    - enable in config (don't want to load integrations I don't use)
>>    - based on Plugin (same interface)
>> Thoughts?

View raw message