airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arthur Wiedmer <arthur.wied...@gmail.com>
Subject Re: [2.0 spring cleaning] Changes in import paths
Date Thu, 18 Apr 2019 22:23:19 GMT
> > You can not put this document on the mailing list because the document
> contains advanced formatting.
>
> I will try to put this document in the wiki in the near future to the
> archival purpose. I would like to point out that this is not the final
> document and contains the final recommendations. I wanted to publish final
> recommendations as AIP on the wiki in the future. I treat this document as
> an intermediate step to develop the final document.
>
>
Fair enough. I do not think there is anything which could not have been
explained via plain text.


>
> > > Case #3 I agree about some form of submodules for large cloud
> providers.
> > It
> > > > sounds helpful. One question: where would I contribute an
> > > > S3_to_GCS_operator in some of the new models?
> > > >
> > > >
> > > "Operators that integrate with two services will not change."
> > > They will be in the package with the remaining operators. The specific
> > > place depends on the adoption of other solutions.
> > >
> >
> > I am mostly worried about dependencies. What if the single integration
> > breaks the multiple integration operators. We do not have this issue yet,
> > but it might crop up once we split them more evenly.
> >
> > I am not sure if I understood you correctly. Changes in imports should
> not
> cause pip dependency problems. The changes presented in the document are
> not intended to divide Airflow into many pip packages.
>

Apologies, I got confused after reading more about namespaces and thought
this is where we were going with this.
But you are right it is a different and somewhat orthogonal concern.
Please disregard my original comment.



>
> >
> > > > Case #5 Are we talking about the class name or the file name? The
> class
> > > > name is fine to me, but we could remove _operator from the file name.
> > > >
> > >
> > > Case #2 describes the change of file names
> > > Case #5 describes the change of class names
> > >
> > > I added examples to two cases to better describe the changes.
> > >
> >
> > Thank you. I guess I see it for file names, but I am wondering about the
> > operators and name collisions.
> > Say I need both the HiveOperator and I inline a custom operator for
> which I
> > need a hive hook.
> > # Would you recommmend the following? Or does case #5 only apply to
> > Operators?
> > from airflow.operators.hive import Hive as HiveOperator
> > from airfow.hooks.hive import Hive as HiveHook
> >
> > I was just thinking it might be nice to be able to import what I need
> > without renaming and not worry too much about names shadowing others.
> > Especially if I am new-ish to Apache Airflow.
> >
> >
> I think that the operator should describe the behavior(verb), hook should
> describe a service(noun). These are other parts of speech.
> In my opinion, HiveOperator should be named HiveExecuteQuery. Hook will be
> called Hive. Then there will be no name conflicts.
>

But we will still provide the backwards compatibility for a while with
aliases to the old names, correct?

I completely agree that the operator names have been a bit muddled,
HiveExecuteQuery sounds better to me.

Thanks again for taking the time, Kamil, I really appreciate it.

Best,
Arthur

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message