airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Palmer <ch...@crpalmer.com>
Subject Re: AIP-21 (Move operators to Core) - "cross_transfer" packages
Date Mon, 23 Sep 2019 17:42:33 GMT
On Mon, Sep 23, 2019 at 1:22 PM Kamil BreguĊ‚a <kamil.bregula@polidea.com>
wrote:

> On Mon, Sep 23, 2019 at 7:04 PM Chris Palmer <chris@crpalmer.com> wrote:
> >
> > Is there a reason why we can't use symlinks to have copies of the files
> > show up in both subpackages? So that `gcs_to_s3.py` would be under both
> > `aws/operators/` and `gcp/operators`. I could imagine there may be
> > technical reasons why this is a bad idea, but just thought I would ask.
> >
> Symlinks is not supported by git.
>
>
Why do you say that? This blog post
<https://www.mokacoding.com/blog/symliks-in-git/> details how you can use
them, and the caveats with regards to needing relative links not absolute.
The example repo he links to at the end includes a symlink which worked
fine for me when I cloned it. But maybe not relevant given the below:



> > Likewise, someone who spends 99% of their time working in AWS and using
> all
> > the operators in that subpackage, might not think to look in the GCP
> > package the first time they need a GCS to S3 operator. I'm admittedly
> > terrible at documentation, but if duplicating the files via symlinks
> isn't
> > an option, then is there an easy way we could duplicate the documentation
> > for those operators so they are easily findable in both doc sections?
> >
>
> Recently, I updated the documentation:
> https://airflow.readthedocs.io/en/latest/integration.html
> We have list of all integration in AWS, Azure, GCP.  If the operator
> concerns two cloud proivders, it repeats in two places. It's good for
> documentation.  DRY rule is only valid for source code.
> I am working on documentation for other operators.
> My work is part of this ticket:
> https://issues.apache.org/jira/browse/AIRFLOW-5431
>
>
This updated documentation looks great, definitely heading in a direction
that makes it easier and addresses my concerns. (Although it took me a
while to realize those tables can be scrolled horizontally!).

Chris

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message