airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe Gagnon <philgagn...@gmail.com>
Subject Re: Does anybody deploy DAGs in zip files?
Date Sat, 08 Jun 2019 18:13:40 GMT
We do, but it brings a host of issues so I would not be disinclined to
adjust our workflow if this functionality was removed. Our teams are very
decentralized so we keep every dag in a separate git repository. During our
deployment process (triggered by CI/CD) we zip everything up and put the
zip file in a S3 bucket. Our airflow components sync their dag folder with
this bucket. Having everything zipped up gives us some assurance that when
a DAG is updated, the whole thing is replaced as a single unit.

On Sat, Jun 8, 2019 at 1:28 PM Bas Harenslak <basharenslak@godatadriven.com>
wrote:

> General question: does anybody deploy their DAGs inside zip files?
>
> I understand this functionality exists for historical reasons, but I’ve
> never heard of anybody actually using it.
>
> If there’s nobody, I think the DAG parsing could be simplified by removing
> this functionality.
> If there is somebody using it, could you explain why you deploy zip files
> instead of “normal" files?
>
> Bas
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message