airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Davydov <ddavy...@twitter.com.INVALID>
Subject Re: Does anybody deploy DAGs in zip files?
Date Mon, 10 Jun 2019 15:34:01 GMT
I know the code around this is pretty hacky (if use_zip_file then...
instead of an abstraction). I know when it was added it was a bit
controversial, I would be +1 on removing it. That being said I feel the
entire DAG parsing process needs to be moved to the client-side (users who
write DAGs), with a serialization scheme. I'm working on a proposal for
this at the moment, and this would obviate even reading DAGs from the local
filesystem.

Having everything zipped up gives us some assurance that when a DAG is
> updated, the whole thing is replaced as a single unit.

I don't believe unzipping files gives this atomicity guarantee any more
than something like rsync --delete.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message