airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <>
Subject Mutli-layered official image for Airflow
Date Wed, 16 Jan 2019 21:59:55 GMT
Hello Everyone,

Following the discussion we had on Mono-layered vs. Multi-layered official
image for Airflow here, I
prepared a proof-of-concept PR of multi-layered image (based on the
mono-layered one) and I performed calculations and reached some conclusions
in this proposal (I wanted to have some hard numbers to back the statement
that multi-layered Docker file is better) :

The conclusions I reached:

   - The multi-layered image is even slightly smaller than the mono-layered
   one - so multi-layered image is even better when you download it once
   - Downloading the image regularly by the users is way better in case of
   multi-layered image - for simulated user, downloading airflow image twice a
   week it is:  5.7 GB  (multi-layered) vs. 16.15 GB (mono-layered) downloads
   over the course of 8 weeks.\
   - Multi-layered image is better choice.

I based those calculations on the PR I prepared: where I implemented rather nice
multi-layered Dockerfile that can be easily maintained.

It's  based on my experience with Airflow Breeze
<> - the GCP Development
environment we used to develop 30+ GCP based operators recently.

I hope we can reach the conclusion as the community that multi-layered is
better and that we can go in this direction :). I am happy to iterate on my
PR to make it even better.



Jarek Potiuk
Polidea <> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message