airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] kaxil commented on issue #6516: [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing
Date Thu, 09 Jan 2020 20:00:22 GMT
kaxil commented on issue #6516: [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing
URL: https://github.com/apache/airflow/pull/6516#issuecomment-572730254
 
 
   > Hey @gerardo @dimberman @kaxil @ashb @mik-laj @nuclearpinguin
   > 
   > I think you all might be interested with this change to see/review/understand how
kind implementation evolves and how Breeze will let people to work with Kubernetes Executor
in development mode. Kind seems to be really nice for both CI and local development.
   > 
   > I have finally implemented the "best" approach with Kind I think. One that is much
faster and seems to work really well for both - CI (much faster than previous approach with
separate docker container) - it also helps with local development with kind cluster . Now
I am testing all the ins/outs of it but it seems that it is really nice for local development
as well (with regards to resources/time pulling images/using latest sources etc.). Together
with the follow-up change of separating integration and slimming down Breeze: #7091 - it becomes
really useful for local development.
   > 
   > Some of the notable properties:
   > 
   > * I switched to the latest kind 6.1 (it does not require separate exporting of the
".kube folder - it is done automatically)
   > * I pass the docker socket from host to the container for "ENABLE_KIND_CLUSTER" build
- this way I can use the same docker image that airflow-testing is run on - all the Kind containers
are run in that docker image (and visible from the host via docker ps). This means a) no separate
docker container is needed b) no need to pull image from apache/airflow - it's already there!
I am checking if container is already created and skip creating it if so. You can also force
cluster recreation by passing --recreate-kind cluster flag or stop
   > * this also means that if I exit the Breeze/airflow-testing container - the cluster
is still running there and I do not recreate the cluster when I re-enter the container.
   > * However I still run "rebuild_airflow_image.sh" when entering the container. This
way I am 100% sure I use latest sources (both locally and in CI). Plus I can later change
the sources and run "rebuild_airflow_image.sh" without leaving the airflow-testing container!
What's nice about it - it's rather fast as it only rebuilds latest layers. It allows to iterate
semi-quickly while working on Kubernetes :). Still loading image to kind takes quite some
time (6 minutes for me) so it is not perfect. I will improve it when I have prod image - it
should be much faster to load rather than the CI one
   > * last but not least - the .kube folder is mounted from the HOST (${AIRFLOW_SOURCES}/.kube)
so you should be able to interact with the cluster from the host as well (You just need to
update hostname to localhost). I might want to document it or maybe even automatically copy
the configuration to include the configuration that will support kubectl/connecting to cluster-run
webserver from the localhost directly. All ports are exposed already so it should be rather
easy.
   
   Nice work @potiuk
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message