airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Use Docker for running Airflow tests
Date Sun, 19 Aug 2018 18:43:20 GMT
Hi all,

Gerardo is doing some awesome work on Dockerizing the CI pipeline. The PR
is still open here: https://github.com/apache/incubator-airflow/pull/3393

To make this work, and not store any Docker images in private repositories,
I've created a new repository which stores the Docker image which contains
a lot of the dependencies we use to test against. The repository is here:
https://github.com/apache/incubator-airflow-ci
For most of the tests, we use mocks, but sometimes we also use the actual
service, for example for some Hadoop, Kerberos, Hive functionality. The
images are build in the Dockerhub service:
https://hub.docker.com/r/airflowci/incubator-airflow-ci/builds/
I've created it for now. I'm open for another name, but Airflow was already
taken, and hyphens are not allowed.

This change will greatly decouple our dependency on Travis and will make it
much easier to test the code locally. Instead of setting up tox
environments, we can just use Docker instead. I'm really enthusiastic about
the change, and really thankful for Gerardo to put all this effort in
there.

Recently Holden Karau did a lifestream
<https://www.youtube.com/watch?v=5Il3w2P7IcM>, and did her first PR. I
think the most important learning from it was that testing was not as
trivial as it should be. Therefore I think we should see if we can merge
the PR from Gerardo soon.

Cheers, Fokko

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message