airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <>
Subject Re: Request to test the new CI test environment / CI Docker Image
Date Tue, 09 Jul 2019 17:36:22 GMT
Hey Stefan,

All the important problems should be fixed now! You can pull the latest
changes from my branch and retry:

   - The warning with /root/.docker/config.json is nothing to worry about
   (it tries to access your host configuration for credentials - but we do not
   need any credentials for the environment).
   - The CLEAN_FILES unbound variable was an interesting one - not sure why
   it was not set at the first run and set it on the other (likely because
   CLEAN_FILES is propagated through docker-compose settings and set to empty
   after initialisation) but I protected against such case now
   - Running flake/pylint is not supposed to be run from the container but
   from the host. I looked at the docs and realised it was not at all clear
   :). It was obvious for me but not for someone new :). That's big value of
   such tests. I updated with a note and also added extra
   protection. When you try those scripts from container you will get the
   following error now:

*You are inside the Airflow docker container!You should only run this
script from the host.Learn more about how we develop and test airflow in:*
<> *

Please try to run them from the host (and after pull you can try from the
container and you will see the error). It will run a bit slower now because
it checks whether docker should be rebuilt but soon I will optimise it away
with pre-commit hooks (I already have a POC working).

Anyone else wants to try - feel free and let me know!

On Tue, Jul 9, 2019 at 7:03 PM Jarek Potiuk <>

> Thanks Stefan! - will look into all those and fix :). Those are some of
> the small things that should be super easy to fix.
> The root user for Linux is is one of the differences Mac <> Linux that is
> super difficult to resolve (in current Docker) when you mount local
> volumes. The problem is that inside the container you will have the same
> uid/gid as your host user but you will not have that user inside the
> container (and then you will have no permissions to run airflow inside the
> container). Unfortunately mapping the users host <-> container is not easy
> (requires engine restart and cannot be done per-container). And we cannot
> assume fixed UID (different host users have different ids).
> Here is detailed explanation from our PR:
> This problem does not occur on MacOS because on Mac there is a (slower)
> osxfs filesystem used that maps users from host to linux VM -> and then to
> container. I am going to workaround it very soon for Linux as well.. In the
> follow-up "simplified development environment" AIP-7 those root-owned files
> will be cleaned up on entering the environment and you will have a simple
> command to clean them up. Maybe not a perfect solution but it is better
> than manual cleanup with sudo.
> J.
> On Tue, Jul 9, 2019 at 6:38 PM Stefan Seelmann <>
> wrote:
>> Hi Jarek,
>> tested quickly on ArchLinux. Overall looks very good, thanks for all the
>> effort!
>> When building the images for the first time I saw the following warning:
>> ```
>> WARNING: Error loading config file: /root/.docker/config.json: stat
>> /root/.docker/config.json: permission denied
>> ```
>> When running `./scripts/ci/` the first time
>> it fails:
>> ```
>> $ PYTHON_VERSION=3.6 BACKEND=postgres ENV=docker
>> ./scripts/ci/
>> /opt/airflow/scripts/ci/in_container/ line 57:
>> CLEAN_FILES: unbound variable
>> ```
>> I added `CLEAN_FILES=false` to the command then it worked. Next time the
>> additional env variable was not required.
>> Within the dev env I was able to run test using `run-tests`.
>> However running flake8 or pylint fails, seems the `hooks` folder is not
>> mounted into the docker container:
>> ```
>> /opt/airflow /opt/airflow
>> scripts/ci/ line 32: ./hooks/build: No such file or directory
>> ```
>> Another observation: Files and folders created by the environment have
>> user/group `root` (examples are `logs` and `__pycache__`). I assume the
>> reason is that the container runs as user root. Maybe it's possible to
>> run the container with uid/gid of the curren user using `-u $(id
>> -u):$(id -g)`?
>> Kind Regards,
>> Stefan
> --
> Jarek Potiuk
> Polidea <> | Principal Software Engineer
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <>


Jarek Potiuk
Polidea <> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message