airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Request to test the new CI test environment / CI Docker Image
Date Tue, 09 Jul 2019 17:03:26 GMT
Thanks Stefan! - will look into all those and fix :). Those are some of the
small things that should be super easy to fix.

The root user for Linux is is one of the differences Mac <> Linux that is
super difficult to resolve (in current Docker) when you mount local
volumes. The problem is that inside the container you will have the same
uid/gid as your host user but you will not have that user inside the
container (and then you will have no permissions to run airflow inside the
container). Unfortunately mapping the users host <-> container is not easy
(requires engine restart and cannot be done per-container). And we cannot
assume fixed UID (different host users have different ids).

Here is detailed explanation from our PR:
https://github.com/apache/airflow/pull/4938#issuecomment-507790924

This problem does not occur on MacOS because on Mac there is a (slower)
osxfs filesystem used that maps users from host to linux VM -> and then to
container. I am going to workaround it very soon for Linux as well.. In the
follow-up "simplified development environment" AIP-7 those root-owned files
will be cleaned up on entering the environment and you will have a simple
command to clean them up. Maybe not a perfect solution but it is better
than manual cleanup with sudo.

J.


On Tue, Jul 9, 2019 at 6:38 PM Stefan Seelmann <mail@stefan-seelmann.de>
wrote:

> Hi Jarek,
>
> tested quickly on ArchLinux. Overall looks very good, thanks for all the
> effort!
>
>
> When building the images for the first time I saw the following warning:
> ```
> WARNING: Error loading config file: /root/.docker/config.json: stat
> /root/.docker/config.json: permission denied
> ```
>
>
> When running `./scripts/ci/local_ci_enter_environment.sh` the first time
> it fails:
> ```
> $ PYTHON_VERSION=3.6 BACKEND=postgres ENV=docker
> ./scripts/ci/local_ci_enter_environment.sh
> /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh: line 57:
> CLEAN_FILES: unbound variable
> ```
> I added `CLEAN_FILES=false` to the command then it worked. Next time the
> additional env variable was not required.
>
>
> Within the dev env I was able to run test using `run-tests`.
>
>
> However running flake8 or pylint fails, seems the `hooks` folder is not
> mounted into the docker container:
> ```
> /opt/airflow /opt/airflow
> scripts/ci/ci_flake8.sh: line 32: ./hooks/build: No such file or directory
> ```
>
>
> Another observation: Files and folders created by the environment have
> user/group `root` (examples are `logs` and `__pycache__`). I assume the
> reason is that the container runs as user root. Maybe it's possible to
> run the container with uid/gid of the curren user using `-u $(id
> -u):$(id -g)`?
>
>
> Kind Regards,
> Stefan
>
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message