From dev-return-8855-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Tue Jul 9 17:36:41 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6F76C18062B for ; Tue, 9 Jul 2019 19:36:41 +0200 (CEST) Received: (qmail 7459 invoked by uid 500); 9 Jul 2019 17:36:40 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 7443 invoked by uid 99); 9 Jul 2019 17:36:39 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jul 2019 17:36:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 9F417180C8A for ; Tue, 9 Jul 2019 17:36:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.105 X-Spam-Level: *** X-Spam-Status: No, score=3.105 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, PDS_NO_HELO_DNS=1.294, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=polidea.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id KhrWBK6sjkD3 for ; Tue, 9 Jul 2019 17:36:36 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::d33; helo=mail-io1-xd33.google.com; envelope-from=jarek.potiuk@polidea.com; receiver= Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 54E6D7E209 for ; Tue, 9 Jul 2019 17:36:35 +0000 (UTC) Received: by mail-io1-xd33.google.com with SMTP id j6so44950611ioa.5 for ; Tue, 09 Jul 2019 10:36:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=polidea.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=C4RQlixILRNkqGLy/vOpozs2Zybm1LhAqBUMc7GOkXg=; b=uV8SdXRhbTgfJA+GuOkNlmucnIwgvztjTqB3zAgDTRUgcHR8JZ3CdmiReZtPJ0g4Jq N6hrgQ6RZ+CDCdPTf9hlGwHZKjqU2ndpYkk1cRGAOHPr9EXsxn4pWegfmE+v2D+GeNM1 HlJ449+MlkWb0RND3toGBpwPPRhCMWPzehb54= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=C4RQlixILRNkqGLy/vOpozs2Zybm1LhAqBUMc7GOkXg=; b=RJEqQokijgZiTe8/sIcSxJj8TPekATr1Xp7hfXAiRDl/qSsYdx19ipZqd3pyfqeiCf AuZqApDtSL+QiDbWz1Jg1PpHnfHnPNZiPrJwR83RYlTU2uxgwL6Sycmqr20zQW1rwKBy ylaJtL4PZ5Ru0OSdDHGIeKs3/EyXV32/u96s6jEFBHbkIz7MWSKWfWTwnkKO1pfTrVRG 8o/fttg/NGPxfV8WL5OxvReNvXtHFqatAdQrK384ByCYmk6eetvV1+NdPH5C+Uu7wIn9 x+niF2PS46GMeJopfWCq8AIRt+Qgl3fUaBwOBWcP047RYG03Vl1gXlbhJEYkVAWOylyU WdRA== X-Gm-Message-State: APjAAAXs283n3qkI95boztk/7wUHxlVOh15eNvcGr1jqHthTkGURYVpZ iERieA6AdLSu9Y/ocUVn05TTNvN/7uE3ZXg9WN/fQyaisOg= X-Google-Smtp-Source: APXvYqyc9HGI9Ac0BH6CfFjtjYORULV9lRhhwX/oxU+I0fi6Og84RwJHhOgAhy2iStcX5m/g3crXSuqvtlg8OOW9gtw= X-Received: by 2002:a5d:9d58:: with SMTP id k24mr26039166iok.116.1562693793629; Tue, 09 Jul 2019 10:36:33 -0700 (PDT) MIME-Version: 1.0 References: <9417be7a-cd16-692c-0f0c-1680751aa462@stefan-seelmann.de> In-Reply-To: From: Jarek Potiuk Date: Tue, 9 Jul 2019 19:36:22 +0200 Message-ID: Subject: Re: Request to test the new CI test environment / CI Docker Image To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="000000000000766d6b058d43003d" --000000000000766d6b058d43003d Content-Type: text/plain; charset="UTF-8" Hey Stefan, All the important problems should be fixed now! You can pull the latest changes from my branch and retry: - The warning with /root/.docker/config.json is nothing to worry about (it tries to access your host configuration for credentials - but we do not need any credentials for the environment). - The CLEAN_FILES unbound variable was an interesting one - not sure why it was not set at the first run and set it on the other (likely because CLEAN_FILES is propagated through docker-compose settings and set to empty after initialisation) but I protected against such case now - Running flake/pylint is not supposed to be run from the container but from the host. I looked at the docs and realised it was not at all clear :). It was obvious for me but not for someone new :). That's big value of such tests. I updated CONTRIBUTING.md with a note and also added extra protection. When you try those scripts from container you will get the following error now: *You are inside the Airflow docker container!You should only run this script from the host.Learn more about how we develop and test airflow in:* *https://github.com/apache/airflow/blob/master/CONTRIBUTING.md * Please try to run them from the host (and after pull you can try from the container and you will see the error). It will run a bit slower now because it checks whether docker should be rebuilt but soon I will optimise it away with pre-commit hooks (I already have a POC working). Anyone else wants to try - feel free and let me know! J. On Tue, Jul 9, 2019 at 7:03 PM Jarek Potiuk wrote: > Thanks Stefan! - will look into all those and fix :). Those are some of > the small things that should be super easy to fix. > > The root user for Linux is is one of the differences Mac <> Linux that is > super difficult to resolve (in current Docker) when you mount local > volumes. The problem is that inside the container you will have the same > uid/gid as your host user but you will not have that user inside the > container (and then you will have no permissions to run airflow inside the > container). Unfortunately mapping the users host <-> container is not easy > (requires engine restart and cannot be done per-container). And we cannot > assume fixed UID (different host users have different ids). > > Here is detailed explanation from our PR: > https://github.com/apache/airflow/pull/4938#issuecomment-507790924 > > This problem does not occur on MacOS because on Mac there is a (slower) > osxfs filesystem used that maps users from host to linux VM -> and then to > container. I am going to workaround it very soon for Linux as well.. In the > follow-up "simplified development environment" AIP-7 those root-owned files > will be cleaned up on entering the environment and you will have a simple > command to clean them up. Maybe not a perfect solution but it is better > than manual cleanup with sudo. > > J. > > > On Tue, Jul 9, 2019 at 6:38 PM Stefan Seelmann > wrote: > >> Hi Jarek, >> >> tested quickly on ArchLinux. Overall looks very good, thanks for all the >> effort! >> >> >> When building the images for the first time I saw the following warning: >> ``` >> WARNING: Error loading config file: /root/.docker/config.json: stat >> /root/.docker/config.json: permission denied >> ``` >> >> >> When running `./scripts/ci/local_ci_enter_environment.sh` the first time >> it fails: >> ``` >> $ PYTHON_VERSION=3.6 BACKEND=postgres ENV=docker >> ./scripts/ci/local_ci_enter_environment.sh >> /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh: line 57: >> CLEAN_FILES: unbound variable >> ``` >> I added `CLEAN_FILES=false` to the command then it worked. Next time the >> additional env variable was not required. >> >> >> Within the dev env I was able to run test using `run-tests`. >> >> >> However running flake8 or pylint fails, seems the `hooks` folder is not >> mounted into the docker container: >> ``` >> /opt/airflow /opt/airflow >> scripts/ci/ci_flake8.sh: line 32: ./hooks/build: No such file or directory >> ``` >> >> >> Another observation: Files and folders created by the environment have >> user/group `root` (examples are `logs` and `__pycache__`). I assume the >> reason is that the container runs as user root. Maybe it's possible to >> run the container with uid/gid of the curren user using `-u $(id >> -u):$(id -g)`? >> >> >> Kind Regards, >> Stefan >> >> >> > > -- > > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] > > -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] --000000000000766d6b058d43003d--