From dev-return-8966-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Wed Jul 24 08:05:16 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 463E618025F for ; Wed, 24 Jul 2019 10:05:16 +0200 (CEST) Received: (qmail 88476 invoked by uid 500); 24 Jul 2019 08:05:13 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 88457 invoked by uid 99); 24 Jul 2019 08:05:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Jul 2019 08:05:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 7E3461803B3 for ; Wed, 24 Jul 2019 08:05:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.811 X-Spam-Level: * X-Spam-Status: No, score=1.811 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=polidea.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 64fyFDqlGXS8 for ; Wed, 24 Jul 2019 08:05:09 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.166.44; helo=mail-io1-f44.google.com; envelope-from=jarek.potiuk@polidea.com; receiver= Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 7C866BC781 for ; Wed, 24 Jul 2019 08:05:09 +0000 (UTC) Received: by mail-io1-f44.google.com with SMTP id z3so87722111iog.0 for ; Wed, 24 Jul 2019 01:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=polidea.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=XIaN7m5yZrZvTTfqUyAJ45lBCKaVwihJjMgMU30X5H0=; b=lVbAdo1oIdaifLOWUrrtj2546LrQgvalmnycjsYgxzrvphLOi4A0+RIRYVKkf2tzU+ uvNl4H7plIKSaQvh+REnZGZoll3ko3frajvPIK2omcWNZ3AHp6RwHIbAix+5HajJP9av M0xs7ZQpRsjtopL1Oe8ExS1qH4Cf7hDCdA2PU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=XIaN7m5yZrZvTTfqUyAJ45lBCKaVwihJjMgMU30X5H0=; b=c8t6id+iJO5Q1sOB+ODIZXTIN0tj/ujrJb6e0QokIpwiu54f7prtwAOquB/dOTUFXA I6HfRFFLr11N6m1qlQ6BmLCUMuBlF3YHxSkzdTTDojqRMDtYCxpFU/oJo1yMhDBnGc7u tf9eSPKlybzKejKNZIrHz0GG0wmBuv4mUMR2DAgTHEyfPhIQpJG/7oY5kSs1vmdOEmdf A/q8ZnupsA87T92I8syKmQ3GKkRlZuVxL5yIpFNEllielKI34jssZjSLfmDI4dHLS+9F b5V9njL4nQWXmTCetmO/dEHpXyL3+05669iOBjuauAyoRJ73cYwkCsv/C7Web1aoiv/A m/YA== X-Gm-Message-State: APjAAAUy5nqyZSAjhj1Lc1ESzgudY0eh2UcP0z63muAB90F4o5gRcp9I CZ2fVMotEc1yjo1RHwll3Cxx3KrozaR4WgAq5/UnXg== X-Google-Smtp-Source: APXvYqw1STz5wKU/FhQ6ki3s0wL+ZznBSSwxXsGNm95qIwjQepsbuiBqu3Fnpz1sYAfGM7eEqWQCbUwS9dExzOp7k+s= X-Received: by 2002:a5e:9917:: with SMTP id t23mr5688553ioj.23.1563955502683; Wed, 24 Jul 2019 01:05:02 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jarek Potiuk Date: Wed, 24 Jul 2019 10:04:34 +0200 Message-ID: Subject: Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI" To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="0000000000002eafb2058e68c434" --0000000000002eafb2058e68c434 Content-Type: text/plain; charset="UTF-8" Of course ! One of the considerations is to keep travis CI build intact - so that anyone will be able to have their own Travis Fork for the time being. I will also do it in the way that once you have your own GCP account and your own GKE cluster, you will be able to replicate it as well (there will be instructions on how to set it up). We can even (long term) make it in the way that you will not need a separate GKE cluster but it will run using just your personal GitLab (free). This should be possible - I am really trying to make it underlying-infrastructure-agnostic. The non-cluster personal GitLab is not a priority now (Travis forks will hopefully work ;) so it might not work initially, but there aren't fundamental reasons it should not work. We will have to just use GitLabCI registry instead of the GCP one and avoid assuming we are running in the GKE cluster and have some secrets/accounts distributed differently. All looks doable. J. J. On Wed, Jul 24, 2019 at 9:03 AM Chao-Han Tsai wrote: > Thanks Jarek for putting this together. We really need a stable and fast > CI. > > Question: will we still be able to build our personal fork of Airflow on > our own Travis? > > Chao-Han > > On Tue, Jul 23, 2019 at 1:00 PM Jarek Potiuk > wrote: > > > > > > > Question - what is the purpose of introducing kaniko instead of using > > > regular docker build? > > > > > > > Indeed. We want to be as agnostic as possible. What I plan to do is to > use > > Kubernetes Runner in GitlabCI. This means that all the jobs will run as > > Kubernetes PODs in GKE - Gitlab CI will only be UI + runner that > > orchestrates the builds. This means that our test jobs will be run inside > > docker - they will not run in virtual machine, but they will run inside > the > > container. This is how modern CI systems work (for example Gitlab, > > CloudBuild, also Argo - new kid in the > block > > which is Kubernetes-Native). Argo is a bit too fresh to consider it, but > > they all work similarly - all steps are run inside docker. > > > > As the first part of our build we have to build the images with latest > > sources (and dependencies if needed) that then will be used for > subsequent > > steps. This means that we need to build the images from within docker - > > which is not as trivial as running docker command. There are three ways > to > > approach it - docker-in-docker (requires priviledged docker containers), > > using same docker engine which is used by Kubernetes Cluster (not > > recommended as Kubernetes manages docker engine on their own and might > > delete/remove images at any time) or use Kaniko. Kaniko was created > exactly > > for this purpose - to be able to run docker build from within a POD that > > runs in Kubernetes cluster. > > > > I hope it explains :). Kaniko is pretty much standard way of doing it and > > it really Kubernetes-native way of doing it. > > > > > > > > > > Regards > > > Shah > > > > > > > > > > > > > > > On Tue, Jul 23, 2019 at 5:12 PM Jarek Potiuk > > > > wrote: > > > > > > > Hello Everyone, > > > > > > > > I prepared a short docs where I described general architecture of the > > > > solution I imagine we can deploy fairly quickly - having GitLab CI > > > support > > > > and Google provided funding for GCP resources. > > > > > > > > I am going to start working on Proof-Of-Concept soon but before I > start > > > > doing it, I would like to get some comments and opinions on the > > proposed > > > > approach. I discussed the basic approach with my friend Kamil who > works > > > at > > > > GitLab and he is a CI maintainer and this is what we think will be > > > > achievable in fairly short time. > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI > > > > > > > > I am happy to discuss details and make changes to the proposal - we > can > > > > discuss it here or as comments in the document. > > > > > > > > Let's see what people think about it and if we get to some consensus > we > > > > might want to cast a vote (or maybe go via lasy consensus as this is > > > > something we should have rather quickly) > > > > > > > > Looking forward to your comments! > > > > > > > > J. > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] > > > > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] > > > > > -- > > Chao-Han Tsai > -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] --0000000000002eafb2058e68c434--