From dev-return-8798-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Fri Jun 28 12:21:54 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1A43018062B for ; Fri, 28 Jun 2019 14:21:54 +0200 (CEST) Received: (qmail 4526 invoked by uid 500); 28 Jun 2019 12:21:52 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 4501 invoked by uid 99); 28 Jun 2019 12:21:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jun 2019 12:21:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6BA8EC214A for ; Fri, 28 Jun 2019 12:21:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.253 X-Spam-Level: ** X-Spam-Status: No, score=2.253 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=driesprong-frl.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id yrGaHCdhpogD for ; Fri, 28 Jun 2019 12:21:45 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::b36; helo=mail-yb1-xb36.google.com; envelope-from=fokko@driesprongen.nl; receiver= Received: from mail-yb1-xb36.google.com (mail-yb1-xb36.google.com [IPv6:2607:f8b0:4864:20::b36]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id F3558BC95A for ; Fri, 28 Jun 2019 12:21:44 +0000 (UTC) Received: by mail-yb1-xb36.google.com with SMTP id p2so3629793ybl.13 for ; Fri, 28 Jun 2019 05:21:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=driesprong-frl.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fdfqVvn2Rfnu6s57IRy0axMjsEmsktWKhcJs3rSen8Q=; b=VlXWORP//NDnLvUkYxfoPqDBlztpUuo66xQvJ5XUmXcwvVQXzEEWlAxMTUxbhHYR04 FE398CZIghvaErbgwgn9ipSlS6OF5eZE3dKWCjmAZdT5FipVs6jG3aKhUK6ItslzvS24 pDCW7NVp9VttyZhFts83YaBxTUjfDikvH8dZaKpWltrbYIy73B/Tx0qa7JJqptxrJX3p UtiMH5FVGUkO5UF8P/HZgAis7I/A7h25F/vZvrT0JD/wMFNb3IoIrT/jIk3jXOoIpEz1 xQTaPeJmXlGaswIEYtCe6lS/fwVw9l2j4RupIaKk4IXheHsRvXG7qe2/tj86RsnjFOrh 0EsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fdfqVvn2Rfnu6s57IRy0axMjsEmsktWKhcJs3rSen8Q=; b=UcJJaXQqI9WjiYb+4iH2sFRBquEuRcT8gc6YpzMgJ0TLkJzBaCx6BZVx1CKYTFNL5H 2rvin0XPbdKe3BOZ2j4VBEb9NNexX+EI/me/K1N8xCGbq3mWq2PhOUB0VlmqL7K2DIuY Hh7apdKV8udQE9mVCYkx+0lmY7JvL9cSmPo3DZaQENKJKXoBAerS/p7SyHdPWssV20Xr z17gbEM7OIZCtwrVhcdtWpe0VhFfYrz54WK/3tkiQSjErSd51UxT0ropma67g6CqQUbz XRcbuve+eJUX7tiKMh3ykw8eVrx8Ggimo7D6neGmACr9Gl2TA76dw+wrwpCgBR4tHqJ/ zTQQ== X-Gm-Message-State: APjAAAXq1/TCvl5Kj31aCy4N3R1MHiRGfx4/k6rplOuCjT/yrgArwO3J 2xU/H8+Ar3Hi+1sqVcbpIMFpyKSxsuMizAIRvHzve7Yr X-Google-Smtp-Source: APXvYqy+GyRDiJ2FzEasexguNKtVA1R3TH4g3ausK8LWspTy6Kgexb1tiOcSKLz1uWxa1qKRVjx+R/y1/pr+hlx8+Dk= X-Received: by 2002:a5b:acd:: with SMTP id a13mr5761339ybr.64.1561724497862; Fri, 28 Jun 2019 05:21:37 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: "Driesprong, Fokko" Date: Fri, 28 Jun 2019 14:21:26 +0200 Message-ID: Subject: Re: Travis builds in a queue for hours To: dev@airflow.apache.org Cc: Ash Berlin-Taylor Content-Type: multipart/alternative; boundary="000000000000eeba03058c615177" --000000000000eeba03058c615177 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yes Conrad, this is actually true. However I think it is important to test different backends, as an example, we had a lot of issues with MySQL's timezone behavior, besides that, it would be possible to test specific database evolution steps for certain backends. Cheers, Fokko Op vr 28 jun. 2019 om 14:11 schreef Conrad Dean : > Regarding the matrix of backends to test -- is this testing specific > platform integrations, or if everything is routed through SQLAlchemy's OR= M > does this test suite just end up testing SQLAlchemy? > > It makes sense to have the ability to swap backends out for the sake of > reproducing issues when something comes up, but exhaustively providing > coverage for a library that Airflow uses feels a bit out-of-place. This > would only make sense if the project has its own extensions or plugins on > top of SQLAlchemy, or uses its own forked version of it. > > If the test suite was limited to one backend for CI it would reduce the > build times by quite a bit. > > On Thu, Jun 27, 2019 at 9:28 AM Jarek Potiuk > wrote: > > > Yeah. I also have a working version of Cloud build configuration and we > can > > run the tests on cloud build if we can get some credits from Google. An= d > > the changes from the upcoming CI image will make it much easier to run > > tests on any CI provider. Except Kubernetes tests they are pretty much > > CI-agnostic. Kubernetes tests will likely be also fixed soon. > > > > Another idea: I thought that in the future we can also run only subset = of > > postgres/mysql/sqlite tests on all combinations. I think there are just > > handful of tests that are specific for backend (and we already know whi= ch > > ones they are - they are skipped-if). > > > > J. > > > > Principal Software Engineer > > Phone: +48660796129 > > > > czw., 27 cze 2019, 15:12 u=C5=BCytkownik Philippe Gagnon < > philgagnon1@gmail.com > > > > > napisa=C5=82: > > > > > I think the combinations that you are proposing are sensible for > > pre-merge > > > checks. > > > > > > I am working on a proposal to offload extra combinations to another C= I > > > provider (Azure DevOps specifically seems like a good candidate), > either > > > pre or post merge. Ideally I'd like to run more combinations pre-merg= e > > but > > > there is a trade-off to be conscious of here between development > velocity > > > and quality assurance, which I think this issue highlights quite well= . > > > > > > Please let me know your thoughts > > > > > > Philippe > > > > > > On Thu, Jun 27, 2019 at 9:05 AM Jarek Potiuk > > > > wrote: > > > > > > > Agree that we should be thoughtful about others as well: In the > latest > > > push > > > > (few minutes ago) of the upcoming official CI image i implemented t= he > > > > change we discussed in the Github where we limit the number of > > > combinations > > > > we test: > > > > > > > > You can see it yourself: > > > > https://travis-ci.org/apache/airflow/builds/551305240 > > > > > > > > Those are the combinations I propose: > > > > > > > > Python: 3.6 > > > > BACKEND=3Dmysql ENV=3Ddocker > > > > > > > > Python: 3.6 > > > > BACKEND=3Dpostgres ENV=3Ddocker > > > > > > > > Python: 3.5 > > > > BACKEND=3Dsqlite ENV=3Ddocker > > > > > > > > Python: 3.6 > > > > BACKEND=3Dpostgres ENV=3Dkubernetes KUBERNETES_VERSION=3Dv1.13.0 > > > > > > > > J, > > > > > > > > > > > > On Thu, Jun 27, 2019 at 11:00 AM Driesprong, Fokko > > > > > > > > > wrote: > > > > > > > > > We got this message last year: > > > > > > > > > > > Hello, Airflow PPMC. > > > > > > While going through the usage statistics for our Travis CI > > service, I > > > > > > have noticed that the Airflow project is using an abnormally > large > > > > > > amount of resources, 2600 hours per month or the equivalent of > > having > > > > > > almost 4 machines building airflow non-stop 24/7. As this is no= t > > > free, > > > > > > but rather costing us money, I'm contacting you with the > intention > > of > > > > > > figuring out ways to reduce the use of Travis for the project. > > > > > > > > > > > We would greatly prefer that the project itself comes up with a > > > > solution > > > > > > to lower the usage of Travis, as we'd hate to simply turn it of= f > > for > > > > > > you, but the usage is at a rather severe level, totaling more > than > > > 21% > > > > > > of the total build time of all projects using Travis, so > something > > > > > > actionable should be decided upon and (preferably) completed by > the > > > end > > > > > > of May that will reduce the consumption of Travis resources. > > > > > > > > > > > Alternately, if you are unable to lower the pressure on Travis, > the > > > > > > podling and/or IPMC may ask the board of directors for a separa= te > > > > budget > > > > > > for additional build nodes to cope with the added load - I'll > leave > > > > this > > > > > > for the podling and IPMC to decide on. > > > > > > > > > > > Please let us know when you have decided on a plan to remedy th= is > > > > > situation. > > > > > > > > > > > With regards, > > > > > > Daniel on behalf of ASF Infrastructure. > > > > > > > > > > I think more and more projects are still migrating to the ASF > Travis, > > > so > > > > I > > > > > think natural that there is more load. However, this still leaves > the > > > > > question if we have to run the full matrix. > > > > > > > > > > Cheers, Fokko > > > > > > > > > > > > > > > > > > > > Op do 27 jun. 2019 om 10:56 schreef Jarek Potiuk < > > > > Jarek.Potiuk@polidea.com > > > > > >: > > > > > > > > > > > I think we should really involve infra to increase the slot > number > > or > > > > > maybe > > > > > > even somehow allocate slots per project. > > > > > > The problem is that we cannot control what other apache project= s > > are > > > > > doing, > > > > > > so even if we decrease our runtime, it's the other projects tha= t > > > might > > > > > hold > > > > > > us in the queue :( > > > > > > > > > > > > J. > > > > > > > > > > > > On Thu, Jun 27, 2019 at 10:19 AM Driesprong, Fokko > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I've noticed this at other Apache projects as well, sometimes > it > > > > takes > > > > > up > > > > > > > to 7-8 hours. The only thing we can do, is reduce the runtime > of > > > the > > > > > jobs > > > > > > > so we take less slots :-) > > > > > > > > > > > > > > Cheers, Fokko > > > > > > > > > > > > > > Op wo 26 jun. 2019 om 21:59 schreef Jarek Potiuk < > > > > > > Jarek.Potiuk@polidea.com > > > > > > > >: > > > > > > > > > > > > > > > Yep. That's what I suggested as the reason in the ticket - = I > > > guess > > > > > > INFRA > > > > > > > > are the only people who can do anything about it (increase > > > > > concurrency > > > > > > ? > > > > > > > > pay more for Travis :)? ). > > > > > > > > > > > > > > > > On Wed, Jun 26, 2019 at 9:51 PM Ash Berlin-Taylor < > > > ash@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > I asked Travis on twitter and they said it was due to the > > > Apache > > > > > > other > > > > > > > > > projects build queues > > > > > > > > > > > > > > > > > > https://twitter.com/travisci/status/1143893051460526080 > > > > > > > > > > > > > > > > > > -ash > > > > > > > > > > > > > > > > > > On 26 June 2019 20:48:33 BST, Jarek Potiuk < > > > > > Jarek.Potiuk@polidea.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> Hello everyone, > > > > > > > > >> > > > > > > > > >> For the last few days the Travis builds for apache/airfl= ow > > > > project > > > > > > are > > > > > > > > >> waiting in a queue for hours. This is not a normal > > situation. > > > > I've > > > > > > > > opened > > > > > > > > >> INFRA ticket for that: > > > > > > > > https://issues.apache.org/jira/browse/INFRA-18657 > > > > > > > > >> > > > > > > > > >> J. > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Jarek Potiuk > > > > > > > > Polidea | Principal Software > > Engineer > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > [image: Polidea] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Jarek Potiuk > > > > > > Polidea | Principal Software Enginee= r > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] > > > > > > > > > > --000000000000eeba03058c615177--