From dev-return-80233-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Mon Jul 20 17:49:52 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 95DB4180647 for ; Mon, 20 Jul 2020 19:49:52 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 2EC76124B82 for ; Mon, 20 Jul 2020 17:49:49 +0000 (UTC) Received: (qmail 16495 invoked by uid 500); 20 Jul 2020 17:49:47 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 16455 invoked by uid 99); 20 Jul 2020 17:49:46 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2020 17:49:46 +0000 Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 2CA2B4122D for ; Mon, 20 Jul 2020 17:49:46 +0000 (UTC) Received: by mail-lj1-f182.google.com with SMTP id z24so21161585ljn.8 for ; Mon, 20 Jul 2020 10:49:46 -0700 (PDT) X-Gm-Message-State: AOAM531CR0txoWYvkmEntrjxIaaWGJNutSAJC/JHDPOkI/7qIhUcybjs GDcBFjSzBwJ2p9P7lpDTwXRCbZoAXjw2+LtssNw= X-Google-Smtp-Source: ABdhPJym9KFPx1QDjpUv2XBmRizwRmOOCnqA6ijTFIHpSeTs/y9U8QU03dEiFqxLBiEixFIABZG5p6FBm3SUxY39q1Y= X-Received: by 2002:a2e:161a:: with SMTP id w26mr11380048ljd.369.1595267385055; Mon, 20 Jul 2020 10:49:45 -0700 (PDT) MIME-Version: 1.0 From: Sean Busbey Date: Mon, 20 Jul 2020 12:49:33 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: [DISCUSS] we need to take action if we want asf jenkins managed tests after Aug 15 2020. To: dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi folks! Back in April there was a brief discussion[1] about ASF Infra's notification that builds.a.o is going away and we are currently slated to migrate to a set of CI servers for "Hadoop and related projects". This is the ci farm that will contain the bulk of the H* worker nodes that are donated by Yahoo!, which are the nodes we've been running on for ages[2]. Migration discussion still happens on the hadoop-migrations@i.a.o list[3] and recently ASF Infra set a target date of August 15th for turning off the existing builds.a.o server[4]. That gives us a little under 4 weeks to have things up and working on the new ci-hadoop.a.o jenkins coordinator[5]. it=E2=80=99s not clear to me that the level of effort we=E2=80=99ll need to spend is worth what we get o= ut of a continuation of the status quo on builds.a.o. I did a quick test by updating the nightly job on ci-hadoop.a.o to run just branch-2, since that has been stable on builds.a.o. It failed with a Jenkins pipeline DSL syntax error[6] so I'm assuming migrating will be a slog. As far as I can see our options are: * Do nothing. Have no testing or automated website publication in mid Augus= t. * Transition website publication and nothing else (probably can be done in a day) * Transition just precommit testing for various repos (probably can be done in a few days) * Transition everything (no idea how long it takes due to nightly, flaky stuff, etc) The alternatives if we do not transition any given job to ci-hadoop: * Try to move to GitHub Actions * Try to move to Travis CI * Try to move to Jenkins infra we maintain ourselves (presumably by soliciting project specific donations for worker nodes on cloud vendors) It's important to remember that as a project we have a heavy footprint wherever our nightly tests run. For context, a given branch's nightly can keep 3-4 executors busy for 6+ hours on the current builds.a.o setup. There's been a bunch of great work lately on bringing down what it takes to run the full test suite, but applying that work to nightly is itself a significant undertaking. What are folks thinking? Most importantly who is ready to work towards any given approach? [1] [DISCUSS] Migrating HBase to new CI Master https://s.apache.org/fux1o [2] https://builds.apache.org/view/H-L/view/HBase/ [3] https://lists.apache.org/list.html?hadoop-migrations@infra.apache.org [4] [IMPORTANT] - 2 more HADOOP nodes migrated over to ci-hadoop https://s.apache.org/7e1nq [5] https://ci-hadoop.apache.org/job/HBase/ [6] https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2= /2/console