From dev-return-49225-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Sun Apr 5 18:23:22 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5FF14180638 for ; Sun, 5 Apr 2020 20:23:22 +0200 (CEST) Received: (qmail 54105 invoked by uid 500); 5 Apr 2020 18:23:21 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 54093 invoked by uid 99); 5 Apr 2020 18:23:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Apr 2020 18:23:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BF868C1E80 for ; Sun, 5 Apr 2020 18:23:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.201 X-Spam-Level: X-Spam-Status: No, score=-0.201 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id yYRNrmBiAIlf for ; Sun, 5 Apr 2020 18:23:19 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.182; helo=mail-lj1-f182.google.com; envelope-from=paul.joseph.davis@gmail.com; receiver= Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id A66F4BB8A7 for ; Sun, 5 Apr 2020 18:23:18 +0000 (UTC) Received: by mail-lj1-f182.google.com with SMTP id t17so12233841ljc.12 for ; Sun, 05 Apr 2020 11:23:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=lyYeWMDMRarA367rGLX27DPmIDrVRuFm2U/4cTzzn24=; b=cgLrrDiExFQEwHSxgAy10RRl4OI2TaP/pZV7Ullnqei3mgJyl4qQQyXhymLKkdAXpK ibvFSqarYEf39MEk/6kqDlDN5FixuwUhV6SnCGPfuSsoaaCFWzlqfB4mB5BU0zQZsKge 3uqgcrWbGoCE3e2gOsbFCzrv8lDdn3UBXwUJGrdW1eL8GeDyjYY8cNozldFswPbsKgH5 6LgkXliLYKTtS0E70H4X8hWE0G5PQ6QnxjM6hvciZ8n3l0TTNmXkzHKZt3anS5Wsl5WR tGFM7jE8EnPSMSCuQ2Z+pkbZtJlwIGRc/aCAxnro/jUFxhpoBUSJS+AOIyq+pxJc6BIZ 135w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=lyYeWMDMRarA367rGLX27DPmIDrVRuFm2U/4cTzzn24=; b=YTy5z+zcLJp4CMAhM3WHbik+bBxgv34OZVNZxmj1DkAMZCysCqQ56gWkKIG1VllisA ly4XmpY7SNSPvzkxlR8fdEta2p95kl5P+1DD1PmzhfvrlVeoYjAjYTusgSHzwRsV+syB n6jKXX6GJpSYb5IMk+umLs+PCdKr2UD5+0gzU992Ca2EZvbfeNo6cEBbHfCeogt2ll4S qlGvZnv1r8FffL2KBsn6CXWOXgXeKbgzELEMqpenTlbwqwlDd2B/oW9tSjtrqMGHUp4P 5uu9i/GLwjwuUu7jI7ur267aqIyVUXNYYyrsUUqCZz1ubgQSAX3lrNxgiA0BvgmTJWnW qfiw== X-Gm-Message-State: AGi0Puaw7imoVOKTqEWQ8dsvPCVX6mkR9J0QD28BAIqLzxObuvJYtqXT PgsAvZ/adRWFvPbLzCvHwd+93FL05yxaMTYoR4rcAw== X-Google-Smtp-Source: APiQypKFwxO686VLJ2MH9XPu2VZC5sJ5/FnmqtE2lwNVEKSdMeEfiTmBsqDabggtsxKj8ejnWynYixGrG/gZflHAIpw= X-Received: by 2002:a05:651c:30b:: with SMTP id a11mr9940591ljp.164.1586110997276; Sun, 05 Apr 2020 11:23:17 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Paul Davis Date: Sun, 5 Apr 2020 13:22:41 -0500 Message-ID: Subject: Re: Cloudbees Operations Center and Client Masters upgrade this weekend. To: dev@couchdb.apache.org, gmcdonald@apache.org Content-Type: text/plain; charset="UTF-8" Howdy Gavin, I've updated everything besides the two FreeBSD nodes that Joan manages. Is there a trick to having Jenkins realize a node has upgraded its slave.jar? Took me an extra twenty minutes fumbling around before realizing that slave.jar is now actually agent.jar and that even though Jenkins realized the jar had actually been updated. I ended up realizing that I had to go through each node in the web UI to re-enable each worker. Joan, for the FreeBSD ones I'm not sure how you set up the run scripts. For me it was just a `sudo sv restart jenkins` on each node because I download the agent.jar as part of `/etc/sv/jenkins/run`. Then it was just a random Googling till I happened to notice that was enough to upgrade the jar and needed to click the button. Paul On Sun, Apr 5, 2020 at 11:32 AM Gavin McDonald wrote: > > Hi All, > > This is now completed. > > Overall time spent was around 3 hours on the Operations Center and 5 > masters. > > When the OC was being upgraded, there was no downtime for the masters. > > Each master had downtime of around 20 minutes to 30 minutes, including > multiple restarts > , backups and plugin upgrades. > > CouchDB folks - most of your nodes are connected via 'node connects to the > master' meaning that you'll > have to update each slave.jar manually. The preferred method is ssh i/o > non-blocking, all nodes connected > in this fashion all came back on their own without intervention. > > Thanks all > > Gavin McDonald (ASF Infra) > > > On Sat, Apr 4, 2020 at 3:20 PM Gavin McDonald wrote: > > > Hi All, > > > > I'll be upgrading the Cloudbees Core Operations Center and all attached > > Masters on Sunday 5th April at around 10am UTC until completed. > > > > I'll turn off new builds around 7am UTC. > > > > I'm 'expecting' a couple of hours downtime assuming all goes well, > > including upgrading all > > associated plugins, performing backups etc. But, this is the first time > > I'm upgrading this setup so there may be the odd gotcha that surprises me > > who knows! - I'm as prepared as possible and I am not expecting any > > problems. > > > > Thanks > > > > Gavin McDonald (ASF Infra) > > > >