From dev-return-29245-archive-asf-public=cust-asf.ponee.io@geode.apache.org Thu Jul 19 02:32:37 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 67EB4180636 for ; Thu, 19 Jul 2018 02:32:36 +0200 (CEST) Received: (qmail 19731 invoked by uid 500); 19 Jul 2018 00:32:35 -0000 Mailing-List: contact dev-help@geode.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.apache.org Delivered-To: mailing list dev@geode.apache.org Received: (qmail 19719 invoked by uid 99); 19 Jul 2018 00:32:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Jul 2018 00:32:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 55D66180103 for ; Thu, 19 Jul 2018 00:32:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.3 X-Spam-Level: * X-Spam-Status: No, score=1.3 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id zMxLDiQIO-dV for ; Thu, 19 Jul 2018 00:32:32 +0000 (UTC) Received: from mx0a-00296801.pphosted.com (mx0a-00296801.pphosted.com [148.163.150.38]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 3E4C45F39E for ; Thu, 19 Jul 2018 00:32:32 +0000 (UTC) Received: from pps.filterd (m0114583.ppops.net [127.0.0.1]) by mx0a-00296801.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w6J0VnHP002377 for ; Thu, 19 Jul 2018 00:32:30 GMT Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by mx0a-00296801.pphosted.com with ESMTP id 2k77xkuv44-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 19 Jul 2018 00:32:30 +0000 Received: by mail-lj1-f197.google.com with SMTP id n3-v6so1348940ljc.17 for ; Wed, 18 Jul 2018 17:32:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=S0Od5XQb33o0hROxwmF1I7N0Tmcb2vxzF/hF0l+UjB8=; b=dnEzPXOAa8udP3sXmKEzMbZ77D6BkpGUA4N4D4Pr36U0olyqsgInH6TkfxmmvkMcqt NJjmbSZCZ8K1ZqsciCiAYl7DIaGWIVXvEeWabvB73UXGcwTOyrH9alxebzCgPB2MdFHo KHN7+0gIdGXs8f45WK4wXX8PgR/24otDjI7Indd4gQ9w1t6pEz2qJ/YsikuxeMgHBsSt u+xjqZv5zYTPBw43y0tULgmDtX0kL3cLcJpOmuQtajhMDSWGDQFLV9EXbgNIMxi++F3G KJzmetK6yAsaflwp/rvN4Zo9AsiTQnc6jAbX+A85dHVUWNUxsgIRUtpvQMobKzObCIoO +gQQ== X-Gm-Message-State: AOUpUlEHzc4g4gvEWaSULf+L3blZr00AyPahMH4PHY/jKaKlsIOO3vZt ar1klDXEFNKeuBSGcM+/xNLx9AWA69ZCah7emfJ8AFUQWmksfgF17aJCwbVuzhTV2SmqKp81A9S rGTyzQFDPDWj3OoKDPmd9eN4wTc6uMpnXh1v63dsU5SceFlusc1jUMJI= X-Received: by 2002:a2e:9a95:: with SMTP id p21-v6mr6061428lji.60.1531960347541; Wed, 18 Jul 2018 17:32:27 -0700 (PDT) X-Google-Smtp-Source: AAOMgpea+j73ADX/Zna/ck5lliUEO9MQj5xqNq/+LOy4wOv7x+z7l8KK7jePS1pNdSJoABGw4zaVThmIJlHwUN1cQ7Q= X-Received: by 2002:a2e:9a95:: with SMTP id p21-v6mr6061423lji.60.1531960347374; Wed, 18 Jul 2018 17:32:27 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a19:ae09:0:0:0:0:0 with HTTP; Wed, 18 Jul 2018 17:32:26 -0700 (PDT) In-Reply-To: References: From: Alexander Murmann Date: Wed, 18 Jul 2018 17:32:26 -0700 Message-ID: Subject: Re: Slow DUnit tests for rolling upgrades To: dev@geode.apache.org Content-Type: multipart/alternative; boundary="00000000000050c98405714f50ad" X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-07-18_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1806210000 definitions=main-1807190003 --00000000000050c98405714f50ad Content-Type: text/plain; charset="UTF-8" Thanks for catching this, Dan! Not sure what happened there. Let's see if this works better: Why are rolling upgrade tests slow? ================================== Six versions of servers have to to be 1. Started with old version 2. Do some work 3. Shut down 4. Updated 5. Restarted 6. Do more work and make assertions The above happens for every test method individually. This takes ~1 minute per method and per version. This is done serially right now for all methods within a class. Our other DUnit tests can reuse JVMs. These tests are starting a fresh JVM. How can we make them faster? ============================ Option 1: Move every test method into its own class. This will allow us to run them in parallel This would allow us to run ~45 tests in parallel Might increase ease of exploring test suite. Option 2: Move rolling upgrade tests into their own Concourse job. This can be done in addition to other options. Option 3: Restricting what version jumps are supported. Can we decide to not support certain version jumps? Will we support jumping from Geode 1.x to arbitrary Geode 2.x version? Option 4: Change test runner to run for different version transitions in parallel. We could generate classes that contain the desired source and destination versions and then runs those in parallel. Next Actions =========== Options 1 & 2 seem to be fairly simple changes without much downside and should get us far. Implementing those two solutions and seeing how long that leaves our test suite seem like the best first step. Jacob Barrett will take the lead on turning these two options into action. On Wed, Jul 18, 2018 at 5:04 PM, Dan Smith wrote: > Can you fix the formatting of your message? Seems to have gotten mangled. > > On Wed, Jul 18, 2018 at 4:20 PM, Alexander Murmann > wrote: > > > Hi all, > > > > We just had a conversation among some members of the community about the > > increasingly slow DUnit tests that are related to rolling upgrades. > > > > Please chime in with questions and concerns! > > > > Here are my notes from the discussion: > > > > > > > > > > > > > > > > *Why are rolling upgrade tests slow?Six versions of servers have to to be > > 1. Started with old version2. Do some work3. Shut down4. Updated5. > > Restarted6. Do more work and make assertionsThe above happens for every > > test method individually. This takes ~1 minute per method and per > version. > > This is done serially right now for all methods within a class.Our other > > DUnit tests can reuse JVMs. These tests are starting a fresh JVM.How can > we > > make them faster?Option 1: Move every test method into its own class. > This > > will allow us to run them in parallelThis would allow us to run ~45 tests > > in parallelMight increase ease of exploring test suite.Option 2: Move > > rolling upgrade tests into their own Concourse job.This can be done in > > addition to other options.Option 3: Restricting what version jumps are > > supported.Can we decide to not support certain version jumps?Will we > > support jumping from Geode 1.x to arbitrary Geode 2.x version?Option 4: > > Change test runner to run for different version transitions in > parallel.We > > could generate classes that contain the desired source and destination > > versions and then runs those in parallel.Next ActionsOptions 1 & 2 seem > to > > be fairly simple changes without much downside and should get us far. > > Implementing those two solutions and seeing how long that leaves our test > > suite seem like the best first step. Jacob Barrett will take the lead on > > turning these two options into action.* > > > --00000000000050c98405714f50ad--