From dev-return-26635-apmail-geode-dev-archive=geode.apache.org@geode.apache.org Fri Sep 15 20:08:41 2017 Return-Path: X-Original-To: apmail-geode-dev-archive@minotaur.apache.org Delivered-To: apmail-geode-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 68B131A200 for ; Fri, 15 Sep 2017 20:08:41 +0000 (UTC) Received: (qmail 81831 invoked by uid 500); 15 Sep 2017 20:08:41 -0000 Delivered-To: apmail-geode-dev-archive@geode.apache.org Received: (qmail 81786 invoked by uid 500); 15 Sep 2017 20:08:40 -0000 Mailing-List: contact dev-help@geode.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.apache.org Delivered-To: mailing list dev@geode.apache.org Received: (qmail 81774 invoked by uid 99); 15 Sep 2017 20:08:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Sep 2017 20:08:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1471518547C for ; Fri, 15 Sep 2017 20:08:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.781 X-Spam-Level: * X-Spam-Status: No, score=1.781 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=pivotal-io.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id yI9HE5uHBg4X for ; Fri, 15 Sep 2017 20:08:36 +0000 (UTC) Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 902905F6C8 for ; Fri, 15 Sep 2017 20:08:36 +0000 (UTC) Received: by mail-it0-f54.google.com with SMTP id 4so4669068itv.4 for ; Fri, 15 Sep 2017 13:08:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pivotal-io.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=9zJJyeXkdnvmJQDAhNQcixuq+dSKePyd5/N3rL84v/c=; b=sPyiCBcF7uGG15UgYU3X300cIe3zMayFNTbujgPa7+ahdFEHwmerQTJvgSEIt+81iA /wqCxgk/ijugxulLBg4El/iZciub0OxMjYmzYiyWI53syImBqDXk7bHqHuPVHt+r/eMn oLhCO4ql6ysHGFVfrLfWNqzci5WxFxq/svrjvt9s+XOg9xm+m59DC4LcZkkPrNQTD5Yt bA9YYZej9qifnGcg3y4v29R/kbciPiDxfhLbB/eX0pP2t/uJPuPvx1Z99/Slb/WRD3SZ 0qde9gZNGG7FWB9xKjkz4smJDMGNwAXylgdozjkKCn7WWf1Cp5thkT1uvJsmylTgRLiq 79rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=9zJJyeXkdnvmJQDAhNQcixuq+dSKePyd5/N3rL84v/c=; b=cDuZAAlWooaVV0TFUUrWAUHdwsAxjTjthLVefGe33ubwJ0t+dUXofqn1K3EydqmBYs 3SzD5c0X6UptQsv2NGDPqC9fQl1OBVWtlZA1bMSFXHMkS/Wbr76Se8ocGW8BF3So/gLw 1GWmscABZ2/AYZguNh8ByH2ID+qglTgBcdZ+CQ23blwR1B/2u3kOJEU5z2S8Sgh85wav QtfaTymEbU+QU2G5elyrmmZIkcLUO8DPQQfbDS4AXroIy3ELWSYpvHh7UboyKywpJ8Hv 4mJnV3TDoR+9yp5bD2hVagXjvtFV0ZwdqH9cH2QVeKjHLwb1MGRLLmduitU7Hk7a/A2K Tt4A== X-Gm-Message-State: AHPjjUi7n4RPy380ak1g01el7kD6ZPmZVOHPXoJeQ/5ByCHBiK8MMno7 4/4+gTQ+QSsYp2JI5Q//wsjZx48jIWvyZip1bsEq5w== X-Google-Smtp-Source: AOwi7QA+3bTeQ7qudi8uCBVjIXj/s8mRfiIIZu9n6R5z9gTBoT4mDeIcpou50UWkP4OXxadIRqEV9bq8m7aiGkIF3iY= X-Received: by 10.36.211.203 with SMTP id n194mr7183815itg.106.1505506115665; Fri, 15 Sep 2017 13:08:35 -0700 (PDT) MIME-Version: 1.0 References: <6EE7DC98-E723-4C79-AD3F-B74E6006597A@pivotal.io> <58e920cc-c66c-7233-9161-a485f411d93d@pivotal.io> In-Reply-To: From: Jason Huynh Date: Fri, 15 Sep 2017 20:08:25 +0000 Message-ID: Subject: Re: [DISCUSS] Clean build takes 10minutes to complete now To: geode Content-Type: multipart/alternative; boundary="001a1145f7203b985505593ff50b" --001a1145f7203b985505593ff50b Content-Type: text/plain; charset="UTF-8" For the original issue, where the old version is pulling down the old versions during compilation time, we have a pull request to use the maven repo: https://github.com/apache/geode/pull/790 The rest of the tests added for the session state are marked as @Category ({DistributedTest.class, BackwardCompatibilityTest.class}) geode-old-versions happens to be required at compile time. The old versions will now (after the pull request is checked in) pull down the zip into the local repo once. Unzipping will still be required but that should be a lot shorter than downloading. Whether we should move the old versions out of the compile task is a different issue... On Fri, Sep 15, 2017 at 8:55 AM Kirk Lund wrote: > The actual tests marked with UnitTest category are actually pretty good. > They all run in just over one minute and almost all of them use Mockito to > isolate one class. I remember seeing one newer Lucene UnitTest that touches > File System which should be recategorized as IntegrationTest. > > If we could move the pulling down of previous versions of Geode out of the > main build+unit-test target, that would help a lot. > > Even prior to the pulling down of previous versions for backwards compat > testing, the main build (without unit-test) was too slow and I think it's > because our project is a little too complex for what Gradle is designed to > handle. > > Code generation and javadocs are two of the tasks in our main build > (without unit-test) that contributes to it taking too long. > > Also, the way Gradle handles junit categories is designed and coded very > inefficiently -- if we could change their junit runner to use > FastClasspathScanner to find all tests containing the targeted junit > category annotation then that would speed up all of our testing targets > immensely. Any testing target that forks JVMs runs super slow due to the > way they handle categories (this effects IntegrationTest, DistributedTest, > FlakyTest). > > On Fri, Sep 15, 2017 at 8:44 AM, Alexander Murmann > wrote: > > > I fully agree with Udo here. The main build should be for Unit tests. Our > > "Unit Tests" are already exercising much more of the system than they > > should. Adding unit tests that not only too much or our current code but > > also old code is moving us in the wrong direction. Let's keep the tests, > > but please appropriately mark them as IntegrationTest. > > > > On Tue, Sep 12, 2017 at 9:30 AM, Udo Kohlmeyer > > wrote: > > > > > My apologies, I might gotten the commit reason incorrect. I just know > > that > > > downloading the older product version every time is becoming painful. > > > Yes, sometimes it is faster than other times, but imo, this is not > > > something that should be part of the main build path. > > > > > > Backwards compat or integration testing should not be running as part > of > > > the main build task. > > > > > > --Udo > > > > > > On Tue, Sep 12, 2017 at 9:05 AM, Nabarun Nag wrote: > > > > > > > As we are working on fixing this issue, some extra parameters may > help > > > the > > > > build to get bit quicker on your machine. > > > > > > > > using -xjavadoc -xdoc > > > > Eg: ./gradlew clean build -Dskip.tests=true -xjavadoc -xdocs > > > > BUILD SUCCESSFUL > > > > Total time: 2 mins 2.729 secs > > > > > > > > > > > > Also, I think as Jason mentioned that the slow down is due to full > > > product > > > > download for session state tests. LuceneSearchWithRollingUpgradeDUnit > > > > tests > > > > were added in July. Please do correct me if I am wrong. > > > > > > > > Regards > > > > Nabarun > > > > > > > > > > > > On Tue, Sep 12, 2017 at 11:47 AM Alexander Murmann < > > amurmann@pivotal.io> > > > > wrote: > > > > > > > > > Could we make it so that these tests for now are only run as part > of > > > > > pre-checkin till we got this ironed out and then revisit this? > > > > > > > > > > On Tue, Sep 12, 2017 at 8:32 AM, Bruce Schuchardt < > > > > bschuchardt@pivotal.io> > > > > > wrote: > > > > > > > > > > > The geode-old-versions module was originally created to pull in > old > > > > > > version jar files into your gradle cache. This happened only > once > > > and > > > > > you > > > > > > were good to go. I don't think that part should be backed out as > > it > > > > has > > > > > > minimal impact and is not affecting build time. > > > > > > > > > > > > The recent changes for lucene testing seem to be pulling in full > > > > > > installations of old versions and these are deleted as part of > the > > > > > "clean" > > > > > > gradle task. That's causing them to be downloaded again each > time > > > you > > > > > do a > > > > > > clean&build. Dan put changes in place so that the files aren't > > > > > downloaded > > > > > > again if you build without cleaning but clearly more needs to be > > done > > > > in > > > > > > this area. > > > > > > > > > > > > > > > > > > > > > > > > On 9/11/17 11:23 AM, Jacob Barrett wrote: > > > > > > > > > > > >> Agreed, integration tests should not be part of the build > process. > > > > This > > > > > >> is clearly an integration test. > > > > > >> > > > > > >> On Sep 11, 2017, at 11:00 AM, Udo Kohlmeyer > > wrote: > > > > > >>> > > > > > >>> Hi there, > > > > > >>> > > > > > >>> With a recent addition to the build scripts, to test lucene > > > backwards > > > > > >>> compatibility, a step was added to download a previous version > of > > > > > GEODE. > > > > > >>> > > > > > >>> This is causing longer build times now, which is a real > > > distraction. > > > > In > > > > > >>> cases where one would like to work on a branch, rebase that on > > > > develop > > > > > and > > > > > >>> merge that, this step becomes a real time hog. > > > > > >>> > > > > > >>> I request that we remove this default behavior from a clean > build > > > > until > > > > > >>> we have a better solution to this issue. > > > > > >>> > > > > > >>> I also believe that if anyone wants to add behavior like this > > into > > > > the > > > > > >>> default build, that it at least is discussed on the dev list > > before > > > > > >>> implementing this. > > > > > >>> > > > > > >>> --Udo > > > > > >>> > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Kindest Regards > > > ----------------------------- > > > *Udo Kohlmeyer* | *Snr Solutions Architect* |*Pivotal* > > > *Mobile:* +61 409-279-160 <+61%20409%20279%20160> | > ukohlmeyer@pivotal.io > > > > > > www.pivotal.io > > > > > > --001a1145f7203b985505593ff50b--