Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Message-ID: <57A4A2B3.4090709@gmail.com>
Date: Fri, 05 Aug 2016 10:29:07 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Postbox 3.0.11 (Macintosh/20140602)
MIME-Version: 1.0
To: dev@accumulo.apache.org
Subject: Re: [DISCUSS] Time for a 1.8.0 release?
References: <CAMJ0=qG7gFVb_TP=jPQ_1JVpmvb4zPbfxS93aQ4P0rJTiW8kkA@mail.gmail.com> <CAL5zq9ZP883BsKvi6pcYRGiSKo6xm_VqRc3W9UM4Bacb-1wB3A@mail.gmail.com> <CAMJ0=qEGjH8UtFNtufup=2c5K+F5DB4CvFvkJtPktxAFCyR8fg@mail.gmail.com> <CAMJ0=qHdu3m3YLZzv8Te5uZp4gUZE-ixj9Ksv4Zy6Q48wjTctQ@mail.gmail.com> <CAGUtCHocFhBy6XDhA-H+y-ykWpEkMRY+mG1BhKNBCkdHzaoy1A@mail.gmail.com> <CAMJ0=qGQo31uR=-usfPztRVRMjsdCd8St7z7HkpUMAKFnOWqKA@mail.gmail.com> <CAGHyZ6+GMDwpy04yUQSbxTgf=uKaxDS69i7HBmMh=-WUF8xRew@mail.gmail.com> <CAMJ0=qGPFcNUW6_mnxq1EUthkUOS9_vVoJ6n2tRWb55p_7-OuA@mail.gmail.com> <CAGHyZ6KgodQ=ZzFdPNDpKM5r-5LsOm8e64XPjgVDzL14K6fq7g@mail.gmail.com> <CAL5zq9aY98FF17WEXhutuMxzEm8Y6W3OgxSLKFjeiQqHQGi2cQ@mail.gmail.com> <CAGHyZ6JNQr8qipMisSGm_2jEXsHNwmOr2xxM_U49fn4ysZ1x0g@mail.gmail.com>
In-Reply-To: <CAGHyZ6JNQr8qipMisSGm_2jEXsHNwmOr2xxM_U49fn4ysZ1x0g@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Fri, 05 Aug 2016 14:29:14 -0000

Sean Busbey wrote:
> On Wed, Aug 3, 2016 at 5:17 PM, Christopher<ctubbsii@apache.org>  wrote:
>> On Wed, Aug 3, 2016 at 5:47 PM Sean Busbey<busbey@cloudera.com>  wrote:
>>
>>> My understanding was that maintenance releases (aka double dot, e.g.
>>> 1.7.2) had relaxed criteria because we expected the scope of changes
>>> in them to be more limited. Even so, the release notes for 1.7.2,
>>> 1.7.1, and 1.7.0 all claim the ITs passed.
>>>
>>>
>> Even those releases have periodic IT failure.
>>
>>
>>> Is there a reason we can't parallelize the ITs?
>>
>> We can. Eric's mrit effort was all intended towards that. But, that's not
>> the same as CI passing. I don't know what it would take to parallelize them
>> in a CI server.
>>
>>
>>> What's stopping
>>> builds.a.o from running them? Specific requests from projects to asf
>>> infra can get us resources if that's the problem.
>>>
>>>
>> I spoke to infra in HipChat about this a a few weeks ago, and mentioned a
>> few things which impact builds on ASF jenkins (builds.apache.org):
>>
>> 1. Accumulo has an excessive number of tests to run.
>> 2. Build timeouts with Jenkins can abort builds.
>> 3. Tests are timing sensitive, and are affected by VM/host configuration
>> and contention with other concurrent builds from other projects.
>> 4. Tests need lots of RAM and storage (at least 4GB RAM, but ideally no
>> less than 16GB, and at least 6 GB for a workspace)
>> 5. Tests need specialized system configuration, (increasing ulimits,
>> optimizing kernel settings for swappiness, etc.)
>>
>> What we really need for reliable IT passing in CI, is exclusive use of
>> dedicated, bare-metal beefy build machines, for 6+ hours per build x 4
>> branches minimum, plus another 6+ hours for each pull request and other
>> builds which skipITs, so we can get immediate feedback on unit tests and
>> compilation errors.
>>
>
> I took a first pass at a nightly (~once per 12 hours) job on asf build for
> master and it did okay, considering that I haven't spent any time trying to
> tune anything:
>
> https://builds.apache.org/job/Accumulo-master-IT/1/
>
> 2 hr 9 min, 7 failures out of 202 tests.
>
> I think we can do this; if anyone else is interested I'll start a new thread
> where we can discuss.

+1 it would be great to do this on ASF infra.