Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <515F0D5B.8090600@sbcglobal.net>
Date: Fri, 05 Apr 2013 10:43:55 -0700
From: Mike Matrigali <mikem_app@sbcglobal.net>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:17.0) Gecko/20130328 Thunderbird/17.0.5
MIME-Version: 1.0
To: derby-dev@db.apache.org
Subject: concern about nightly test failures on 10.10 branch prior to release.
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I have been looking at nightly results of the last few weeks, with an
eye to making sure 10.10 release does not have regressions over previous
releases.

I think in the past we have tried to get clean nightly test runs before 
making a release.  It is a problem as there are known intermittent 
errors, which make it hard to know if errors are regressions or not.

Looking at the public nightly test runs for 10.10 I see:

Java DB Testing:
http://download.java.net/javadesktop/derby/10.10.html
currently has 1 in a row clean recent runs, 3 out of most recent 7 have 
failures.  I think it is only running on checkins, so testing
of intermittent bugs is sparse.

IBM Testing:
http://people.apache.org/~myrnavl/derby_test_results/v10_10/windows/derbyall_history.html
http://people.apache.org/~myrnavl/derby_test_results/v10_10/windows/suites.All_history.html
http://people.apache.org/~myrnavl/derby_test_results/v10_10/linux/derbyall_history.html
http://people.apache.org/~myrnavl/derby_test_results/v10_10/linux/suites.All_history.html
currently has 0 in a row clean recent runs, and I don't think there
has been a "clean" day for 2 weeks.

I have not had time to look at all the failures, to determine if they
are regressions or not.  While not totally clean the 10.9 runs
for the IBM Testing are much cleaner, so just using that metric it
seems 10.10 is not ready to ship.:
http://people.apache.org/~myrnavl/derby_test_results/v10_9/windows/suites.All_history.html