From dev-return-29554-apmail-harmony-dev-archive=harmony.apache.org@harmony.apache.org Tue Oct 02 07:47:29 2007 Return-Path: Delivered-To: apmail-harmony-dev-archive@www.apache.org Received: (qmail 51781 invoked from network); 2 Oct 2007 07:47:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 2 Oct 2007 07:47:28 -0000 Received: (qmail 47048 invoked by uid 500); 2 Oct 2007 07:47:16 -0000 Delivered-To: apmail-harmony-dev-archive@harmony.apache.org Received: (qmail 47020 invoked by uid 500); 2 Oct 2007 07:47:16 -0000 Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@harmony.apache.org Delivered-To: mailing list dev@harmony.apache.org Received: (qmail 47011 invoked by uid 99); 2 Oct 2007 07:47:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Oct 2007 00:47:16 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of stepan.mishura@gmail.com designates 66.249.92.175 as permitted sender) Received: from [66.249.92.175] (HELO ug-out-1314.google.com) (66.249.92.175) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Oct 2007 07:47:18 +0000 Received: by ug-out-1314.google.com with SMTP id u40so1986112ugc for ; Tue, 02 Oct 2007 00:46:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=dbpCHfBBJMrVI5FH804fx+H1GpiPunZt7NJIfXwKIEI=; b=uQAqBKgZ6oBBnX12R0jEXW0oAbqMciRMxWCT67YIQ50Us/pz61IDqAtw8WGkVf1A5rjplfodLgNyfjQTzCCheT/blyohhJJUujzhHNK7bt9ojQ4o50aVtdB+JXhvGC0lQiJtY1Ztsnyfbb4Oo1nEA3N4OZOFy+7his/GWNgiZm8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=fI48aCG1DjdgX2EIkHH7ltscar1o52SrYtCKW5G7zxdFBX6os/UunN5YQ1mqbQ3ZjHzHN+DjA9xcPu2XfgWRDAL051Y+YcfwnPffxTqWtiT9jScrmbaC8A29YfDAXoJPLlzm/BGFho8yB1YWaYJcuNbdpTCZ5EDxVOuSxioH3DE= Received: by 10.67.24.11 with SMTP id b11mr680610ugj.1191311199892; Tue, 02 Oct 2007 00:46:39 -0700 (PDT) Received: by 10.67.20.15 with HTTP; Tue, 2 Oct 2007 00:46:39 -0700 (PDT) Message-ID: <6e47b64f0710020046g598cc2d8o1c5ba90a92fb2d3e@mail.gmail.com> Date: Tue, 2 Oct 2007 14:46:39 +0700 From: "Stepan Mishura" To: dev Subject: Re: [general][M3] release candidate status In-Reply-To: <6e47b64f0709280102t5fe06a2fid47fa8b1ebb7b3f3@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <6e47b64f0709280102t5fe06a2fid47fa8b1ebb7b3f3@mail.gmail.com> X-Virus-Checked: Checked by ClamAV on apache.org Hi, Currently, the next milestone candidate (r580985) is under testing. There were a number of investigation reports for r579330 and I'd like to update its summary to track the progress. All * Class library: - Windows: 1 crash (hard to reproduce) - Linux: 1 crash (HARMONY-4873), 1 failure (incorrect test - HARMONY-4872) * Eclipse 3.2 unit tests: - Windows: 4 failures - 1 is expected, 3 hard to reproduce (exclude list was updated) - Linux: issue with org.eclipse.jdt.core.tests.compiler.regression.TestAll suite timeout still remains. * Eclipse 3.3 unit tests: Success rate on Windows - 99.77% and Linux 99.46%. This results will be taken as baseline for tracking regressions. * Eclipse Geronimo Application (EGA) scenario x 48h - Windows: failed after 8 running hours, most probably the problem with scenario automation - no crashes/exceptions in the output * Eclipse TPTP Tests: - Linux: 1 intermittent failure * Functional Test Suite: 4 regressions were found - HARMONY-4866, HARMONY-4874, HARMONY-4857, HARMONY-4861 6 tests are invalid - HARMONY-4864(2 cases), HARMONY-4859 (1 case), JIRA for others?? HARMONY-4867 - should be defined whether this is regression or not Also there are many failures due to timeout or intermittent failures. The suite definitely must be improved. * Geronimo Unit Tests 2 regressions on Windows/Linux - no evaluation yet * JDKTools tests: Windows: 1 failure - reproducible only if the whole suite is run. (HARMONY-4868) * Reliability Test Suite: - Windows: 13 failures - according to the evaluation report most of them are intermittent and additional investigation is required - Linux: 14 failures - the same as for Windows * Stress Test Suite: - Windows: 3 failures - Linux: 11 failures Most of test failures due to timeout or OutOfMemoryError - the suite must be improved. There is HARMONY-4114 with 1Mb patch for the suite and harness but it is too risky to apply it during code freeze - will be applied in M4. * VTS VM Test Suite - Linux: 1 intermittent failure - problems in DRLVM shutdown code (HARMONY-????). Thanks, Stepan. On 9/28/07, Stepan Mishura wrote: > Hi all, > > OK. Here is testing status for r579330 M3 milestone candidate. > It is based on testing status page[1] and investigation reports for > suites on dev-list. > It is provided to decide whether r579330 is OK to be released or we > should fix found issues and build next release candidate. > > Status reporter please correct me if I miss something. > > I'd like to say that this in this milestone we used BTI for the first > time to get 100% of testing results. Some issues in BTI were fixed but > there are some still to be resolved, for example instability of > several suites, mostly Eclipse unit tests on Linux. > > > The following suites passed on Linux/Windows ia32 platform: Ant > Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests, > DRLVM regression tests, Jetty scenario, Scimark, Struts scenario, > Tomcat scenario > > The the next list of issues we have (per suite): > - Classlibrary unit tests: > 1 crash on Windows that is hard to reproduce: > 1 crash on Linux that is reproducible and it is bug in DRLVM (HARMONY-????): > 1 test failure on Linux (incorrect test HARMONY-????) > > - Eclipse unit tests 3.2 > Windows: 4 failures. 1 failure is expected (exclude list should be > updated). 3 failure are not reproducible > Linux: crash of > org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due > timeout > > - Eclipse unit tests 3.3 > Windows: 49 failures, pass rate 99.77%. We haven't tried the suite > on M2 so it is a good baseline for M3. > Linux: the same problem as for 3.2. I've setup another CC Linux > host for the suite and see no crashes, pass rate is 99.45% > > So I'm going to publish these reports for M3 and run 3.2 suite on > these lucky host to obtain results without the crash. > > - Eclipse Geronimo Application (EGA) scenario x 48h > Windows: there may be and issue with the scenario automation - the > scenario run more then 8 hours. As I understand several iterations > were done successfully. And it failed because of missing window, there > are no crash/exceptions in the log. > > Also the scenario passed on Linux. > > - Eclipse TPTP Tests > 1 failure on Linux - most probably intermittent > > - Functional Test Suite > 21 failures on Windows most of then failure due to timeout. 1 > regression (HARMONY-4857) > Uploading Linux failed - I'm going to check why. From the first > glance the results are same as for Windows. > > - Geronimo Unit Tests > 2 possible(I haven't seen analysis report) regressions on > Windows/Linux. These test passed on M2: > org.apache.geronimo.tomcat.ContainerTest > org.apache.geronimo.tomcat.JAASSecurityTest > > - JDKTools Tests > 1 failure on Windows. The test stably fails when it runs in a bunch > with others and may fail if run standalone : > > - Reliability Test Suite > The status is a bit unclear to me. There is 1 test is incorrect > test. And some tests failed because of timeout but its increase didn't > help. > Also some tests were removed from exclude list for r579330 but some > of them failed. > > - Stress Test Suite > 3 failures on Windows and 11 on Linux - I haven't seen analysis report > > - VTS VM Test Suite > 1 failure on Linux - the test passed on previous snapshot should be analysed. > > Any comments are welcome. > > [1] http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html > > Thanks, > Stepan Mishura > Intel Enterprise Solutions Software Division