Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (athena.apache.org: domain of andrey.yakushev@gmail.com
 designates 64.233.166.181 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=XXEjv5Wb2VIMWiI/ruXSNOY1m4aMHt7m+f0rF5JVr3kmcnxsQLylwp6kyj8H6AF35jWLmfK9UoKena2g51hQbKlvDxAlZ6siC79K5XELiyox1dJvVoPi87qEdOLBIRtQ1E5Ucv4FncshNTntL25ySzA73a/wHsIzDwxKOAKHoLw=
Message-ID: <3c7e1c080710010245v10b85ee1i16a0f83e3383a0c2@mail.gmail.com>
Date: Mon, 1 Oct 2007 13:45:41 +0400
From: "Andrey Yakushev" <andrey.yakushev@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [general][M3] release candidate status
In-Reply-To: <3c7e1c080709280815x60bf2053m2925e0d9b61e0b8e@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <6e47b64f0709280102t5fe06a2fid47fa8b1ebb7b3f3@mail.gmail.com>
	 <46FCC7F4.1060403@gmail.com>
	 <3c7e1c080709280815x60bf2053m2925e0d9b61e0b8e@mail.gmail.com>

> > >   Also some tests were removed from exclude list for r579330 but some
> > > of them failed.
> Also strange, they never failed on my test runs. Need additional investigation.

I looked at fails and it appeared that two tests were reported as failed:

api.text.DecimalFormat_Locales
api.zip.ZlibTest

But these tests worked file until killing by timeout, and
api.text.DecimalFormat_Locales made 28 successful iteration, and
api.zip.ZlibTest - 225 successfull iterations on Linux. So it's again
host slowing down or probably algorithm of stoping the iteraration
could be rechecked.

My overall impression is that reliability was increased between M2 and M3.

Thanks,
Andrey


On 9/28/07, Andrey Yakushev <andrey.yakushev@gmail.com> wrote:
> > > - Reliability Test Suite
> > >   The status is a bit unclear to me. There is 1 test is incorrect
> > > test. And some tests failed because of timeout but its increase didn't
> > > help.
>
> Really it helped for 5 tests of 9, but 9 different tests failed.
> Almost all reliability tests failures are intermittent. And my local
> run for r579492 showed much lower failure ratio: 6 valid fails on
> Windows and 3 on Linux x86. So cruise control hosts could catch
> additional race conditions or similar problems somehow.
>
> > >   Also some tests were removed from exclude list for r579330 but some
> > > of them failed.
>
> Also strange, they never failed on my test runs. Need additional investigation.
>
> >
> > Is anybody familiar enough with the history of the reliability and
> > stress test results to offer an opinion here?
>
> I watched reliability results, and they improved from 25-30 usual
> fails in August to 10-15 on the latest builds. More precisely:
>
> Date Windows-x86 Linux-x86 failures
> Sept/25 13 14
> Sept/22 9 13
> Sept/20 ?? 10
> Sept/18 15 15
> Sept/16 18 15
> Sept/14 15 18
> Sept/11 10 9
> Sept/7 ?? 12
>
>
> Thanks,
> Andrey
>
>
> On 9/28/07, Tim Ellison <t.p.ellison@gmail.com> wrote:
> > This is a wonderful summary, thanks Stepan.
> >
> > Comments in-lined below.
> >
> > Stepan Mishura wrote:
> > > Hi all,
> > >
> > > OK. Here is testing status for r579330 M3 milestone candidate.
> > > It is based on testing status page[1] and investigation reports for
> > > suites on dev-list.
> > > It is provided to decide whether r579330 is OK to be released or we
> > > should fix found issues and build next release candidate.
> > >
> > > Status reporter please correct me if I miss something.
> > >
> > > I'd like to say that this in this milestone we used BTI for the first
> > > time to get 100% of testing results. Some issues in BTI were fixed but
> > > there are some still to be resolved, for example instability of
> > > several suites, mostly Eclipse unit tests on Linux.
> >
> > Yes, this is getting a lot more testing, which is great to see.
> >
> > > The following suites passed on Linux/Windows ia32 platform: Ant
> > > Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests,
> > > DRLVM regression tests, Jetty scenario, Scimark, Struts scenario,
> > > Tomcat scenario
> >
> > Cool.
> >
> > > The the next list of issues we have (per suite):
> > > - Classlibrary unit tests:
> > >     1 crash on Windows that is hard to reproduce:
> > >     1 crash on Linux that is reproducible and it is bug in DRLVM (HARMONY-????):
> > >     1 test failure on Linux (incorrect test HARMONY-????)
> >
> > I looked into that one failing test in java.net and I maintain that the
> > test is prone to failure because it is using the deprecated
> > Support_PortManager.  I created HARMONY-4872 to track it, but believe it
> > is not a blocking issue for M3.
> >
> > > - Eclipse unit tests 3.2
> > >     Windows: 4 failures. 1 failure is expected (exclude list should be
> > > updated). 3 failure are not reproducible
> > >     Linux: crash of
> > > org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due
> > > timeout
> >
> > It crashes rather than fails when timing out?  Is there a JIRA for this
> > one too that you know about?
> >
> > > - Eclipse unit tests 3.3
> > >    Windows: 49 failures, pass rate 99.77%. We haven't tried the suite
> > > on M2 so it is a good baseline for M3.
> > >    Linux: the same problem as for 3.2. I've setup another CC Linux
> > > host for the suite and see no crashes, pass rate is 99.45%
> >
> > Looks good enough for where we are today.
> >
> > >   So I'm going to publish these reports for M3 and run 3.2 suite on
> > > these lucky host to obtain results without the crash.
> > >
> > > - Eclipse Geronimo Application (EGA) scenario x 48h
> > >    Windows: there may be and issue with the scenario automation - the
> > > scenario run more then 8 hours. As I understand several iterations
> > > were done successfully. And it failed because of missing window, there
> > > are no crash/exceptions in the log.
> > >
> > >   Also the scenario passed on Linux.
> >
> > That is looking good too then.
> >
> > > - Eclipse TPTP Tests
> > >   1 failure on Linux - most probably intermittent
> > >
> > > - Functional Test Suite
> > >    21 failures on Windows most of then failure due to timeout. 1
> > > regression (HARMONY-4857)
> > >    Uploading Linux failed - I'm going to check why. From the first
> > > glance the results are same as for Windows.
> >
> > Would be good to figure out what is happening with HARMONY-4857.  How
> > about we mark this as an M3 blocker until we understand it better?
> >
> > > - Geronimo Unit Tests
> > >   2 possible(I haven't seen analysis report) regressions on
> > > Windows/Linux. These test passed on M2:
> > >   org.apache.geronimo.tomcat.ContainerTest
> > >   org.apache.geronimo.tomcat.JAASSecurityTest
> >
> > JIRA issues for these too?
> >
> > > - JDKTools Tests
> > >   1 failure on Windows. The test stably fails when it runs in a bunch
> > > with others and may fail if run standalone :
> > >
> > > - Reliability Test Suite
> > >   The status is a bit unclear to me. There is 1 test is incorrect
> > > test. And some tests failed because of timeout but its increase didn't
> > > help.
> > >   Also some tests were removed from exclude list for r579330 but some
> > > of them failed.
> > >
> > > - Stress Test Suite
> > >   3 failures on Windows and 11 on Linux - I haven't seen analysis report
> >
> > Is anybody familiar enough with the history of the reliability and
> > stress test results to offer an opinion here?
> >
> > > - VTS VM Test Suite
> > >   1 failure on Linux - the test passed on previous snapshot should be analysed.
> > >
> > > Any comments are welcome.
> >
> > Good job! I think that although there are many failures reported, there
> > are many more tests and testing scenarios being exercised.  We should
> > chase down the critical bugs for M3, but be prepared to triage the
> > others into the M4 task list.
> >
> > Regards,
> > Tim
> >
> > > [1] http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html
> > >
> > > Thanks,
> > > Stepan Mishura
> > > Intel Enterprise Solutions Software Division
> > >
> >
>
>
> --
> Thanks,
> Andrey
>


-- 
Thanks,
Andrey