harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oleg Oleinik" <oleg.olei...@gmail.com>
Subject Re: [DRLVM] General stability
Date Wed, 08 Nov 2006 05:31:17 GMT
>  "build test" is necessary but not sufficient for where we need to go.  In
a sense, committing code with only "build test" to prevent regression is the
equivalent to flying in the fog without instrumentation.

Right.

Here I see 2 aspects - creating VM/JIT regression tests and running various
tests regularaly / tracking regressions timely.

1. VM/JIT regression test development.

We can start with these guidelines:

Who creates/integrates a regression test? - committer.
How? - typically, from bug's reproducer code.
Where? - drlvm\trunk\src\test\vm\<component>\harmony-XXXX\ or
drlvm\trunk\src\test\jit\<component>\harmony-XXXX\
 Format: Junit, Jasmin
When? - Consider creating a regression test for each fixed bug against
Harmony DRLVM or JIT, if reasonable and technically possible.
Creation of regression test may be omitted if:
- Bug in documentation, build, test, code comment, code style, enhancement
is fixed.
- Performance-related bug is fixed.
- Bug is found by existing Harmony test (Harmony regression or unit test).
// to avoid duplication of tests

Try to use public API to provide implementation independence and stability
of the tests.

This test suite can/should be used in pre-commit testing.


2. Timely regression tracking via test execution.

Now we have some solution for code integrity testing via Cruise Control
(buildtest/trunk/CC + HARMONY-995) - good start point to track regressions
hourly using classlib unit tests and "build test" on your specific platform.

I see HARMONY-2038 which is about automation of Ecilpse Unit Tests execution
in context of buildtest/ infrastructure - to start running EUT regularly
using buildtest/ (say, nightly) and reporting timely about new test failures
(and causing commits).

I think, it would be great if we continue adding more automation scripts
into buildtest/ for various public unit test suites and scenarios (Derby,
Tomcat, etc.), so that one could take automation scripts from buildtest/ and
run them regularly on his specific platform, report regressions timely (what
Nina, I suppose, is going to do with EUT).



On 11/4/06, Weldon Washburn <weldonwjw@gmail.com> wrote:
>
> Folks,
>
> I have spent the last two months committing patches to the VM.  While we
> have added a ton of much needed functionality, the stability of the system
> has been ignored.  By chance, I looked at thread synchronization design
> problems this week.  Its very apparent that  we lack the regression
> testing
> to really find threading bugs, test the fixes and test against regression.
> No doubt there are similar problems in other VM subsystems.   "build test"
> is necessary but not sufficient for where we need to go.  In a sense,
> committing code with only "build test" to prevent regression is the
> equivalent to flying in the fog without instrumentation.
>
> So that we can get engineers focused on stability, I am thinking of coding
> the JIRAs that involve new features as "later" or even "won't
> fix".  Please
> feel free to comment.
>
> We also need to restart the old email threads on regression tests.  For
> example, we need some sort of automated test script that runs Eclipse and
> tomcat, etc. in a deterministic fashion so that we can compare test
> results.  It does not have to be perfect for starts, just repeatable and
> easy to use.  Feel free to beat me to starting these threads :)
>
> --
> Weldon Washburn
> Intel Enterprise Solutions Software Division
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message