harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Luht" <anton.l...@gmail.com>
Subject Re: [Stress tests] generator: looking forward
Date Thu, 25 May 2006 14:57:42 GMT
Geir,

I've tried to put this proposal to the site. Please see the patch at
http://issues.apache.org/jira/browse/HARMONY-511 - hope I did it correctly.

-- 
Regards,
Anton Luht,
Intel Middleware Products Division


On 5/23/06, Geir Magnusson Jr <geir@pobox.com> wrote:
> How about a patch for the website, including instructions on what to do?
>
> Fedotov, Alexei A wrote:
> > Alex,
> >
> >> Please, find the new version (0.2) and all previous versions here:
> >> http://issues.apache.org/jira/secure/ManageAttachments.jspa?id=12340105
> >
> > Good. I like it (everyone likes when his suggestions are implemented
> > :-). So let me try to outline where we are in Harmony stress testing.
> >
> > === TEST DESIGN ===
> >
> >    * Stress tests are built from simple building blocks according to
> > configuration strings.
> >
> >    * Tests have junit interface.
> >        [Case study] Imagine someone puts tests into SVN which implements
> > different test interface. To reuse them we can add another generator to
> > convert these tests to junit interface.
> >
> >    * Configuration string list is maintained manually. If we plan to use
> > junit runner to launch a sequent of the stress tests, then the most
> > straightforward model is to wrap configuration strings into junit test
> > cases and put documentation into javadoc for these test cases.
> >
> > === FURTHER STEPS ===
> >
> >    * You wrote, "stress test suite should generate relevant bugs". Since
> > usually stress behavior is unspecified, we need to introduce something
> > measurable instead of pass/fail result for the stress tests. See my
> > thoughts about a comparative approach below.
> >
> >    * I will continue code reviews.
> >
> >    * All should create tests and run them against Harmony VM and RI.
> > This would be a real-life testing for our approach.
> >
> > === COMPARATIVE APPROACH ===
> >
> > The simplest example of comparative apporach is the following.
> >       Tester: My test fails on Harmony VM and passes on RI. Please,
> > fix Harmony VM.
> >
> > This usually does not work for stress tests.
> >       Developer: Who told you that OutOfMemoryError should be thrown
> > in your thread? My finalizer thread is just a normal java thread, like
> > yours, and it can fail as well. You have a bug in your test.
> >
> > There are multiple reasons why we always will have such bugs in the
> > tests.
> >    * These bugs keep showing up. The time to fix all these bugs
> > regularly is too high.
> >    * Stress testing reuses tests which are usually not designed for
> > stress execution, for example, multithread execution.
> >    * These bugs are dependent on VM internal structure. Test authors do
> > not posess sufficient knowledge of the problem and the structure.
> >    * Sometimes Java is not rich enough.
> >
> > How can we have a maintainable test product takung all this limitation
> > into account? We need to learn how to live with occasional failures of
> > the stress tests. This means, instead of fail, the test should better
> > report how good it is on Harmony VM compared to RI:
> >    * Failures with the worst relative metric can be evaluated first.
> >    * We can detect that a relative metric for a test worsened on the
> > recent build.
> >
> > Developers are better convinsed to fix "the worst issue" or
> > "dergadation" instead of "some issue".
> >
> > Now let me list here several metrics for each test.
> >    * Pass rate: assuming the test is 100% reliable on RI we can
> > calculate a percentage of failures.
> >    * Number of times the test can be executed sequentionally before a
> > fail.
> >    * Memory consumption: a generator can preallocate more and more
> > memory before launching the test in a loop.
> >    * Max threads supported: a generator can exponentially increase
> > number of threads launching the test in parallel.
> >    * Here is your metric.
> >    * Execution time: have you noticed that all this apparatus is quite
> > close to performance testing methodology? There is no need to compete
> > with them in their field though. :-)
> >
> > The thing I like most about this approach is it can be introduced on a
> > stress test generator level.
> >
> > With best regards,
> > Alexei Fedotov,
> > Intel Middleware Products Division
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message