harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: [Stress tests] generator: looking forward
Date Tue, 23 May 2006 17:45:12 GMT
How about a patch for the website, including instructions on what to do?

Fedotov, Alexei A wrote:
> Alex,
> 
>> Please, find the new version (0.2) and all previous versions here:
>> http://issues.apache.org/jira/secure/ManageAttachments.jspa?id=12340105
> 
> Good. I like it (everyone likes when his suggestions are implemented
> :-). So let me try to outline where we are in Harmony stress testing.
> 
> === TEST DESIGN ===
> 
>    * Stress tests are built from simple building blocks according to
> configuration strings.
> 
>    * Tests have junit interface.
>        [Case study] Imagine someone puts tests into SVN which implements
> different test interface. To reuse them we can add another generator to
> convert these tests to junit interface.
> 
>    * Configuration string list is maintained manually. If we plan to use
> junit runner to launch a sequent of the stress tests, then the most
> straightforward model is to wrap configuration strings into junit test
> cases and put documentation into javadoc for these test cases.
> 
> === FURTHER STEPS ===
> 
>    * You wrote, "stress test suite should generate relevant bugs". Since
> usually stress behavior is unspecified, we need to introduce something
> measurable instead of pass/fail result for the stress tests. See my
> thoughts about a comparative approach below.
> 
>    * I will continue code reviews.
> 
>    * All should create tests and run them against Harmony VM and RI.
> This would be a real-life testing for our approach.
>  
> === COMPARATIVE APPROACH ===
> 
> The simplest example of comparative apporach is the following.
> 	Tester: My test fails on Harmony VM and passes on RI. Please,
> fix Harmony VM.
>    
> This usually does not work for stress tests.
> 	Developer: Who told you that OutOfMemoryError should be thrown
> in your thread? My finalizer thread is just a normal java thread, like
> yours, and it can fail as well. You have a bug in your test. 
> 
> There are multiple reasons why we always will have such bugs in the
> tests.
>    * These bugs keep showing up. The time to fix all these bugs
> regularly is too high.
>    * Stress testing reuses tests which are usually not designed for
> stress execution, for example, multithread execution.
>    * These bugs are dependent on VM internal structure. Test authors do
> not posess sufficient knowledge of the problem and the structure.
>    * Sometimes Java is not rich enough.
> 
> How can we have a maintainable test product takung all this limitation
> into account? We need to learn how to live with occasional failures of
> the stress tests. This means, instead of fail, the test should better
> report how good it is on Harmony VM compared to RI:
>    * Failures with the worst relative metric can be evaluated first.
>    * We can detect that a relative metric for a test worsened on the
> recent build.
> 
> Developers are better convinsed to fix "the worst issue" or
> "dergadation" instead of "some issue".
> 
> Now let me list here several metrics for each test.
>    * Pass rate: assuming the test is 100% reliable on RI we can
> calculate a percentage of failures.
>    * Number of times the test can be executed sequentionally before a
> fail.
>    * Memory consumption: a generator can preallocate more and more
> memory before launching the test in a loop.
>    * Max threads supported: a generator can exponentially increase
> number of threads launching the test in parallel.
>    * Here is your metric.
>    * Execution time: have you noticed that all this apparatus is quite
> close to performance testing methodology? There is no need to compete
> with them in their field though. :-)
> 
> The thing I like most about this approach is it can be introduced on a
> stress test generator level.
> 
> With best regards,
> Alexei Fedotov,
> Intel Middleware Products Division
> 
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> 
> 


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message