harmony-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mloe...@apache.org
Subject svn commit: r410246 - in /incubator/harmony/standard/site: docs/documentation/ docs/subcomponents/stresstest/ xdocs/documentation/ xdocs/subcomponents/stresstest/
Date Tue, 30 May 2006 11:47:09 GMT
Author: mloenko
Date: Tue May 30 04:47:08 2006
New Revision: 410246

URL: http://svn.apache.org/viewvc?rev=410246&view=rev
Log:
committing stress test proposal (HARMONY-511)

Added:
    incubator/harmony/standard/site/docs/subcomponents/stresstest/
    incubator/harmony/standard/site/docs/subcomponents/stresstest/index.html
    incubator/harmony/standard/site/xdocs/subcomponents/stresstest/
    incubator/harmony/standard/site/xdocs/subcomponents/stresstest/index.xml
Modified:
    incubator/harmony/standard/site/docs/documentation/documentation.html
    incubator/harmony/standard/site/xdocs/documentation/documentation.xml

Modified: incubator/harmony/standard/site/docs/documentation/documentation.html
URL: http://svn.apache.org/viewvc/incubator/harmony/standard/site/docs/documentation/documentation.html?rev=410246&r1=410245&r2=410246&view=diff
==============================================================================
--- incubator/harmony/standard/site/docs/documentation/documentation.html (original)
+++ incubator/harmony/standard/site/docs/documentation/documentation.html Tue May 30 04:47:08
2006
@@ -240,7 +240,12 @@
       <tr><td>
         <blockquote>
                                     <ul>
-  <li>The <a href="http://wiki.apache.org/harmony">Harmony Wiki</a></li>
+  <li>The <a href="http://wiki.apache.org/harmony">Harmony Wiki</a>
+  </li>
+  <li><a href="../subcomponents/stresstest/index.html">Proposal for the Stress
Test Framework</a>
+  </li>
+  <li><b>Note:</b> Other proposals can be found at <a href="../subcomponents/classlibrary/index.html">Classlibrary
page</a>
+  </li>
 </ul>
                             </blockquote>
       </td></tr>

Added: incubator/harmony/standard/site/docs/subcomponents/stresstest/index.html
URL: http://svn.apache.org/viewvc/incubator/harmony/standard/site/docs/subcomponents/stresstest/index.html?rev=410246&view=auto
==============================================================================
--- incubator/harmony/standard/site/docs/subcomponents/stresstest/index.html (added)
+++ incubator/harmony/standard/site/docs/subcomponents/stresstest/index.html Tue May 30 04:47:08
2006
@@ -0,0 +1,371 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+
+<!--
+Copyright 1999-2004 The Apache Software Foundation
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+
+<!-- Content Stylesheet for Site -->
+
+        
+<!-- start the processing -->
+    <!-- ====================================================================== -->
+    <!-- GENERATED FILE, DO NOT EDIT, EDIT THE XML FILE IN xdocs INSTEAD! -->
+    <!-- Main Page Section -->
+    <!-- ====================================================================== -->
+    <html>
+        <head>
+            <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
+
+                                                    <meta name="author" value="Harmony
Documentation Team">
+            <meta name="email" value="harmony-dev@incubator.apache.org">
+            
+           
+            
+            
+            
+            
+            
+            <title>Apache Harmony - Stress test framework</title>
+        </head>
+
+        <body bgcolor="#ffffff" text="#000000" link="#525D76">        
+            <table border="0" width="100%" cellspacing="0">
+                <!-- TOP IMAGE -->
+                <tr>
+                    <td align='LEFT'>
+                    <table><tr><td>
+                    <td align="left">
+<a href="http://incubator.apache.org/harmony/"><img  src="../../images/harmony-logo.png"
alt="Apache Harmony" border="0"/></a>
+</td>
+</td>
+                    </td></tr></table>
+                    </td>
+                    <td align='RIGHT'>
+                    <a href="http://incubator.apache.org/harmony/">
+                    <img src="http://incubator.apache.org/harmony/images/apache-incubator-logo.png"
+                    	alt="Apache Harmony" border="0"/></a>                    
+                    </td>
+                </tr>
+            </table>
+            <table border="0" width="100%" cellspacing="4">
+                <tr><td colspan="2">
+                    <hr noshade="" size="1"/>
+                </td></tr>
+
+                <tr>
+                    <!-- LEFT SIDE NAVIGATION -->
+                    <td width="20%" valign="top" nowrap="true">
+
+                    <!-- special ACon Logo - leave here for next time 
+                    <a href="http://apachecon.com/2005/US/">
+                        <img src="http://apache.org/images/ac2005us_blue_125x125.jpg"
height="125"
+                             width="125" border="0" alt="ApacheCon US 2005" />
+                    </a>
+                    -->
+
+                   <!-- regular menu -->
+
+                    
+    <!-- ============================================================ -->
+
+                <p><strong>General</strong></p>
+        <ul>
+                    <li>    <a href="../../index.html">Home</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../license.html">License</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../contribution_policy.html">Contribution
Policy</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../guidelines.html">Project Guidelines</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="http://www.apache.org/">ASF</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../downloads.html">Downloads</a>
+</li>
+            
+            <table>
+                        </table>
+                </ul>
+            <p><strong>Community</strong></p>
+        <ul>
+                    <li>    <a href="../../get-involved.html">Get Involved</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../contributors.html">Committers</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../mailing.html">Mailing Lists</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../documentation/documentation.html">Documentation</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../faq.html">FAQ</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="http://wiki.apache.org/harmony">Wiki</a>
+</li>
+            
+            <table>
+                        </table>
+                </ul>
+            <p><strong>Development</strong></p>
+        <ul>
+                    <li>    <a href="../../status.html">How Are We Doing?</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../roadmap.html">Road Map / TODO</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../svn.html">Source Code</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="http://issues.apache.org/jira/browse/HARMONY">JIRA</a>
+</li>
+            
+            <table>
+                        </table>
+                    <li>    <a href="../../related.html">Other Projects</a>
+</li>
+            
+            <table>
+                        </table>
+                </ul>
+            <p><strong>Subcomponents</strong></p>
+        <ul>
+                    <li>    <a href="../../subcomponents/classlibrary/index.html">Class
Library</a>
+</li>
+            
+            <table>
+                          <tr><td>
+                &nbsp;&nbsp;    <a href="../../subcomponents/classlibrary/status.html">Status</a>
+              </td></tr>
+                        </table>
+                    <li>    <a href="../../subcomponents/jchevm/index.html">Virtual
Machine JCHEVM</a>
+</li>
+            
+            <table>
+                        </table>
+                </ul>
+                        </td>
+                    <td width="80%" align="left" valign="top">
+                                                                    <table border="0"
cellspacing="0" cellpadding="2" width="100%">
+      <tr><td bgcolor="#525D76">
+        <font color="#ffffff" face="arial,helvetica,sanserif">
+          <a name="Test Design"><strong>Test Design</strong></a>
+        </font>
+      </td></tr>
+      <tr><td>
+        <blockquote>
+                                    <p>
+		<big><em>The following guidelines are currently PROPOSED and being discussed
on the
+		development mailing list <code>harmony-dev@incubator.apache.org</code>.  Please
+		direct comments and questions there.</em></big>
+	</p>
+                                                <P>
+<UL>
+<LI>Stress tests are built from simple building blocks according to configuration strings.
+</LI>
+<LI>Tests have junit interface.
+<BLOCKQUOTE>
+       [Case study] Imagine someone puts tests into SVN which implements
+different test interface. To reuse them we can add another generator to
+ convert these tests to junit interface.
+</BLOCKQUOTE>
+</LI>
+<LI>Configuration string list is maintained manually. If we plan to use
+ junit runner to launch a sequent of the stress tests, then the most
+ straightforward model is to wrap configuration strings into junit test
+cases and put documentation into javadoc for these test cases.
+</LI>
+</UL>
+</P>
+                            </blockquote>
+        </p>
+      </td></tr>
+      <tr><td><br/></td></tr>
+    </table>
+                                                <table border="0" cellspacing="0" cellpadding="2"
width="100%">
+      <tr><td bgcolor="#525D76">
+        <font color="#ffffff" face="arial,helvetica,sanserif">
+          <a name="Further Steps"><strong>Further Steps</strong></a>
+        </font>
+      </td></tr>
+      <tr><td>
+        <blockquote>
+                                    <P>
+<UL>
+<LI>Stress tests are expected to generate relevant bugs. Since
+usually stress behavior is unspecified, we need to introduce something
+measurable instead of pass/fail result for the stress tests. See 
+<a href="#Comparative Approach">comparative approach</a> below.
+</LI>
+<LI>All should create tests and run them against Harmony VM and RI.
+This would be a real-life testing for our approach.
+</LI>
+</UL>
+</P>
+                            </blockquote>
+        </p>
+      </td></tr>
+      <tr><td><br/></td></tr>
+    </table>
+                                                <table border="0" cellspacing="0" cellpadding="2"
width="100%">
+      <tr><td bgcolor="#525D76">
+        <font color="#ffffff" face="arial,helvetica,sanserif">
+          <a name="Comparative Approach"><strong>Comparative Approach</strong></a>
+        </font>
+      </td></tr>
+      <tr><td>
+        <blockquote>
+                                    <P>
+ The simplest example of comparative apporach is the following.
+</P>
+                                                <BLOCKQUOTE>
+ 	Tester: My test fails on Harmony VM and passes on RI. Please,
+ fix Harmony VM.
+</BLOCKQUOTE>
+                                                <P>
+ This usually does not work for stress tests.
+</P>
+                                                <BLOCKQUOTE>
+ 	Developer: Who told you that OutOfMemoryError should be thrown
+ in your thread? My finalizer thread is just a normal java thread, like
+ yours, and it can fail as well. You have a bug in your test. 
+</BLOCKQUOTE>
+                                                <P>
+ 
+ There are multiple reasons why we always will have such bugs in the
+ tests.
+</P>
+                                                <UL>
+<LI>These bugs keep showing up. The time to fix all these bugs
+ regularly is too high.
+</LI>
+<LI>Stress testing reuses tests which are usually not designed for
+ stress execution, for example, multithread execution.
+</LI>
+<LI>These bugs are dependent on VM internal structure. Test authors do
+ not posess sufficient knowledge of the problem and the structure.
+</LI>
+<LI>Sometimes Java is not rich enough.
+</LI>
+</UL>
+                                                <P>
+ How can we have a maintainable test product takung all this limitation
+ into account? We need to learn how to live with occasional failures of
+ the stress tests. This means, instead of fail, the test should better
+ report how good it is on Harmony VM compared to RI:
+</P>
+                                                <UL>
+<LI>Failures with the worst relative metric can be evaluated first</LI>
+<LI>We can detect that a relative metric for a test worsened on the recent build</LI>
+</UL>
+                                                <P>
+Developers are better convinsed to fix "the worst issue" or "dergadation" instead of "some
issue".
+</P>
+                                                <P>
+
+Several metrics for each test:
+<UL>
+<LI>Pass rate: assuming the test is 100% reliable on RI we can
+ calculate a percentage of failures.
+</LI>
+<LI>Number of times the test can be executed sequentionally before a fail
+</LI>
+<LI>Memory consumption: a generator can preallocate more and more
+ memory before launching the test in a loop.
+</LI>
+<LI>Max threads supported: a generator can exponentially increase
+ number of threads launching the test in parallel.
+</LI>
+<LI>Execution time: all this apparatus is quite
+ close to performance testing methodology. There is no need to compete
+ with them in their field though.
+</LI>
+</UL>
+</P>
+                            </blockquote>
+        </p>
+      </td></tr>
+      <tr><td><br/></td></tr>
+    </table>
+                                        </td>
+                </tr>
+
+                <!-- FOOTER -->
+                <tr><td colspan="2">
+                    <hr noshade="" size="1"/>
+                </td></tr>
+                <tr><td colspan="2">
+                    <div align="center"><font color="#525D76" size="-1"><em>
+                    Copyright &#169; 2003-2006, The Apache Software Foundation
+                    </em></font></div>
+                </td></tr>
+            </table>
+        </body>
+    </html>
+<!-- end the processing -->
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

Modified: incubator/harmony/standard/site/xdocs/documentation/documentation.xml
URL: http://svn.apache.org/viewvc/incubator/harmony/standard/site/xdocs/documentation/documentation.xml?rev=410246&r1=410245&r2=410246&view=diff
==============================================================================
--- incubator/harmony/standard/site/xdocs/documentation/documentation.xml (original)
+++ incubator/harmony/standard/site/xdocs/documentation/documentation.xml Tue May 30 04:47:08
2006
@@ -1,80 +1,86 @@
-<?xml version="1.0" encoding="ISO-8859-1"?>
-
-<!--
-
-    Copyright 2004 The Apache Software Foundation
-
-    Licensed under the Apache License, Version 2.0 (the "License");
-    you may not use this file except in compliance with the License.
-    You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-    Unless required by applicable law or agreed to in writing, software
-    distributed under the License is distributed on an "AS IS" BASIS,
-    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-    See the License for the specific language governing permissions and
-    limitations under the License.
--->
-
-
-<document>
-
- <properties>
-  <title>Apache Harmony</title>
-  <author email="harmony-dev@harmony.apache.org">Harmony Documentation Team</author>
- </properties>
-
- <body>
-<section name="Apache Harmony Documentation">
-
-<p>
-There is not yet a full manual or similar official documentation.  However,
-the resources below will help new users and developers get up to speed with
-the technology.
-</p>
-
-<subsection name="Project Documentation">
-
-<ul>
-  <li>
-    <a href="build_website.html">Building and Deploying the Apache Harmony Website</a>
-  </li>
-  <li>
-    <a href="../subcomponents/classlibrary/index.html">Documentation for the Classlibrary
Code</a>
-  </li>
-  <li>
-      <a href="../externals/security/ASN1Framework.htm">Documentation for Harmony's
ASN1 Framework</a>
-  </li>
-  <li>
-      <a href="../externals/regex/Regexp.htm">Documentation for Harmony's Regular Expression
Framework</a>      
-  </li>
-</ul>
-
-</subsection>
-
-<subsection name="Project Resource">
-
-<ul>
-  <li>The <a href="http://wiki.apache.org/harmony">Harmony Wiki</a></li>
-</ul>
-
-</subsection>
-
-<subsection name="Books">
-</subsection>
-
-    <subsection name="Research Papers and Talks">
-
-    </subsection>
-
-<subsection name="Articles and Interviews">
-
-</subsection>
-
-</section>
-
-</body>
-
-</document>
-
+<?xml version="1.0" encoding="ISO-8859-1"?>
+
+<!--
+
+    Copyright 2004 The Apache Software Foundation
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+-->
+
+
+<document>
+
+ <properties>
+  <title>Apache Harmony</title>
+  <author email="harmony-dev@harmony.apache.org">Harmony Documentation Team</author>
+ </properties>
+
+ <body>
+<section name="Apache Harmony Documentation">
+
+<p>
+There is not yet a full manual or similar official documentation.  However,
+the resources below will help new users and developers get up to speed with
+the technology.
+</p>
+
+<subsection name="Project Documentation">
+
+<ul>
+  <li>
+    <a href="build_website.html">Building and Deploying the Apache Harmony Website</a>
+  </li>
+  <li>
+    <a href="../subcomponents/classlibrary/index.html">Documentation for the Classlibrary
Code</a>
+  </li>
+  <li>
+      <a href="../externals/security/ASN1Framework.htm">Documentation for Harmony's
ASN1 Framework</a>
+  </li>
+  <li>
+      <a href="../externals/regex/Regexp.htm">Documentation for Harmony's Regular Expression
Framework</a>      
+  </li>
+</ul>
+
+</subsection>
+
+<subsection name="Project Resource">
+
+<ul>
+  <li>The <a href="http://wiki.apache.org/harmony">Harmony Wiki</a>
+  </li>
+  <li><a href="../subcomponents/stresstest/index.html">Proposal for the Stress
Test Framework</a>
+  </li>
+  <li><b>Note:</b> Other proposals can be found at <a href="../subcomponents/classlibrary/index.html">Classlibrary
page</a>
+  </li>
+</ul>
+
+
+</subsection>
+
+<subsection name="Books">
+</subsection>
+
+    <subsection name="Research Papers and Talks">
+
+    </subsection>
+
+<subsection name="Articles and Interviews">
+
+</subsection>
+
+</section>
+
+</body>
+
+</document>
+

Added: incubator/harmony/standard/site/xdocs/subcomponents/stresstest/index.xml
URL: http://svn.apache.org/viewvc/incubator/harmony/standard/site/xdocs/subcomponents/stresstest/index.xml?rev=410246&view=auto
==============================================================================
--- incubator/harmony/standard/site/xdocs/subcomponents/stresstest/index.xml (added)
+++ incubator/harmony/standard/site/xdocs/subcomponents/stresstest/index.xml Tue May 30 04:47:08
2006
@@ -0,0 +1,145 @@
+<?xml version="1.0" encoding="ISO-8859-1"?>
+
+<!--
+
+    Copyright 2006 The Apache Software Foundation
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+-->
+
+<document>
+
+<properties>
+	<title>Stress test framework</title>
+	<author email="harmony-dev@incubator.apache.org">Harmony Documentation Team</author>
+</properties>
+
+<body>
+
+<section name="Test Design">
+	<p>
+		<big><em>The following guidelines are currently PROPOSED and being discussed
on the
+		development mailing list <code>harmony-dev@incubator.apache.org</code>.  Please
+		direct comments and questions there.</em></big>
+	</p>
+<P>
+<UL>
+<LI>Stress tests are built from simple building blocks according to configuration strings.
+</LI>
+<LI>Tests have junit interface.
+<BLOCKQUOTE>
+       [Case study] Imagine someone puts tests into SVN which implements
+different test interface. To reuse them we can add another generator to
+ convert these tests to junit interface.
+</BLOCKQUOTE>
+</LI>
+<LI>Configuration string list is maintained manually. If we plan to use
+ junit runner to launch a sequent of the stress tests, then the most
+ straightforward model is to wrap configuration strings into junit test
+cases and put documentation into javadoc for these test cases.
+</LI>
+</UL>
+</P>
+</section>
+
+<section name="Further Steps">
+<P>
+<UL>
+<LI>Stress tests are expected to generate relevant bugs. Since
+usually stress behavior is unspecified, we need to introduce something
+measurable instead of pass/fail result for the stress tests. See 
+<a href="#Comparative Approach">comparative approach</a> below.
+</LI>
+<LI>All should create tests and run them against Harmony VM and RI.
+This would be a real-life testing for our approach.
+</LI>
+</UL>
+</P>
+</section>
+
+<section name="Comparative Approach">
+<P>
+ The simplest example of comparative apporach is the following.
+</P>
+<BLOCKQUOTE>
+ 	Tester: My test fails on Harmony VM and passes on RI. Please,
+ fix Harmony VM.
+</BLOCKQUOTE>
+<P>
+ This usually does not work for stress tests.
+</P>
+<BLOCKQUOTE>
+ 	Developer: Who told you that OutOfMemoryError should be thrown
+ in your thread? My finalizer thread is just a normal java thread, like
+ yours, and it can fail as well. You have a bug in your test. 
+</BLOCKQUOTE>
+<P>
+ 
+ There are multiple reasons why we always will have such bugs in the
+ tests.
+</P>
+<UL>
+<LI>These bugs keep showing up. The time to fix all these bugs
+ regularly is too high.
+</LI>
+<LI>Stress testing reuses tests which are usually not designed for
+ stress execution, for example, multithread execution.
+</LI>
+<LI>These bugs are dependent on VM internal structure. Test authors do
+ not posess sufficient knowledge of the problem and the structure.
+</LI>
+<LI>Sometimes Java is not rich enough.
+</LI>
+</UL>
+ 
+<P>
+ How can we have a maintainable test product takung all this limitation
+ into account? We need to learn how to live with occasional failures of
+ the stress tests. This means, instead of fail, the test should better
+ report how good it is on Harmony VM compared to RI:
+</P>
+<UL>
+<LI>Failures with the worst relative metric can be evaluated first</LI>
+<LI>We can detect that a relative metric for a test worsened on the recent build</LI>
+</UL>
+ 
+<P>
+Developers are better convinsed to fix "the worst issue" or "dergadation" instead of "some
issue".
+</P>
+<P>
+
+Several metrics for each test:
+<UL>
+<LI>Pass rate: assuming the test is 100% reliable on RI we can
+ calculate a percentage of failures.
+</LI>
+<LI>Number of times the test can be executed sequentionally before a fail
+</LI>
+<LI>Memory consumption: a generator can preallocate more and more
+ memory before launching the test in a loop.
+</LI>
+<LI>Max threads supported: a generator can exponentially increase
+ number of threads launching the test in parallel.
+</LI>
+<LI>Execution time: all this apparatus is quite
+ close to performance testing methodology. There is no need to compete
+ with them in their field though.
+</LI>
+</UL>
+</P>
+
+</section>
+
+
+</body>
+</document>



Mime
View raw message