harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley1 <GHAR...@uk.ibm.com>
Subject Re: [jira] Commented: (HARMONY-31) Move peformance timing of unit tests into a decorator class.
Date Wed, 18 Jan 2006 21:54:24 GMT
Hi Mikhail, 


>  When we run the tests (those ones that we call unit tests :) in 
performance mode
> we start a timer, then run the same testcase in a loop millions of
> times (and we do NOT call them via reflection) then we stop a timer

Here is a flavour of what works for a lot of people...
 
Use the JUnit type RepeatedTest (see [1]) to wrap any concrete JUnit Test 
(e.g. a TestSuite, a TestCase etc) so that the test (or tests if it is a 
TestSuite) get run as many million times as you want. Then, take the very 
same RepeatedTest object and wrap that inside a JUnitPerf TimedTest (see 
[2]), also passing in a maximum elapsed time that the RepeatedTest has to 
run in. If the RepeatedTest (which is running the original test case or 
suite millions of times for you) doesn't complete in the specified maximum 
time then the test is a failure. 


> As I understand your point, you are going to measure a single call of
> a test method and that call is over reflection, or multiple calls but
> each of them over reflection.

No, that was not my point. My point is that, at the unit test level, it is 
possible to carry out performance timings using *established* JUnit 
techniques that are *dynamic* in nature and do *not* require adding in a 
superclass into the inheritance hierarchy meaning that every child class 
"is a" performance test spouting out lots of "== test Foo passed OK ==" 
messages. What happens if you want to carry out some stress testing with 
the unit tests ? Does it mean the addition of a new superclass at the top 
of the hierarchy so that every unit test class "is a" stress test as well 
? And so on for interop tests ... etc .. etc ... ?


> Just wanted to say that reflection part of the call might take 99% of 
method
> execution and results would not be reliable. Please correct me if I 
wrong.

You could adjust your target timings to take into account the effects of 
the test framework. What would be the problem in doing that ? If you are 
looking for the introduction of performance problems then isn't it the 
relative timings (e.g. "before versus after the introduction of new code" 
or "test on JRE1 versus identical test on JRE2") that matter ? 


Best regards, 
George

[1] 
http://www.junit.org/junit/javadoc/3.8.1/junit/extensions/RepeatedTest.html
[2] http://clarkware.com/software/JUnitPerf.html#howtouse
________________________________________
George C. Harley





Mikhail Loenko <mloenko@gmail.com> 
18/01/2006 17:06
Please respond to
harmony-dev@incubator.apache.org


To
harmony-dev@incubator.apache.org
cc

Subject
Re: [jira] Commented: (HARMONY-31) Move peformance timing of unit tests 
into a decorator class.






Hi George

When we run the tests (those ones that we call unit tests :) in 
performance mode
we start a timer, then run the same testcase in a loop millions of
times (and we do
NOT call them via reflection) then we stop a timer

As I understand your point, you are going to measure a single call of
a test method and that call is over reflection, or multiple calls but
each of them over reflection.
Just wanted to say that reflection part of the call might take 99% of 
method
execution and results would not be reliable. Please correct me if I wrong.

Thanks,
Mikhail

On 1/18/06, George Harley1 <GHARLEY@uk.ibm.com> wrote:
> Hi Mikhail,
>
> > The messages are important to analyze failures also.
>
> What is JUnit's own failure reporting mechanism not providing to you ?
>
>
> > And the possibility to test perfromance is also important
>
> Yes it is. But - to return to the original point of the HARMONY-31 JIRA
> issue - this can be done without the need to either bring in a 
superclass
> in the test hierarchy and/or scatter logging calls around the test code.
> Performance testing using JUnit should be done in a transparent manner
> using a decorator. As an example of what I mean please take a look at 
the
> JUnitPerf site at http://clarkware.com/software/JUnitPerf.html . 
JUnitPerf
> is an extension to JUnit that helps with the creation of performance 
tests
> that match existing unit tests. It does this using a decorator design.
> This provides for a separation of concerns that benefits developers and
> performance engineers.
>
> The decorator approach means that we effectively *extend* JUnit with
> simple wrapper classes in which we can make full use of the JUnit API to
> give us the additional behaviour needed when running the tests (maybe 
this
> could be used to give whatever extra failure analysis data you say is
> lacking ?). And if somebody doesn't want the extension behaviour ? They
> just run the suite without the custom wrappers.
>
>
> > When we have a powerful performance suite we may revisit this.
>
> Don't you mean when we have a powerful unit test suite ?
>
> Take care,
> George
> ________________________________________
> George C. Harley
>
>
>
>
>
> Mikhail Loenko <mloenko@gmail.com>
> 18/01/2006 12:57
> Please respond to
> harmony-dev@incubator.apache.org
>
>
> To
> harmony-dev@incubator.apache.org
> cc
>
> Subject
> Re: [jira] Commented: (HARMONY-31) Move peformance timing of unit tests
> into a decorator class.
>
>
>
>
>
>
> Well, those messages were important for developers when
> they were writing code and tests. Then tests came to repository 'as is'.
>
> The messages are important to analyze failures also.
> And the possibility to test perfromance is also important
>
> For me any option that does not break functionality in favor of beauty
> looks
> good. I suggest swithcing off logging by default and ether keep
> PerformanceTest
> super class as is or replace it with a simple Logger class.
>
> When we have a powerful performance suite we may revisit this.
>
> Thanks,
> Mikhail.
>
> On 1/18/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> > Absolutely right -- writing meaningful performance tests is hard.
> > Implementing your own Logger would not solve the problem though<g>.
> >
> > Best to avoid the 'This test worked OK' log messages altogether, and
> > stick to assertions.
> >
> > Regards,
> > Tim
> >
> > Mikhail Loenko wrote:
> > > It might be a problem...
> > >
> > > When we use java.util.logging we do not just compare performance of
> security
> > > API functions, the result is also depends on difference in 
performance
> of
> > > java.util.logging in standard classes vs. Harmony classes. So if we
> use
> > > non-trivial functionality from there then our results will be 
spoiled
> a little.
> > >
> > > Will investigate more...
> > >
> > > Thanks,
> > > Mikhail.
> > >
> > > On 1/17/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> > >> neither is the Logger class -- so my point is if you are going to
> write
> > >> some logging code why not do it in java.util.logging?  You may 
choose
> to
> > >> only do simple stubs for now until somebody steps up to do a real
> impl.
> > >>
> > >> Regards,
> > >> Tim
> > >>
> > >> Mikhail Loenko wrote:
> > >>> It's not yet implemented.
> > >>>
> > >>> thanks,
> > >>> Mikhail
> > >>>
> > >>> On 1/17/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> > >>>> Why not use java.util.logging?
> > >>>>
> > >>>> Regards,
> > >>>> Tim
> > >>>>
> > >>>> Mikhail Loenko (JIRA) wrote:
> > >>>>>     [
> 
http://issues.apache.org/jira/browse/HARMONY-31?page=comments#action_12362910

> ]
> > >>>>>
> > >>>>> Mikhail Loenko commented on HARMONY-31:
> > >>>>> ---------------------------------------
> > >>>>>
> > >>>>> This is not what I meant.
> > >>>>>
> > >>>>> I was going to create a Logger class at this point like this:
> > >>>>>
> > >>>>> public class Logger {
> > >>>>>         public static boolean printAllowed = false;
> > >>>>>       public static void log(String message) {
> > >>>>>               if (printAllowed) System.out.print(message);
> > >>>>>       }
> > >>>>>       public static void logln(String message) {
> > >>>>>               if (printAllowed) System.out.println(message);
> > >>>>>       }
> > >>>>>       public static void logError(String message) {
> > >>>>>               if (printAllowed) System.err.print(message);
> > >>>>>       }
> > >>>>>       public static void loglnError(String message) {
> > >>>>>               if (printAllowed) System.err.println(message);
> > >>>>>       }
> > >>>>> }
> > >>>>>
> > >>>>> And replace log() with Logger.log() everywhere in the tests.
> > >>>>>
> > >>>>> All the remaining functionality in the PerformanceTest is
> obsolete.
> > >>>>>
> > >>>>>
> > >>>>>> Move peformance timing of unit tests into a decorator class.
> > >>>>>> ------------------------------------------------------------
> > >>>>>>
> > >>>>>>          Key: HARMONY-31
> > >>>>>>          URL: http://issues.apache.org/jira/browse/HARMONY-31
> > >>>>>>      Project: Harmony
> > >>>>>>         Type: Improvement
> > >>>>>>     Reporter: George Harley
> > >>>>>>     Assignee: Geir Magnusson Jr
> > >>>>>>     Priority: Minor
> > >>>>>>  Attachments: PerfDecorator.java
> > >>>>>>
> > >>>>>> There has been some low-level discussion on the dev mailing

list
> recently about the inclusion of performance-related logging code near 
the
> top of a unit test class inheritance hierarchy (see
> com.openintel.drl.security.test.PerformanceTest in the HARMONY-16
> contribution). This particular issue suggests an alternative way of 
adding
> in timing code but without making it the responsibility of the unit 
tests
> themselves and without the need to introduce a class in the inheritance
> hierarchy.
> > >>>>>> The basic approach is to exploit the
> junit.extensions.TestDecorator type in the JUnit API to add in timing
> behaviour before and after each test method runs. This will be
> demonstrated with some simple sample code.
> > >>>> --
> > >>>>
> > >>>> Tim Ellison (t.p.ellison@gmail.com)
> > >>>> IBM Java technology centre, UK.
> > >>>>
> > >> --
> > >>
> > >> Tim Ellison (t.p.ellison@gmail.com)
> > >> IBM Java technology centre, UK.
> > >>
> > >
> >
> > --
> >
> > Tim Ellison (t.p.ellison@gmail.com)
> > IBM Java technology centre, UK.
> >
>
>
>



Mime
View raw message