Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of mloenko@gmail.com designates
 66.249.92.202 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=RnbAwCyA+pvVw5Zg7rpwnAp/EGhBRPbwz8Od6kdBaV6461xQI7Csp2XRRCZhy0vUsddkd/ZdMdVoR59cdX0TZAq/mKtpOPxXX6Rrt+UqQ99oodQnrdAwNMkd2q9g4dLjI0GpBPFMW/z7GYMuXxQAaRkEnqykeiryQQyq+98bvDw=
Message-ID: <906dd82e0601180906g6b45f614r83ead3f12101f2c@mail.gmail.com>
Date: Wed, 18 Jan 2006 23:06:41 +0600
From: Mikhail Loenko <mloenko@gmail.com>
To: harmony-dev@incubator.apache.org
Subject: Re: [jira] Commented: (HARMONY-31) Move peformance timing of unit
 tests into a decorator class.
In-Reply-To: 
 <OFE994374B.459377C2-ON802570FA.00484F1E-802570FA.004B727C@uk.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
References: <906dd82e0601180457s6c7616cdr3cb95a5eafeb12eb@mail.gmail.com>
	 <OFE994374B.459377C2-ON802570FA.00484F1E-802570FA.004B727C@uk.ibm.com>

Hi George

When we run the tests (those ones that we call unit tests :) in performance=
 mode
we start a timer, then run the same testcase in a loop millions of
times (and we do
NOT call them via reflection) then we stop a timer

As I understand your point, you are going to measure a single call of
a test method and that call is over reflection, or multiple calls but
each of them over reflection.
Just wanted to say that reflection part of the call might take 99% of metho=
d
execution and results would not be reliable. Please correct me if I wrong.

Thanks,
Mikhail

On 1/18/06, George Harley1 <GHARLEY@uk.ibm.com> wrote:
> Hi Mikhail,
>
> > The messages are important to analyze failures also.
>
> What is JUnit's own failure reporting mechanism not providing to you ?
>
>
> > And the possibility to test perfromance is also important
>
> Yes it is. But - to return to the original point of the HARMONY-31 JIRA
> issue - this can be done without the need to either bring in a superclass
> in the test hierarchy and/or scatter logging calls around the test code.
> Performance testing using JUnit should be done in a transparent manner
> using a decorator. As an example of what I mean please take a look at the
> JUnitPerf site at http://clarkware.com/software/JUnitPerf.html . JUnitPer=
f
> is an extension to JUnit that helps with the creation of performance test=
s
> that match existing unit tests. It does this using a decorator design.
> This provides for a separation of concerns that benefits developers and
> performance engineers.
>
> The decorator approach means that we effectively *extend* JUnit with
> simple wrapper classes in which we can make full use of the JUnit API to
> give us the additional behaviour needed when running the tests (maybe thi=
s
> could be used to give whatever extra failure analysis data you say is
> lacking ?). And if somebody doesn't want the extension behaviour ? They
> just run the suite without the custom wrappers.
>
>
> > When we have a powerful performance suite we may revisit this.
>
> Don't you mean when we have a powerful unit test suite ?
>
> Take care,
> George
> ________________________________________
> George C. Harley
>
>
>
>
>
> Mikhail Loenko <mloenko@gmail.com>
> 18/01/2006 12:57
> Please respond to
> harmony-dev@incubator.apache.org
>
>
> To
> harmony-dev@incubator.apache.org
> cc
>
> Subject
> Re: [jira] Commented: (HARMONY-31) Move peformance timing of unit tests
> into a decorator class.
>
>
>
>
>
>
> Well, those messages were important for developers when
> they were writing code and tests. Then tests came to repository 'as is'.
>
> The messages are important to analyze failures also.
> And the possibility to test perfromance is also important
>
> For me any option that does not break functionality in favor of beauty
> looks
> good. I suggest swithcing off logging by default and ether keep
> PerformanceTest
> super class as is or replace it with a simple Logger class.
>
> When we have a powerful performance suite we may revisit this.
>
> Thanks,
> Mikhail.
>
> On 1/18/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> > Absolutely right -- writing meaningful performance tests is hard.
> > Implementing your own Logger would not solve the problem though<g>.
> >
> > Best to avoid the 'This test worked OK' log messages altogether, and
> > stick to assertions.
> >
> > Regards,
> > Tim
> >
> > Mikhail Loenko wrote:
> > > It might be a problem...
> > >
> > > When we use java.util.logging we do not just compare performance of
> security
> > > API functions, the result is also depends on difference in performanc=
e
> of
> > > java.util.logging in standard classes vs. Harmony classes. So if we
> use
> > > non-trivial functionality from there then our results will be spoiled
> a little.
> > >
> > > Will investigate more...
> > >
> > > Thanks,
> > > Mikhail.
> > >
> > > On 1/17/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> > >> neither is the Logger class -- so my point is if you are going to
> write
> > >> some logging code why not do it in java.util.logging?  You may choos=
e
> to
> > >> only do simple stubs for now until somebody steps up to do a real
> impl.
> > >>
> > >> Regards,
> > >> Tim
> > >>
> > >> Mikhail Loenko wrote:
> > >>> It's not yet implemented.
> > >>>
> > >>> thanks,
> > >>> Mikhail
> > >>>
> > >>> On 1/17/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> > >>>> Why not use java.util.logging?
> > >>>>
> > >>>> Regards,
> > >>>> Tim
> > >>>>
> > >>>> Mikhail Loenko (JIRA) wrote:
> > >>>>>     [
> http://issues.apache.org/jira/browse/HARMONY-31?page=3Dcomments#action_12=
362910
> ]
> > >>>>>
> > >>>>> Mikhail Loenko commented on HARMONY-31:
> > >>>>> ---------------------------------------
> > >>>>>
> > >>>>> This is not what I meant.
> > >>>>>
> > >>>>> I was going to create a Logger class at this point like this:
> > >>>>>
> > >>>>> public class Logger {
> > >>>>>         public static boolean printAllowed =3D false;
> > >>>>>       public static void log(String message) {
> > >>>>>               if (printAllowed) System.out.print(message);
> > >>>>>       }
> > >>>>>       public static void logln(String message) {
> > >>>>>               if (printAllowed) System.out.println(message);
> > >>>>>       }
> > >>>>>       public static void logError(String message) {
> > >>>>>               if (printAllowed) System.err.print(message);
> > >>>>>       }
> > >>>>>       public static void loglnError(String message) {
> > >>>>>               if (printAllowed) System.err.println(message);
> > >>>>>       }
> > >>>>> }
> > >>>>>
> > >>>>> And replace log() with Logger.log() everywhere in the tests.
> > >>>>>
> > >>>>> All the remaining functionality in the PerformanceTest is
> obsolete.
> > >>>>>
> > >>>>>
> > >>>>>> Move peformance timing of unit tests into a decorator class.
> > >>>>>> ------------------------------------------------------------
> > >>>>>>
> > >>>>>>          Key: HARMONY-31
> > >>>>>>          URL: http://issues.apache.org/jira/browse/HARMONY-31
> > >>>>>>      Project: Harmony
> > >>>>>>         Type: Improvement
> > >>>>>>     Reporter: George Harley
> > >>>>>>     Assignee: Geir Magnusson Jr
> > >>>>>>     Priority: Minor
> > >>>>>>  Attachments: PerfDecorator.java
> > >>>>>>
> > >>>>>> There has been some low-level discussion on the dev mailing list
> recently about the inclusion of performance-related logging code near the
> top of a unit test class inheritance hierarchy (see
> com.openintel.drl.security.test.PerformanceTest in the HARMONY-16
> contribution). This particular issue suggests an alternative way of addin=
g
> in timing code but without making it the responsibility of the unit tests
> themselves and without the need to introduce a class in the inheritance
> hierarchy.
> > >>>>>> The basic approach is to exploit the
> junit.extensions.TestDecorator type in the JUnit API to add in timing
> behaviour before and after each test method runs. This will be
> demonstrated with some simple sample code.
> > >>>> --
> > >>>>
> > >>>> Tim Ellison (t.p.ellison@gmail.com)
> > >>>> IBM Java technology centre, UK.
> > >>>>
> > >> --
> > >>
> > >> Tim Ellison (t.p.ellison@gmail.com)
> > >> IBM Java technology centre, UK.
> > >>
> > >
> >
> > --
> >
> > Tim Ellison (t.p.ellison@gmail.com)
> > IBM Java technology centre, UK.
> >
>
>
>