Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: neutral (asf.osuosl.org: 64.74.244.71 is neither permitted nor
 denied by domain of geir@pobox.com)
Message-ID: <43CB9294.3070703@pobox.com>
Date: Mon, 16 Jan 2006 07:33:24 -0500
From: Geir Magnusson Jr <geir@pobox.com>
Reply-To: geir@pobox.com
User-Agent: Thunderbird 1.5 (Windows/20051201)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: Unit test code in HARMONY-16
References: <6694B22B6436BC43B429958787E45498011D7A78@mssmsx402nb>
	 <43CB81DE.4020203@pobox.com>
 <906dd82e0601160404t39b7cc0bx599344af71a3280d@mail.gmail.com>
 <43CB8FEE.1010906@gmail.com>
In-Reply-To: <43CB8FEE.1010906@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

George,

Thanks for the patch :)

(hint, hint)

geir


Tim Ellison wrote:
> at the risk of getting dragged into this ;-)
> 
> I think George's objection is to the extension mechanism being used
> (i.e. a custom subclass rather than a standard JUnit decoration [1]).
> 
> Adding more and more subclasses is not a practical way to go if you
> think of all the different ways you may choose to extend the framework's
> behaviour.
> 
> Regards,
> Tim
> 
> [1] http://junit.sourceforge.net/javadoc/junit/extensions/TestDecorator.html
> 
> Mikhail Loenko wrote:
>> Sorry, but I cannot catch what is the problem.
>> If some output is annoying please send the test name and what is wrong with it.
>> PerformanceTest class does not add any output to what the tests print.
>>
>> For now the only implication from the fact that all the tests extend
>> PerfromanceTest
>> is that the security classes are well performance-tuned and on the majority of
>> scenarios outperform "standard" Java implementations.
>>
>> Thanks,
>> Mikhail Loenko
>> Intel Middleware Products Division
>>
>> On 1/16/06, Geir Magnusson Jr <geir@pobox.com> wrote:
>>
>>> Loenko, Mikhail Y wrote:
>>>
>>>> Hi George
>>>>
>>>> We will remove all the tests and these QA guys will never disturb us :)
>>>>
>>>> Every time we remove a test we leave something untested.
>>>>
>>>> For example, SerializationTest is a base for all the tests that check
>>>> serialization compatibility, and if we remove it because we do not
>>>> think about serialization right now we will lose that compatibility.
>>>>
>>>> The same for performance, it is of importance still. Existing test suite
>>>> allows us seeking performance regressions. Otherwise we will have
>>>> to keep in sync two parallel test suites - one PerformanceTest based
>>>> just for testing the performance and another one - for unit testing.
>>> I love this subject - unit tests an performance tests are different, as
>>> can be regression tests, although one might argue that many of the
>>> so-called regression tests are just unit tests that you forgot to do in
>>> the first place.
>>>
>>>
>>>> So, what is the noise? GUI is thinking that those base classes are
>>>> tests?
>>>> Maybe it makes sense to rename PerformanceTest to e.g. PerformanceTost
>>>> and GUI will be happy?
>>> No. :/
>>>
>>> Can we change the output to be meaningful?  I remember a while back on
>>> another project that I knew the timings of things on my machine just
>> >from practice, and I could tell when we changed something that impacted
>>> performance, because the tests ran longer.  This helped catch problems
>>> early.
>>>
>>> geir
>>>
>>>
>>>
>>>> Thanks,
>>>> Mikhail Loenko
>>>> Intel Middleware Products Division
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: George Harley1 [mailto:GHARLEY@uk.ibm.com]
>>>>> Sent: Friday, January 13, 2006 7:24 PM
>>>>> To: harmony-dev@incubator.apache.org
>>>>> Subject: RE: Unit test code in HARMONY-16
>>>>>
>>>>> Hi Mikhail,
>>>>>
>>>>> Thanks for your very complete answer.
>>>>>
>>>>>
>>>>>> At some point we had different functionality in the
>>>>>> PerformanceTest but it seems to have died now. That is basically it.
>>>>> Do you see this class (and its SecurityTest and SerializationTest
>>>>> subclasses) as candidates for removal then ? When I run the security
>>>> unit
>>>>
>>>>> tests inside my IDE they add some extra lines to the console output
>>>> but,
>>>>
>>>>> since I am not thinking about performance right now, that is just
>>>>> "background noise".
>>>>>
>>>>> Perhaps additional performance-related functionality would be better
>>>> moved
>>>>
>>>>> out of the test class' hierarchy and into some decorator class ? That
>>>> way
>>>>
>>>>> would give developers a bit more flexibility running the tests with or
>>>>> without the intervention of the performance measurement code. Sound
>>>>> reasonable ?
>>>>>
>>>>> Best regards,
>>>>> George
>>>>> ________________________________________
>>>>> George C. Harley
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> "Loenko, Mikhail Y" <mikhail.y.loenko@intel.com>
>>>>> 13/01/2006 12:12
>>>>> Please respond to
>>>>> harmony-dev@incubator.apache.org
>>>>>
>>>>>
>>>>> To
>>>>> <harmony-dev@incubator.apache.org>
>>>>> cc
>>>>>
>>>>> Subject
>>>>> RE: Unit test code in HARMONY-16
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> As far as we did not have special performance tests, we used unit tests
>>>>> to measure performance, i.e. to compare performance of our classes to
>>>>> performance of "standard" classes. So we ran in cycle a single unit
>>>> test
>>>>
>>>>> on both when there are our security classes in bootclasspath and when
>>>>> there are not. And compared time. (Of course, not all the tests passed
>>>>> on "RI")
>>>>>
>>>>> Some unit tests print various logs that make execution time volatile.
>>>> To
>>>>
>>>>> make it more stable we used log() instead of System.out.print() and in
>>>>> the "performance mode" did not print anything. log() is defined in the
>>>>> PerformanceTest. At some point we had different functionality in the
>>>>> PerformanceTest but it seems to have died now. That is basically it.
>>>>>
>>>>> The results helped us to find a number of performance leaks and improve
>>>>> overall quality of the code.
>>>>>
>>>>> Thanks,
>>>>> Mikhail Loenko
>>>>> Intel Middleware Products Division
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: George Harley1 [mailto:GHARLEY@uk.ibm.com]
>>>>>> Sent: Thursday, January 12, 2006 10:05 PM
>>>>>> To: harmony-dev@incubator.apache.org
>>>>>> Subject: Unit test code in HARMONY-16
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have been looking into running the JUnit tests included in
>>>> HARMONY-16
>>>>
>>>>> in
>>>>>
>>>>>> my private sandbox. From what I have seen so far most (all ?) of the
>>>>> test
>>>>>
>>>>>> cases inherit from a base class PerformanceTest in the
>>>>>> com.openintel.drl.security.test package. What is the purpose of this
>>>>> base
>>>>>
>>>>>> class ?
>>>>>>
>>>>>> Best regards,
>>>>>> George
>>>>>> ________________________________________
>>>>>> George C. Harley
>>>>
>