commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phil Steitz <>
Subject Re: [Math] Simple benchmarking utility
Date Fri, 29 Jul 2011 16:08:46 GMT
On 7/29/11 3:23 AM, Gilles Sadowski wrote:
> Hello.
>>>>>>> [...]
>>>>>>> The idea is to have "interleaved" calls to the candidate
>>>>>>> implementations, so
>>>>>>> that (hopefully) they will be penalized (or benefit) in the same
>>>>>>> way
>>>>>>> by what
>>>>>>> the JVM is doing (GC or JIT compilation or ...) while the
>>>>>>> benchmark
>>>>>>> is
>>>>>>> running.
>>>>>>> Does this make sense?
>>>>>> Could it be merged by the FastMath performance tests Sebb set up
>>>>> I don't think so. If you meant rewriting the
>>>>> "FastMathTestPerformance" tests
>>>>> using the proposed utility, I don't think that it is necessary.
>>>> This was what I meant.
>>>> If this feature is not used in any existing tests, perhaps it should go in
>>>> some other directory. Perhaps a new "utilities" or something like that, at
>>>> same level as "main" and "test" ?
>>>> Anyway, if you feel it's useful to have this available around, don't hesitate.
>>> Well, the first use I had in mind was to provide a agreed on way to base a
>>> discussion for requests such as "CM's implementation of foo is not
>>> efficient", and avoid wondering how the reporter got his results. [This
>>> problem occurs for the MATH-628 issue.]. Then, when the problem reported
>>> is confirmed, the new implementation will replace the less efficient one in
>>> CM, so that there won't be any alternative implementation left to compare.
>>> If you agree with the idea of a "standard" benchmark, it would be very much
>>> necessary that several people have a look at the code: It might be that my
>>> crude "methodology" is not right, or that there is a bug.
>>> If the code is accepted, then we'll decide where to put it. Even if,
>>> according to the above, its primary use will not be for long-lived unit
>>> tests, it might still be useful in order to compare the efficiency of CM's
>>> algorithms such as the various optimizers. These comparisons could be added
>>> as performance reports similar to "FastMathTestPerformance".
>> +1 to include it.  I would say start by putting it in a top level
>> package of its own - say, "benchmark" in src/test/java.  That way we
>> can use it in test classes or experimentation that we do using test
>> classes to set up benchmarks.
> Isn't it "safer" to put it in package "o.a.c.m" (under "src/test/java")?
> I was thinking of "PerfTestUtils" for the class name.

What I meant was o.a.c.m.benchmark, but I would be fine including it
at the top level to start.

>> If it evolves into a generically
>> useful microbenchmark generator, we can talk about moving it
>> src/main.  Thanks for doing this.
> I didn't think that this utility would ever move to "main", as it's just for
> internal testing.
> Regards,
> Gilles
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message