Don't know why, the original mail from Alex never hit my mail box ...

On 2/5/07, Alex Karasulu <akarasulu@apache.org > wrote:



---------- Forwarded message ----------
Emmanuel Lecharny wrote:
> Yeah, running integration tests is time consumming. For 1.0.1, it's
> acceptable, as it takes only 3min (depending on your computer), but in
> 1.5, it's around 5 times slower.

We have many more tests about 40% more.

Yeah. In 1.0 we have  218 tests in core-unit, and 365 in 1.5

> We may consider finding out why it takes so long in 1.5 (server
> initialization takes around 10 to 15 seconds, which is not really normal)

Please Emmanuel it does not take this long for startup. You're going to
freak people out by saying stuff like this.  Let's watch these kinds of
exaggerations.

Sorry, I was totally wrong about those numbers. I guess I pulled them out of my head just when I woke up this morning, after a very bad sleep.

Let me correct them :
- each unit test is around 1 sec, startup and shutdown included in 1.5.0, not 10 to 15 seconds.
- in 1.0.1, each test is around 0,2 seconds, startup and shutdown included.

Be aware that those number has been taken on *my* computer, and by no means they are the official numbers of Apache Directory Server.

It does take a little longer in 1.5 but really we have about 40% more
tests and initialization time is slower but no where near 10-15 seconds.

A single integration test case (with one test method) takes about 3.5
seconds to start, stop and cleanup on my old hardware (AMD 2800+).

It's faster on my computer. 3.5 seconds is really above what I really get. I got around 1.8 to 2.5 seconds for single tests, but you have to ake into account the classloading of the unit test and stuff like that, which is not free...

Here's a run of just one empty test case. Committed here:

    mvn -Dtest=OneEmptyITest -Dintegration test
Running org.apache.directory.server.core.schema.OneEmptyITest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.538 sec

I get better results :)

Running org.apache.directory.server.core.schema.OneEmptyITest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.929 sec
 
Man, my imac is twice faster than your old AMD ;)

And as you can see 3.538 seconds is not anywhere near 10-15 seconds
exaggeration.  

Very true. My bad ... Never ever drink on sunday evening ;(


Now you said that this is 5x slower in 1.5 branch than it is in the 1.0
branch. 
So we're only 2x slower in the 1.5 branch than in the 1.0 branch.  Again
that's way less than the 5 times figure you just threw out there.

You are doing micro benchmark here. What I was talking about was the whole Integration test. It takes 2 mins and a half on 1.0.1, and  12 mins 06 seconds on 1.5.0
If you eliminate all the artefacts, it's around 4 times slower, considering we have 40% more tests...

Anyway, beside the wrong numbers I throwed to the world in my mail (trully sorry about that, my bad), here are some elements we must consider :
- all the modifications done in 1.5 are so important that we can live for a while with a slower integration phase
- we have 40% more tests, so it make sense that the integration test phase is slower, but, much more important, that do *not* means that 1.0.1 is not tested correctly : the 40% new tests are related to the new functions we have introduced in 1.5
- Important point too is that we have not tried to improve the performances in 1.5, because it's an intermediate version, the stable one will be 1.6. We can live with this lower performances (we don't launch integration tests every 5 mins, are we ???)

ok, just wanted to clarify my first mail, which was not correct. Thanks Alex for the heads up !
--
Cordialement,
Emmanuel Lécharny
www.iktek.com