directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <akaras...@apache.org>
Subject Re: [Testing] Coping with so many integration tests
Date Mon, 05 Feb 2007 16:37:46 GMT
Emmanuel Lecharny wrote:
> Yeah, running integration tests is time consumming. For 1.0.1, it's 
> acceptable, as it takes only 3min (depending on your computer), but in 
> 1.5, it's around 5 times slower.

We have many more tests about 40% more.

> We may consider finding out why it takes so long in 1.5 (server 
> initialization takes around 10 to 15 seconds, which is not really normal)

Please Emmanuel it does not take this long for startup. You're going to 
freak people out by saying stuff like this.  Let's watch these kinds of 
exaggerations.

It does take a little longer in 1.5 but really we have about 40% more 
tests and initialization time is slower but no where near 10-15 seconds.

A single integration test case (with one test method) takes about 3.5 
seconds to start, stop and cleanup on my old hardware (AMD 2800+).

Here's a run of just one empty test case. Committed here:

    http://svn.apache.org/viewvc?view=rev&revision=503758

Here's how to run it in apacheds/core-unit

    mvn -Dtest=OneEmptyITest -Dintegration test

Here's how much time it took for this ITest to run:

-------------------------------------------------------
  T E S T S
-------------------------------------------------------
Running org.apache.directory.server.core.schema.OneEmptyITest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.538 sec

And as you can see 3.538 seconds is not anywhere near 10-15 seconds 
exaggeration.  Now if you're running this on a really slow machine (400 
Mhz) with bad drives then I can see this being the case.

Now you said that this is 5x slower in 1.5 branch than it is in the 1.0 
branch.  Let's take a look at the same test case in the 1.0 branch. 
Here's there result there:

-------------------------------------------------------
  T E S T S
-------------------------------------------------------
Running org.apache.directory.server.core.schema.OneEmptyITest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.752 sec

So we're only 2x slower in the 1.5 branch than in the 1.0 branch.  Again 
that's way less than the 5 times figure you just threw out there.

Please let's be careful with what we toss out there next time.  People 
are listening and will run with negative news especially when it's not 
true and is coming from the mouths of our own people.

Brotha man, I hope you understand my reasons for asking this and don't 
think its me trying to bust your keyboard :).

...

> May I suggest a third solution? We might use tools like jmeter which are 
> able to launch tests against a ldap server, playing full scenarios. The 
> good part of it is that you can prepare the scenarios with a GUI, and 
> save them as XML files to be replayed within maven (there is a 
> maven-jmeter-plugin). The good points about this scenario that you only 
> launch the server once, tests are easy to write and to maintain, and 
> there is no more relation with the actual code base, so no huge 
> refactoring to do in tests when modifying base classes like AttributeImpl.

This is a great idea but how do we get this to be part of the build 
process?  Basically how do we get maven to launch a test server on 
startup that will be available for the entire run while tests operate on it?

This is a great idea if we can make maven do it.

Alex




Mime
View raw message