On 2/4/07, Alex Karasulu <akarasulu@apache.org> wrote:

The number of integration tests are growing large (we have hundreds) as
we get more coverage for the server and the functionality increases.
These tests start and stop apacheds and clean out working directories on
each run so they can take a while especially if a test case has lots of
tests in it.

I have two ideas on how to ameliorate these problems.  One is short term
and the other long term.

First the short term solution.  I found that I can create large 512K ram
drives (do not try more than 512K it does not work) then combine them
together using RAID to make a large disk.  After having done this test
case classes that took 20 seconds started to take 4 seconds instead.
Not bad but still not ideal since we have so many tests.  This however
does not have that many drawbacks besides still taking 9 minutes and 49
seconds to run all the integration tests.

Long term solution:  See if we can modify the surefire plugin for
distributed operation.  It would be easy to have a java daemon that can
run maven on various projects.  Commands can be sent to this to setup
and run individual test case classes.  The distributed surefire plugin
would simply queue operations and collect the results from several
machines.  This way you get serious parallelism especially if you have
many machines.  With four machines the time could be cut down to 2 and a
half minutes.


Yeah, running integration tests is time consumming. For 1.0.1, it's acceptable, as it takes only 3min (depending on your computer), but in 1.5, it's around 5 times slower.

We may consider finding out why it takes so long in 1.5 (server initialization takes around 10 to 15 seconds, which is not really normal)

However, even with shortest tests, it's still quite long, and it won't be better in the next few months (i'm not sure that the increase rate of tests is lower that Intel processor performance increase rate ... :)

May I suggest a third solution? We might use tools like jmeter which are able to launch tests against a ldap server, playing full scenarios. The good part of it is that you can prepare the scenarios with a GUI, and save them as XML files to be replayed within maven (there is a maven-jmeter-plugin). The good points about this scenario that you only launch the server once, tests are easy to write and to maintain, and there is no more relation with the actual code base, so no huge refactoring to do in tests when modifying base classes like AttributeImpl.


PS : Ole, depending on nighty builds is not an option. When you are working on trunks, you should *not* commit something that break the server. Nighty Builds are just a safety belt, because sometime you just break this rule, without knowing it (damn local repo ...)


Emmanuel Lécharny