directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Lecharny" <>
Subject Re: [Testing] Coping with so many integration tests
Date Mon, 05 Feb 2007 14:22:11 GMT
On 2/4/07, Alex Karasulu <> wrote:
> Hi,
> The number of integration tests are growing large (we have hundreds) as
> we get more coverage for the server and the functionality increases.
> These tests start and stop apacheds and clean out working directories on
> each run so they can take a while especially if a test case has lots of
> tests in it.
> I have two ideas on how to ameliorate these problems.  One is short term
> and the other long term.
> First the short term solution.  I found that I can create large 512K ram
> drives (do not try more than 512K it does not work) then combine them
> together using RAID to make a large disk.  After having done this test
> case classes that took 20 seconds started to take 4 seconds instead.
> Not bad but still not ideal since we have so many tests.  This however
> does not have that many drawbacks besides still taking 9 minutes and 49
> seconds to run all the integration tests.
> Long term solution:  See if we can modify the surefire plugin for
> distributed operation.  It would be easy to have a java daemon that can
> run maven on various projects.  Commands can be sent to this to setup
> and run individual test case classes.  The distributed surefire plugin
> would simply queue operations and collect the results from several
> machines.  This way you get serious parallelism especially if you have
> many machines.  With four machines the time could be cut down to 2 and a
> half minutes.
> Thoughts?

Yeah, running integration tests is time consumming. For 1.0.1, it's
acceptable, as it takes only 3min (depending on your computer), but in 1.5,
it's around 5 times slower.

We may consider finding out why it takes so long in 1.5 (server
initialization takes around 10 to 15 seconds, which is not really normal)

However, even with shortest tests, it's still quite long, and it won't be
better in the next few months (i'm not sure that the increase rate of tests
is lower that Intel processor performance increase rate ... :)

May I suggest a third solution? We might use tools like jmeter which are
able to launch tests against a ldap server, playing full scenarios. The good
part of it is that you can prepare the scenarios with a GUI, and save them
as XML files to be replayed within maven (there is a maven-jmeter-plugin).
The good points about this scenario that you only launch the server once,
tests are easy to write and to maintain, and there is no more relation with
the actual code base, so no huge refactoring to do in tests when modifying
base classes like AttributeImpl.


PS : Ole, depending on nighty builds is not an option. When you are working
on trunks, you should *not* commit something that break the server. Nighty
Builds are just a safety belt, because sometime you just break this rule,
without knowing it (damn local repo ...)


Emmanuel L├ęcharny

View raw message