Hi Emmanuel,

On 9/18/07, Emmanuel Lecharny <elecharny@gmail.com> wrote:
Hi guys,

we currently have insanely long tests, especially when it comes to
integration tests. This is not only due to the fact that our server is
slow ;), but mainly because we are using an old version of JUnit,
which run setup() and teardown() for _each_ test. We have around 3000
tests currently, with around 420 core-unit tests and 180 server-unit
tests.

Yep this is crazy!

Those last tests are really time consuming, as the server is started
and stopped for each single test, and that cost around 2 seconds for
each server start. It cost more than 15 minutes on my laptop ...

Exactly.

Junit 4, the latest version we are currently using, offers a new
annotation system where you can add a @BeforeClass and @AfterClass,
applied on a startup and teardown methods only once for all the tests
in a simple class. If we tune the tests correctly, we can then reduce
the number of server startup.shutdown to 100 startup instead if 600.
The gain will be quite important, as we may save 70% of the total
cost.

How did you get the 100 instead of 600 figure?

I have done some test with JUnit 3.8, using a very ugly hack, and the
result is that for the SearchTest, I went down from 46 seconds to 6
seconds.

This is excellent.

There are a few things to do now :
- I have not tested JUnit 4 right now, but there are everything we
need to speed up the test, as far as I can see
- if we are to switch to JUnit 4, we have to figure out which impact
it has on the existing code, on Eclipse integration, and more
important, on Maven (does Surefire support Junit 4 natively ?)

Oh yeah.  I'm scared to find out.

- another alternative would be to use the more evolved TestNG
framework ; the very same steps should be followed (see previous
point)

TestNG - I checked this out a while back.  I love it but Maven can't deal well with it although technically it supports TestNG in surefire.  It's a work in progress.  JUnit is more mainstream tho - I'd stick to that.

- If we want to start the server only once, we have to be very careful
when writing a new test : it should not impact the existing data
loaded. That mean that a test should left the data in the same state
when it has been completed (successfully or not) than when it started.

This is the tricky part.  Perhaps a reverse application of changes can be done using a Test interceptor.  We can btw do many things with a test interceptor.

There may be some other options, like scripting tests (I was initially
thinking about using JMeter, but it's far from being perfect), or
adding a very simple framework to compare expected entries with those
get from the server.

At this point, I would like to get your opinion.

Been thinking about this for some time as you know.  At first I just made me a ram drive on linux to deal with this :D - it saved like 50% of the time but yeah that's laughable (both the technique and the time saved).  But this was just a quick remedy.  Still 7 minutes is too long to run integration tests.  The bulk of the cost of integration tests comes from bootstrapping specifically off the top of my head I'd list the following top time suckers:

 o partition creation (with various db files)
     o schema partition
     o system partition
 o schema loading and checks

I suspect you can do thousands of adds, deletes and modifies for the cost of bootstrapping.  So my recommendation is to find a way to snapshot the state of the server then roll it back with anti-requests.  This really is not as complex as it sounds.  We have the capability to do this easily using an interceptor much like the changelog interceptor in my embedding workshop which Ersin knocked out a while back.  Perhaps for each log operation tracked by this interceptor we can have a configuration switch to track the anti-ldif.  This would not be too hard to do.

Add => Delete
Delete => Add
Modify (Replace) => Modify (Replace)
Modify (Remove) => Modify (Add)
Modify (Add) => Modify (Remove)
ModifyDn => ModifyDN

The order would be reversed.  So if you have this train of requests { R1, R2, R3, R4, R5 } that correspond to the following LDIF sequence { L1, L2, L3, L4, L5 } then you just need the following anti operations { A5, A4, A3. A2, A1 }.  Another way to have done this nicely is if we had transactions enabled properly.  We could just rolllback :).  But this is a quick workaround to the problem.

Alex