directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Karasulu" <akaras...@apache.org>
Subject Re: Speeding up tests
Date Tue, 18 Sep 2007 23:41:36 GMT
Hi Emmanuel,

On 9/18/07, Emmanuel Lecharny <elecharny@gmail.com> wrote:
>
> Hi guys,
>
> we currently have insanely long tests, especially when it comes to
> integration tests. This is not only due to the fact that our server is
> slow ;), but mainly because we are using an old version of JUnit,
> which run setup() and teardown() for _each_ test. We have around 3000
> tests currently, with around 420 core-unit tests and 180 server-unit
> tests.


Yep this is crazy!

Those last tests are really time consuming, as the server is started
> and stopped for each single test, and that cost around 2 seconds for
> each server start. It cost more than 15 minutes on my laptop ...


Exactly.

Junit 4, the latest version we are currently using, offers a new
> annotation system where you can add a @BeforeClass and @AfterClass,
> applied on a startup and teardown methods only once for all the tests
> in a simple class. If we tune the tests correctly, we can then reduce
> the number of server startup.shutdown to 100 startup instead if 600.
> The gain will be quite important, as we may save 70% of the total
> cost.


How did you get the 100 instead of 600 figure?

I have done some test with JUnit 3.8, using a very ugly hack, and the
> result is that for the SearchTest, I went down from 46 seconds to 6
> seconds.


This is excellent.

There are a few things to do now :
> - I have not tested JUnit 4 right now, but there are everything we
> need to speed up the test, as far as I can see
> - if we are to switch to JUnit 4, we have to figure out which impact
> it has on the existing code, on Eclipse integration, and more
> important, on Maven (does Surefire support Junit 4 natively ?)


Oh yeah.  I'm scared to find out.

- another alternative would be to use the more evolved TestNG
> framework ; the very same steps should be followed (see previous
> point)


TestNG - I checked this out a while back.  I love it but Maven can't deal
well with it although technically it supports TestNG in surefire.  It's a
work in progress.  JUnit is more mainstream tho - I'd stick to that.

- If we want to start the server only once, we have to be very careful
> when writing a new test : it should not impact the existing data
> loaded. That mean that a test should left the data in the same state
> when it has been completed (successfully or not) than when it started.


This is the tricky part.  Perhaps a reverse application of changes can be
done using a Test interceptor.  We can btw do many things with a test
interceptor.

There may be some other options, like scripting tests (I was initially
> thinking about using JMeter, but it's far from being perfect), or
> adding a very simple framework to compare expected entries with those
> get from the server.
>
> At this point, I would like to get your opinion.
>

Been thinking about this for some time as you know.  At first I just made me
a ram drive on linux to deal with this :D - it saved like 50% of the time
but yeah that's laughable (both the technique and the time saved).  But this
was just a quick remedy.  Still 7 minutes is too long to run integration
tests.  The bulk of the cost of integration tests comes from bootstrapping
specifically off the top of my head I'd list the following top time suckers:

 o partition creation (with various db files)
     o schema partition
     o system partition
 o schema loading and checks

I suspect you can do thousands of adds, deletes and modifies for the cost of
bootstrapping.  So my recommendation is to find a way to snapshot the state
of the server then roll it back with anti-requests.  This really is not as
complex as it sounds.  We have the capability to do this easily using an
interceptor much like the changelog interceptor in my embedding workshop
which Ersin knocked out a while back.  Perhaps for each log operation
tracked by this interceptor we can have a configuration switch to track the
anti-ldif.  This would not be too hard to do.

Add => Delete
Delete => Add
Modify (Replace) => Modify (Replace)
Modify (Remove) => Modify (Add)
Modify (Add) => Modify (Remove)
ModifyDn => ModifyDN

The order would be reversed.  So if you have this train of requests { R1,
R2, R3, R4, R5 } that correspond to the following LDIF sequence { L1, L2,
L3, L4, L5 } then you just need the following anti operations { A5, A4, A3.
A2, A1 }.  Another way to have done this nicely is if we had transactions
enabled properly.  We could just rolllback :).  But this is a quick
workaround to the problem.

Alex

Mime
View raw message