On Fri, Feb 4, 2011 at 11:58 PM, Stefan Seelmann <seelmann@apache.org> wrote:
Hi guys,

one of our Hudson builds currently fails [1], the cause is an
IOException (Too many open files). The reason this happens is that in
our integration test (server-integ and ldap-api-test) we open real
network connections. In many cases a new connection is opened in each
test method. I also found a case where we open new connections within
a loop.


Yes the tests all need to be revamped to make integration tests run smooth. Some tests are performance tests like this one that was recently ignored:

     http://goo.gl/Cws6O

We also have network tests happening as you can see in the core-integ area which should not be testing services. I'll take a pass at trying to organize the tests so we at least localize all the network tests under server-integ. 
 
Most times the connections is closed after usage. However, as you
certainly know, there is still a problem with allocating many new
connections: sockets remain in TIME_WAIT state for some time after
bein closed, see [2]. On the Hudson servers we can of course increase
the ulimit -n and reduce the TIME_WAIT value. But I think that won't
scale when connections are opened in a loop.

Agreed.
 
I'd like to discuss possible options to improve the situation:

- Fist I think in general we should avoid to create new connections
within a loop in integration tests. We have other techniques to run
such test [3]. In general such test patterns are stress or performance
tests, but not integration tests.

You're right, we need to be more disciplined with making sure only integration tests are present. We pay for these heavy long durations with builds taking 15+ minutes on laptops.

It's a good thing we're refreshing our build config on hudson. It's showing us we need to clean up a little. 
 
- Second we should avoid to create a new connection for each test
method. One option to achieve this is to use a single connection per
test class without opening/closing them @Before/@After each test.
Maybe the better option is to use a connection pool. Of course when
testing connect/disconnect/bind/unbind it may be required to use a
fresh connection objects.

Thoughts?

+1

If the test permits the reuse then we definitely should be reusing the connection for each test class.

Thanks,
Alex