directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Norval Hope" <>
Subject ApacheDS LDAP codec streaming
Date Tue, 08 Apr 2008 13:05:44 GMT
Hi ApacheDS / MINA-ers,

I am currently doing some load and memory testing against an extension made
to early version of ApacheDS (after it became 1.5 but before a repository
restructure that occurred not long after the rename from 1.0). What I am
finding is that all search result entries are cached in memory due to
ExecutorFilter.fireEvent() queing Events on the session buffer's event queue
that refer to them. Only after the SearchResponseIterator queues the
final SearchResponseDone message and it is encoded and flushed, do any of
the search results actually get flushed out to my JNDI client (JMeter as it
happens). As the searches concerned are attempting to return 100k results
this behaviour leads to a huge heap requirement, and in fact imposes a
seemingly articificial limit on the total number of results which can be
returned by a search (I have gone to great lengths to ensure a "streaming"
approach is used at the level of my custom partitions through the use of
lazy custom NamingEnumerations).

This seems quite bizarre to me as I can see repeated calls to flush encoded
search results but these don't seem to have the effect of streaming search
results incrementally to the client as they become available, which was
my hope / expectation. The streaming behaviour of MINA / ApacheDS was the
subject of a query I posed many months ago and the response from the dev
list at that time was that individual search results would be streamed fine
but that large attributes like JPEGs may pose a problem as they are not
currently streamed. I've spent a full day trying to debug what is going on
between the thread doing the encoding and the thread which recieved the
original search request and then iterated through the search results, but I
still can't see how the SearchResponseDone causes all of the results to
finally be sent when it is eventually written and flushed (and the
synchronize block on the queue is exited) as the same NIO actions of writing
and flushing seem to be applied for all the previous search result messages

Also it seems strange to me that the ExecutorFilter's SessionBuffer
continues to hold references to an Event matching each search result long
after that result was encoded and flushed out. Once the search is 100%
completed and the final search response done message is written, then
the ProcessEventsRunnable.processEvents()
method is called for each of these queued events but actually does nothing
with them.

I was just wanting to check if:

  a) this behaviour in the processing of search results has been noticed
  b) if anyone has any tips on debugging the MINA stack in this particular
case as I've found it extremely hard to get my mind around what is actually
going on re the individual search results which seem to have been flushed
out to NIO but then don't get passed on to the client until the search is
100% finished

as the MINA code concerned is very subtle and I've not tried to groc it

Any help / opinions / pointers gratefully accepted.


View raw message