directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Norval Hope" <>
Subject Re: ApacheDS LDAP codec streaming
Date Tue, 08 Apr 2008 13:09:39 GMT
As a side note here is an excerpt of the old thread which I mentioned with a
question from me and an answer from Alex:

NH> I was just wanting to ask one of the ADS gurus what happens exactly at
NH> the LDAP protocol level when an extremely large result set is returned
NH> by ADS for an LDAP search(). Is a client able to start retreiving
NH> results via their NamingEnumeration before the server has completed
NH> its search? I'm hoping there is some sort of "chunking" at the
NH> protocol level so that batches of results are streamed to the client
NH> as they become available, rather then the waiting for the entire
NH> result set to be available first. Is this the case?

Yes and no :).  There is no chunking of an individual PDU sent back. Meaning
each SearchEntryResponse prepared then sent out the door.  If an entry
you're returning has a 1 mb jpeg in it then this will be used to prepare a
response and return it rather than streaming out the jpeg. Old incarnations
of MINA used to allow us to do this but it was complicated.
For the search operation, we do not collect all entries before sending a
response.  That would kill the server.  ApacheDS will pull out and send a
result entry to you one at a time.  MINA may lag a little bit and some
results may actually sit a while before going out the door.  But for all
practical purposes we pull from the db and pump out results one by one.
The way it works is a bit complicated and will take some time to discuss.  I
can go into it if enough people are interested and willing to document it :)
after I describe how she does this.

On Tue, Apr 8, 2008 at 11:05 PM, Norval Hope <> wrote:

> Hi ApacheDS / MINA-ers,
> I am currently doing some load and memory testing against an extension
> made to early version of ApacheDS (after it became 1.5 but before a
> repository restructure that occurred not long after the rename from 1.0).
> What I am finding is that all search result entries are cached in memory due
> to ExecutorFilter.fireEvent() queing Events on the session buffer's event
> queue that refer to them. Only after the SearchResponseIterator queues the
> final SearchResponseDone message and it is encoded and flushed, do any of
> the search results actually get flushed out to my JNDI client (JMeter as it
> happens). As the searches concerned are attempting to return 100k results
> this behaviour leads to a huge heap requirement, and in fact imposes a
> seemingly articificial limit on the total number of results which can be
> returned by a search (I have gone to great lengths to ensure a "streaming"
> approach is used at the level of my custom partitions through the use of
> lazy custom NamingEnumerations).
> This seems quite bizarre to me as I can see repeated calls to flush
> encoded search results but these don't seem to have the effect of streaming
> search results incrementally to the client as they become available, which
> was my hope / expectation. The streaming behaviour of MINA / ApacheDS was
> the subject of a query I posed many months ago and the response from the dev
> list at that time was that individual search results would be streamed fine
> but that large attributes like JPEGs may pose a problem as they are not
> currently streamed. I've spent a full day trying to debug what is going on
> between the thread doing the encoding and the thread which recieved the
> original search request and then iterated through the search results, but I
> still can't see how the SearchResponseDone causes all of the results to
> finally be sent when it is eventually written and flushed (and the
> synchronize block on the queue is exited) as the same NIO actions of writing
> and flushing seem to be applied for all the previous search result messages
> too.
> Also it seems strange to me that the ExecutorFilter's SessionBuffer
> continues to hold references to an Event matching each search result long
> after that result was encoded and flushed out. Once the search is 100%
> completed and the final search response done message is written, then the ProcessEventsRunnable.processEvents()
> method is called for each of these queued events but actually does nothing
> with them.
> I was just wanting to check if:
>   a) this behaviour in the processing of search results has been noticed
> before
>   b) if anyone has any tips on debugging the MINA stack in this particular
> case as I've found it extremely hard to get my mind around what is actually
> going on re the individual search results which seem to have been flushed
> out to NIO but then don't get passed on to the client until the search is
> 100% finished
> as the MINA code concerned is very subtle and I've not tried to groc it
> before.
> Any help / opinions / pointers gratefully accepted.
> Thanks,
> Norval

View raw message