directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Norval Hope" <>
Subject Re: ApacheDS LDAP codec streaming
Date Tue, 08 Apr 2008 13:11:46 GMT
I should also mention that I'm using JDK 1.5.0_11.

On Tue, Apr 8, 2008 at 11:09 PM, Norval Hope <> wrote:

> As a side note here is an excerpt of the old thread which I mentioned with
> a question from me and an answer from Alex:
> NH> I was just wanting to ask one of the ADS gurus what happens exactly at
> NH> the LDAP protocol level when an extremely large result set is returned
> NH> by ADS for an LDAP search(). Is a client able to start retreiving
> NH> results via their NamingEnumeration before the server has completed
> NH> its search? I'm hoping there is some sort of "chunking" at the
> NH> protocol level so that batches of results are streamed to the client
> NH> as they become available, rather then the waiting for the entire
> NH> result set to be available first. Is this the case?
> Yes and no :).  There is no chunking of an individual PDU sent back.
> Meaning each SearchEntryResponse prepared then sent out the door.  If an
> entry you're returning has a 1 mb jpeg in it then this will be used to
> prepare a response and return it rather than streaming out the jpeg. Old
> incarnations of MINA used to allow us to do this but it was complicated.
> For the search operation, we do not collect all entries before sending a
> response.  That would kill the server.  ApacheDS will pull out and send a
> result entry to you one at a time.  MINA may lag a little bit and some
> results may actually sit a while before going out the door.  But for all
> practical purposes we pull from the db and pump out results one by one.
> The way it works is a bit complicated and will take some time to discuss.
> I can go into it if enough people are interested and willing to document it
> :) after I describe how she does this.
> Alex
>   On Tue, Apr 8, 2008 at 11:05 PM, Norval Hope <> wrote:
> > Hi ApacheDS / MINA-ers,
> >
> > I am currently doing some load and memory testing against an extension
> > made to early version of ApacheDS (after it became 1.5 but before a
> > repository restructure that occurred not long after the rename from 1.0).
> > What I am finding is that all search result entries are cached in memory due
> > to ExecutorFilter.fireEvent() queing Events on the session buffer's event
> > queue that refer to them. Only after the SearchResponseIterator queues the
> > final SearchResponseDone message and it is encoded and flushed, do any of
> > the search results actually get flushed out to my JNDI client (JMeter as it
> > happens). As the searches concerned are attempting to return 100k results
> > this behaviour leads to a huge heap requirement, and in fact imposes a
> > seemingly articificial limit on the total number of results which can be
> > returned by a search (I have gone to great lengths to ensure a "streaming"
> > approach is used at the level of my custom partitions through the use of
> > lazy custom NamingEnumerations).
> >
> > This seems quite bizarre to me as I can see repeated calls to flush
> > encoded search results but these don't seem to have the effect of streaming
> > search results incrementally to the client as they become available, which
> > was my hope / expectation. The streaming behaviour of MINA / ApacheDS was
> > the subject of a query I posed many months ago and the response from the dev
> > list at that time was that individual search results would be streamed fine
> > but that large attributes like JPEGs may pose a problem as they are not
> > currently streamed. I've spent a full day trying to debug what is going on
> > between the thread doing the encoding and the thread which recieved the
> > original search request and then iterated through the search results, but I
> > still can't see how the SearchResponseDone causes all of the results to
> > finally be sent when it is eventually written and flushed (and the
> > synchronize block on the queue is exited) as the same NIO actions of writing
> > and flushing seem to be applied for all the previous search result messages
> > too.
> >
> > Also it seems strange to me that the ExecutorFilter's SessionBuffer
> > continues to hold references to an Event matching each search result long
> > after that result was encoded and flushed out. Once the search is 100%
> > completed and the final search response done message is written, then the ProcessEventsRunnable.processEvents()
> > method is called for each of these queued events but actually does nothing
> > with them.
> >
> > I was just wanting to check if:
> >
> >   a) this behaviour in the processing of search results has been noticed
> > before
> >   b) if anyone has any tips on debugging the MINA stack in this
> > particular case as I've found it extremely hard to get my mind around what
> > is actually going on re the individual search results which seem to have
> > been flushed out to NIO but then don't get passed on to the client until the
> > search is 100% finished
> >
> > as the MINA code concerned is very subtle and I've not tried to groc it
> > before.
> >
> > Any help / opinions / pointers gratefully accepted.
> >
> > Thanks,
> > Norval
> >

View raw message