directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Norval Hope (JIRA)" <>
Subject [jira] Commented: (DIRSERVER-1161) search results are not streamed to the client until final done response is queued
Date Sun, 13 Apr 2008 22:41:06 GMT


Norval Hope commented on DIRSERVER-1161:

Trustin was also kindly helping me sound out my theories etc re this issue. I'm including
his responses to the issue too for completeness:

===> re spawning thread (in my view the critical immediate fix)

Hi Norval,

I think you figured the cause of the problem by yourself.  It seems like
SearchHandler is writing the search results inside the ExecutorFilter's
thread.  Any response messages from the client will not be processed by
IoHandler because ExecutorFilter doesn't allow the execution of more
than one event for one session.  That's why Executor's queue size
increases.  Your solution (i.e. writing the search result from other
thread) will fix the problem.  I think ApacheDS is also designed to be
thread safe so it should be fine.  It's also fine with MINA to do so.


====> re blocking queue (also important, but once the separate thread is spawned above
it only becomes a problem when client and server are running at markedly different speeds
- but when this is so we can end up back at the same "OutOfMemory" problem for big searches).

Hi again,

MINA 2 has a mechanism that copes with too fast client without using
BlockingQueue.  You might want to backport it to MINA 1.1 and include it
into ApacheDS codebase.  Writing too fast from the server side is a
different problem.  You will have to check scheduledWriteBytes property
of the session and wait a little bit if it's too big.


===> I also pinged Trustin about why he felt the blocking queue wasn't a good idea, but
haven't heard back as yet.

> search results are not streamed to the client until final done response is queued
> ---------------------------------------------------------------------------------
>                 Key: DIRSERVER-1161
>                 URL:
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.0
>         Environment: JDK 1.5.0_11 
>            Reporter: Norval Hope
>         Attachments: apacheds_1.5.1_streaming.patch, apacheds_1.5.1_streaming.patch,
apacheds_1.5.1_streaming_log_output.txt, installers_1.5.1_streaming.patch, installers_1.5.1_streaming.patch,
mina_1.1.2_streaming.patch, mina_1.1_trunk_streaming.patch, pom.xml, streaming_log_output.txt,
>   Original Estimate: 120h
>  Remaining Estimate: 120h
> Search results accumulate in Events on the SessionBuffer.eventQueue within ExecutorFilter.fireEvent()
until final done response for the search is written to the session and then all results for
the search (possibly millions depending on the search and state of the directory) are written
out at once. This is a big problem for scalability and I gather from previous correspondence
with Alex that this behaviour is unexpected.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message