directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Norval Hope (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DIRSERVER-1161) search results are not streamed to the client until final done response is queued
Date Wed, 09 Apr 2008 08:14:24 GMT

     [ https://issues.apache.org/jira/browse/DIRSERVER-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Norval Hope updated DIRSERVER-1161:
-----------------------------------

    Attachment: streaming_log_output.txt
                streaming_logging.patch

The .patch attached adds some extra logging to SearchHandler and ExecutorFilter (in MINA)
to make the problem apparent, and adds some extra settings in apacheds/server-main/log4j.properties
to ensure the new messages are logged.

I checked out http://svn.apache.org/repos/asf/directory/apacheds/branches/1.0-with-dependencies
to a directory called "ad" and then checked out http://svn.apache.org/repos/asf/mina/tags/1.0.3
to a directory called "mina" under the top-level "ad", so the patch also includes building
of this custom MINA 1.0.3 in the top-level pom.xml. 

Finally I've attached some log output for a search triggered from JXplorer for all DNs under
ou=system. Note how in attached .txt file we see "size of queue is currently=17" without any
writing being attempted until the final response done is queued, at which time all results
are written. I have verified when running an LDAP client in the debugger (JMeter) that it
sees no search results until the done has been written to the session and the NIO structures
are flushed (and the synchronized block exited). I have also verified this lack of streaming
applies to searches returning 100k results, so it's not a simple timing issue or anything
simple like that.

> search results are not streamed to the client until final done response is queued
> ---------------------------------------------------------------------------------
>
>                 Key: DIRSERVER-1161
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1161
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.0
>         Environment: JDK 1.5.0_11 
>            Reporter: Norval Hope
>         Attachments: streaming_log_output.txt, streaming_logging.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Search results accumulate in Events on the SessionBuffer.eventQueue within ExecutorFilter.fireEvent()
until final done response for the search is written to the session and then all results for
the search (possibly millions depending on the search and state of the directory) are written
out at once. This is a big problem for scalability and I gather from previous correspondence
with Alex that this behaviour is unexpected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message