directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Norval Hope (JIRA)" <>
Subject [jira] Commented: (DIRSERVER-1161) search results are not streamed to the client until final done response is queued
Date Wed, 09 Apr 2008 08:27:27 GMT


Norval Hope commented on DIRSERVER-1161:

Other things I've noticed in analysing this issue (these may or may not be knock-on problems
which tie back to the lack of streaming):
  1. MINA's ProtocolCodecFilter has the following inner class where there are no accessors
for "message" which leaves me thinking this object is for debugging only in which case it
maybe needs to be made conditional

   private static class MessageByteBuffer extends ByteBufferProxy
        private final Object message;
  2. In general when I look at memory usage I see a ton of strings which are RDN and LdapDN
instances that it seems to me could be better handled better using String.intern()
  3. In SearchResponseIterator I see this code and wonder why the LdapDN needs to be created
at @@@@@ for each search result (if this is just testing for validity does it make sense to
make this behaviour configurable via server.xml):

if ( !isReferral || req.getControls().containsKey( ManageDsaITControl.CONTROL_OID ) )
            SearchResponseEntry respEntry = new SearchResponseEntryImpl( req.getMessageId()
            respEntry.setAttributes( result.getAttributes() );
                respEntry.setObjectName( new LdapDN( result.getName() ) ); // @@@@@@
            catch ( InvalidNameException ine )
                log.error( "Invalid object name : " + result.getName(), ine);
                throw new RuntimeException( ine );
            prefetched = respEntry;

   4. Even if incremental streaming is achieved I'm worried that the architecture doesn't
seem to cope with the case where SearchHandler generates results quicker then the JNDI client
consumes them, so that an unbounded amount of encoded results can build up on the server waiting
to be transmitted, or does NIO take care of this (assuming the current streaming bug is fixed)
by blocking the server in some way?

> search results are not streamed to the client until final done response is queued
> ---------------------------------------------------------------------------------
>                 Key: DIRSERVER-1161
>                 URL:
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.0
>         Environment: JDK 1.5.0_11 
>            Reporter: Norval Hope
>         Attachments: streaming_log_output.txt, streaming_logging.patch
>   Original Estimate: 120h
>  Remaining Estimate: 120h
> Search results accumulate in Events on the SessionBuffer.eventQueue within ExecutorFilter.fireEvent()
until final done response for the search is written to the session and then all results for
the search (possibly millions depending on the search and state of the directory) are written
out at once. This is a big problem for scalability and I gather from previous correspondence
with Alex that this behaviour is unexpected.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message