directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Lecharny (JIRA)" <>
Subject [jira] Commented: (DIRSERVER-1161) search results are not streamed to the client until final done response is queued
Date Fri, 11 Apr 2008 22:58:04 GMT


Emmanuel Lecharny commented on DIRSERVER-1161:

Hi Norval,

I read all this thread in the train while coming back from  Amsterdam, and what I saw (and
what I went through while debugging the server) was quite scary ...

I'm not sure that MINA in the current version still hold the data in a queue before sending
them, but anyway, if it's not the case, the slow JNDI client is already a perfect example
of what we should protect the server from !

Ok, after some minimal investigation, I found that MINA was calling the encoder in the ProtocoalEncoderFilter
and flush the resulting bytes directly to the client, but then try to send the message again,
which has obviously already been sent. Here is the code :

ProtocolCodecFilter.filterWrite() {
        ProtocolEncoder encoder = getEncoder(session);
        ProtocolEncoderOutputImpl encoderOut = getEncoderOut(session,
                nextFilter, writeRequest);

        try {
            encoder.encode(session, message, encoderOut);

            // Here, the encoded message is sent.

            // Here an empty message is sent...
            nextFilter.filterWrite(session, new WriteRequest(
                    new MessageByteBuffer(writeRequest.getMessage()),
                    writeRequest.getFuture(), writeRequest.getDestination()));

I don't know if this last empty message sent is related to the issue you mentionned in your
previous comment, as I haven't yet thoroughly tested the code, but it might be a problem.

I think we might went to wait until the data are _really_ read by the client just after the
encoderOut.flush() call before sending the next result, and this would protect the server
from the issue you mentioned before.

Side remark : MINA migth have a cool API, but when it comes to debug it and getting into the
guts of it, this is a nightmare. I just think that many part of it is just over-complicated,
defeating the moto : "Write code for the Average Programmer"

> search results are not streamed to the client until final done response is queued
> ---------------------------------------------------------------------------------
>                 Key: DIRSERVER-1161
>                 URL:
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.0
>         Environment: JDK 1.5.0_11 
>            Reporter: Norval Hope
>         Attachments: apacheds_1.5.1_streaming.patch, apacheds_1.5.1_streaming_log_output.txt,
installers_1.5.1_streaming.patch, mina_1.1.2_streaming.patch, pom.xml, streaming_log_output.txt,
>   Original Estimate: 120h
>  Remaining Estimate: 120h
> Search results accumulate in Events on the SessionBuffer.eventQueue within ExecutorFilter.fireEvent()
until final done response for the search is written to the session and then all results for
the search (possibly millions depending on the search and state of the directory) are written
out at once. This is a big problem for scalability and I gather from previous correspondence
with Alex that this behaviour is unexpected.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message