directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <elecha...@apache.org>
Subject Re: ldap server buffers search results?
Date Sat, 04 Apr 2009 22:41:44 GMT
Some follow-up :

I have done many tests, and I successfully got a working server without 
having all the messages stored in memory, simply by changing the way the 
ExecutorFilter is instanciated. In LdapService, line 361, instead of :

        // Now inject an ExecutorFilter for the write operations
        // We use the same number of thread than the number of IoProcessor
        // (NOTE : this has to be double checked)
        ((DefaultIoFilterChainBuilder)chain).addLast( "executor",
                new ExecutorFilter( new OrderedThreadPoolExecutor( 
getTcpTransport().getNbThreads() ),
                    IoEventType.WRITE ) );

use :

        // Now inject an ExecutorFilter for the write operations
        // We use the same number of thread than the number of IoProcessor
        // (NOTE : this has to be double checked)
        ((DefaultIoFilterChainBuilder)chain).addLast( "executor",
                new ExecutorFilter( new OrderedThreadPoolExecutor( 
getTcpTransport().getNbThreads() ) ) );

Now, the executor is used in both ways, and it seems to work much better.

I still have to understand how the ExecutorFilter() is working in MINA, 
but at least, I think we cant release MINA 2.0.0 without a clear 
documentation about the way to use this executor filter !


Andrea Gariboldi wrote:
> Hi Martin,
>      Emmanuel found that there is a threading design problem within MINA:
>
> The IoProcessor that is handling the search request and putting the encoded
> results in the WriteQueue,
> is the same that should dequeue those results from the queue and send it. So
> once it has finished
> with the request processing it checks for things to write and write all the
> results in the queue, but
> if you have long searches you will most likely get an OOM exception, because
> of the growing queue.
>
> Andrea
>
> 2009/4/3 Martin Alderson <equim@planetquake.com>
>
>   
>> Hi Andrea,
>>
>> Just to let you know, this looks to be the same problem that is being
>> discussed in my OutOfMemoryError thread on the users mailing list.
>>
>> It still seems to be happening in the latest ApacheDS trunk.  I don't
>> really understand why it is happening yet though.
>>
>> Martin
>>
>>
>>
>> Andrea Gariboldi wrote:
>>
>>     
>>> Hi guys,
>>>  i was testing the oracle partition and it looks like the mina filter
>>> chain
>>> buffers the search results until
>>> the search ends and then starts message encoding end sending... Is this a
>>> protocol constraint or a bug?
>>> (org.apache.directory.server.ldap.handlers.SearchHandler : 326 snippet:
>>>
>>>  while ( (count < sizeLimit ) && cursor.next() )
>>>        {
>>>            if ( session.getIoSession().isClosing() )
>>>            {
>>>                break;
>>>            }
>>>
>>>            ClonedServerEntry entry = cursor.get();
>>>            session.getIoSession().write( generateResponse( session, req,
>>> entry ) ); // here the message is "writed" trought mina filters, but is
>>> not
>>> encoded until the end of this loop...
>>>            count++;
>>>        }
>>>
>>> )
>>>
>>>
>>> Doing a long search (1000+ entries) results in long waits for the first
>>> result on the client and java heap consumption.
>>>
>>> am i missing something?
>>>
>>> Andrea
>>>
>>>
>>>       
>
>   


-- 
--
cordialement, regards,
Emmanuel L├ęcharny
www.iktek.com
directory.apache.org



Mime
View raw message