directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <>
Subject Re: Search requests and responses...
Date Sat, 01 Dec 2012 08:13:50 GMT
Le 11/30/12 4:52 PM, Howard Chu a écrit :
> Emmanuel Lécharny wrote:
>> Hi guys,
>> We can do many different things when processing a search, and each of
>> them can collide :
>> o simple search
>>    o with time limit
>>    o with size limit
>>    o with both
>> o abandon request receoved
>> o session closed
>> o search with MandesDSAIt control
>> o paged search
>>    o cancelled paged search
>>    o new paged search
>> o persistent search
>> o replication
>> Of course, we can have many searches running at the same time for one
>> session...
>> The problem is that the way it currently works, beside the inherent
>> complexity, make it possible that the server ends with an OOM : we first
>> compute the set of candidates (it's a set of UUIDs), and we then fetch
>> all the entries one by one, writing them to the client. Of course, if
>> the client is not reading them fast enough, those entries are stored in
>> a queue, in memory, leading to potential OOM very quickly.
>> This is a real problem with the way we use MINA : we don't wait for the
>> entries to be actually *written* in the socket, we just push the entries
>> in the session.
>> In fact, there are three ways to write entries in MINA 2.
>> 1) you do what we currently do, and we are at risk to get an OOM
>> 2) for each write, we get back a WriteFuture, and we wait on this
>> writeFuture for the message to be in the socket. The pb with this
>> approach is that we block the thread until the entry is sent fully. It
>> works because we use an executorFilter, but this executor filter does
>> not have an infinite number of threads in its pool. At some point, we
>> will block completely
>> 3) there is a smartest way, but way more complex : we just write the
>> first entry, and that's it. When the first message is physically sent,
>> we will get a MessageSent event, and we can now process the next entry.
>> Etc. Of course, we will have as many MINA <-> ADS communication as we
>> have entries to send, an overhead we have to consider (it's around 5
>> to 10%)
>> However, doing so guarantee that we don't push anything in the queue :
>> all the entries are flushed one by one, we don't block any thread.
>> Atm, I have started to implement the third solution, and it works pretty
>> well for the simple search. Now, it gets very complex for the
>> pagedSearch, as it's a multiple SearchRequest operation, and we have to
>> deal with many different possibilities.
> This definitely sounds like a lot of work, for an uncertain payoff. I
> would consider a couple tweaks:
>   write two messages before waiting
>     if there are multiple entries in the result set, you can save a
> little overhead.
>     if there's only one entry, and then the searchResult message,
> there's no reason to wait at all.

Second thought : it won't be easy to send more than one message...

MINA send a messageSent event for every message that have been fully
pushed into the socket. If we try to write 2 entries (or more) we will
receive 2 messageSent events (or more).

We will need to do some matching between the last message we sent and
the associated messageSent event, which meas extra work.

Emmanuel Lécharny 

View raw message