On Sun, Apr 5, 2009 at 11:57 AM, Emmanuel Lecharny <elecharny@apache.org> wrote:
Alex Karasulu wrote:
On Sat, Apr 4, 2009 at 6:41 PM, Emmanuel Lecharny <elecharny@apache.org>wrote:

 
Some follow-up :

I have done many tests, and I successfully got a working server without
having all the messages stored in memory, simply by changing the way the
ExecutorFilter is instanciated. In LdapService, line 361, instead of :

     // Now inject an ExecutorFilter for the write operations
     // We use the same number of thread than the number of IoProcessor
     // (NOTE : this has to be double checked)
     ((DefaultIoFilterChainBuilder)chain).addLast( "executor",
             new ExecutorFilter( new OrderedThreadPoolExecutor(
getTcpTransport().getNbThreads() ),
                 IoEventType.WRITE ) );

use :

     // Now inject an ExecutorFilter for the write operations
     // We use the same number of thread than the number of IoProcessor
     // (NOTE : this has to be double checked)
     ((DefaultIoFilterChainBuilder)chain).addLast( "executor",
             new ExecutorFilter( new OrderedThreadPoolExecutor(
getTcpTransport().getNbThreads() ) ) );

Now, the executor is used in both ways, and it seems to work much better.

   

So the Executor is not being used for all event types: reads, and writes.
How is this effecting the dynamic.  I think it's very important for us to
have a good written characterization of the problem before hand.  Then we
can follow up with a summary of what each approach does.
 
There is a bit of an issue : you can just have one single executor in the chain. Right now, I'm analysing the MINA internals to see what are the reason why limitating the IoEvent to WRITE only leads to OOM, or at least, late sending of all the messages, when accepting all the events is just ok...

The reason why I'm being a bit anal about this is because I know the
dynamics of MINA are going to be dramatically impacted when ApacheDS is put
under heavy load.
 
I'm also really scared about what could happen in a real environment (ie thousands of clients, with some of them sending more than one request at the same time).

This is painful ...


Oh yes I know it must be very painful but it is perhaps the most important work anyone can be doing for ApacheDS.

Alex