directory-api mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <elecha...@gmail.com>
Subject Re: DN and valueOf( "<a DN>" ) method
Date Tue, 10 Aug 2010 08:29:47 GMT
  On 8/10/10 9:58 AM, Matthew Swift wrote:
>  [...]
>> But are you going to have 16K threads anyway ?
>
>
> Perhaps not today, but perhaps tomorrow.

Here, we are entering into the plain IO vs NIO debate.

There is no limit in the number of threads you can start on a JVM except 
the available memory, the JVM configuration (especially the -Xss size. 
It defaults to 1024, which is way to big if you are going to have 100K 
threads...) and obviously the OS you are running on.
>
> There are already very large CMT (core multi-threading) machines in 
> the pipeline from various vendors and I have heard and read 
> projections of 10K+ thread CMT machines in the next few years. Hence 
> the fork/join work going on in JDK7 at the moment and the push for 
> closures.
>
> Since many server apps size their thread pools based on the number of 
> available processors (core threads on CMT machines) then it is not 
> that unlikely in the next few years.
>
> I think that even today some app server environments run with tens of 
> thousands of threads.
Most certainly...
>
> Basically, we have no control over how other people use our SDK so, 
> while I might not use 16K threads, someone else might :-(
Well, I see no benefit in having 16K threads if the server is NIO based. 
But it might well be a good idea to switch to plain IOs, if we can scale 
to tens of thousands of connected sessions. This is the main issue here, 
compared to a Http server, for instance (where session are short lived). 
But even for a LDAP server, I'm not sure that we really need to deal 
with hundred of thousands opened sessions... Generally speaking, I'm not 
sure it's secure to 'open' the LDAP server to end users, so it's very 
likely that the server will be accessed by an application, which will 
probably not open a session per user, but instead use a pool of sessions 
and reuse it.

This is an interesting debate...

We are using MINA here, which is a NIO based framework, but I foresee a 
version which would be a pure IO framework, using either plain IO or 
NIO. That could help to conduct tests to see what will be the best 
solution (some like Paul Tyma think - and he even demonstrated - that IO 
is 30% fatser than NIO, even with thousands of threads : 
http://mailinator.blogspot.com/2008/02/kill-myth-please-nio-is-not-faster-than.html)

>
>
>>>
>>> A major problem with this approach if we choose to use it in the 
>>> OpenDS SDK is that common ancestor DNs (e.g. "dc=com") are going to 
>>> end up in a single LHM so, using our current design (RDN + parent 
>>> DN), all decoding attempts will usually end up contending on the 
>>> same lock anyway :-( So we may need to change our DN implementation 
>>> to better cope with this caching strategy.
>>>
>>> We are not alone though: a concurrent Map implementation which can 
>>> be used for caching in a similar manner to LHM is one of the most 
>>> frequently demanded enhancements to the java.util.concurrent library.
>> There might be some other data structure available, we may need to do 
>> some research in this area...
>>
>>
>
> I did have a poke around some time ago and found a couple. I didn't 
> evaluate them though as I seem to remember that they introduced a lot 
> of extra "baggage" and I'd like to keep the size of the SDK to a 
> minimum (to be honest, I think our OpenDS SDK is already too big).
It would be a pain to carry 1 more Mb of useless libs, true...

-- 
Regards,
Cordialement,
Emmanuel L├ęcharny
www.iktek.com


Mime
View raw message