directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Karasulu" <akaras...@apache.org>
Subject Re: [ApacheDS] Default partition design ideas
Date Sun, 03 Feb 2008 06:41:29 GMT
On Feb 2, 2008 1:03 PM, Howard Chu <hyc@symas.com> wrote:

> > Very good point about these opposing factors. So I figure OpenLDAP just
> > uses the cache in the underlying B-Tree instead of managing some kind of
> > separate entry cache?
>
> No, we have a separate entry cache too. I originally wrote back-bdb
> without
> any entry caching, but Jong @ IBM benchmarked/profiled it and implemented
> an
> entry cache for it before 2.1 was released. (Back then I was happy just to
> get
> it working, at speeds comparable to back-ldbm. How things change ;)
>
> The motivation here is that the serialized data that we store in the
> database
> is not directly usable with our in-memory data structures. As such, even
> when
> the B-Tree cache is 100% effective, there's a cost associated with
> deserialization that we can avoid by using our own entry cache.


That's exactly the same problem we have: deserialization + not being able to
manipulate the raw binary data in accordance with internal data structures.
It's good to get this feedback thanks.  This is why we have an entry cache
although there are some nasty issues we have to contend with like memory
management and sharing a single cache across our partitions instead of each
partition managing it's own entry cache.  Unfortunately because of the way
we designed search capabilities into the partition itself we cannot have it
share a global cache.

There are so many problems to solve, we're just scratching the surface now
as we make our way towards 2.0.  IMO 1.0 was a toy.  2.0 has a chance to
solve some neat problems and actually put several ideas to the test.


> There are plenty of interesting data structures/system architectures to
> explore, and now you've got the lab resources to try them out.
>

Yes I am looking forward to that: it will be fun.  I think the first thing I
want to do is move to MINA 2.0 and try the server out with the APR transport
and the NIO transport to compare the results.

/me salivating now


>
> >      > Yeah they're great ideas. We just need to have a solid SLAMD lab
> and
> >      > start testing these ideas out. I got the machines:
> >      >
> >      > 9 load injectos
> >      > 1 SLAMD Server
> >      > 1 beefy server for running ApacheDS>
>

Thanks again,
Alex

Mime
View raw message