directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <kayyag...@apache.org>
Subject Re: Mavibot cache experiment
Date Wed, 11 Sep 2013 02:44:41 GMT
On Tue, Sep 10, 2013 at 9:35 PM, Emmanuel Lécharny <elecharny@gmail.com>wrote:

> Le 9/10/13 5:40 PM, Emmanuel Lécharny a écrit :
> > Hi guys,
> >
> > yestrday and today, I was implementing a cache in Mavibot to replace the
> > WeakReferences we were using previously. The rational was that with the
> > WeakReferences, we were unable to inject more than 50 000 entries in the
> > server (at this point, the GC is just going crazy trying to free some
> > WeakReferences, and it slows down the server to a pint it's unusable...)
> >
> > So after a day fighting with an EhCache implementation in Mavibot, I was
> > able to load 100K entries in the server. So far, so good, except that
> > the performances are anything but good. I can add 26 entries per second,
> > and fetch 555 entries per second. Worse than JDBM...
> >
> > Why is it so slow, especially for a search operation ? There are two
> > reasons :
> > - first, I configured the cache to store only 1000 elements (mainly
> nodes)
> > - second, when we try to update a leaf, we mostly have to load it from
> > disk, as we rarely have it in memory
> > - third, a leaf contains from 8 to 16 entries, and everytime we fetch a
> > leaf from disk, we have to deserialize the Entries, which is extremely
> > costly
> >
> > Fixing this third problem would save us a lot of time, and it's a matter
> > of adding one level of indirection (the entries would be kept as byte[],
> > and deserialized only when needed).
> >
> > If anyone has a better idea...
> indeed : the pb is that we serialize/deserialize too late. It would be
>
I think you mean 'too _early_'

> way better if we process serialized values until the point we return the
> result to the user :
>
> - addition : we receive an object, we immediately serialize it into a
> byte[] and process the addition using this byte[]. We don't anymore have
> to deserialize all the values from the page we will add the new value,
> they are all byte[]
> - search : same thing, we don't deserialize the values until we return
> it to the user.
>
> +1

> The gain will be huge !
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com

Mime
View raw message