Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List" <dev@directory.apache.org>
Received-SPF: pass (athena.apache.org: domain of elecharny@gmail.com
 designates 209.85.212.169 as permitted sender)
Message-ID: <522F4364.4090805@gmail.com>
Date: Tue, 10 Sep 2013 18:05:56 +0200
From: =?UTF-8?B?RW1tYW51ZWwgTMOpY2hhcm55?= <elecharny@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:17.0) Gecko/20130801 Thunderbird/17.0.8
MIME-Version: 1.0
To: dev@directory.apache.org
Subject: Re: Mavibot cache experiment
References: <522F3D81.6070609@symas.com>
In-Reply-To: <522F3D81.6070609@symas.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Le 9/10/13 5:40 PM, Emmanuel Lécharny a écrit :
> Hi guys,
>
> yestrday and today, I was implementing a cache in Mavibot to replace the
> WeakReferences we were using previously. The rational was that with the
> WeakReferences, we were unable to inject more than 50 000 entries in the
> server (at this point, the GC is just going crazy trying to free some
> WeakReferences, and it slows down the server to a pint it's unusable...)
>
> So after a day fighting with an EhCache implementation in Mavibot, I was
> able to load 100K entries in the server. So far, so good, except that
> the performances are anything but good. I can add 26 entries per second,
> and fetch 555 entries per second. Worse than JDBM...
>
> Why is it so slow, especially for a search operation ? There are two
> reasons :
> - first, I configured the cache to store only 1000 elements (mainly nodes)
> - second, when we try to update a leaf, we mostly have to load it from
> disk, as we rarely have it in memory
> - third, a leaf contains from 8 to 16 entries, and everytime we fetch a
> leaf from disk, we have to deserialize the Entries, which is extremely
> costly
>
> Fixing this third problem would save us a lot of time, and it's a matter
> of adding one level of indirection (the entries would be kept as byte[],
> and deserialized only when needed).
>
> If anyone has a better idea...
indeed : the pb is that we serialize/deserialize too late. It would be
way better if we process serialized values until the point we return the
result to the user :

- addition : we receive an object, we immediately serialize it into a
byte[] and process the addition using this byte[]. We don't anymore have
to deserialize all the values from the page we will add the new value,
they are all byte[]
- search : same thing, we don't deserialize the values until we return
it to the user.

The gain will be huge !

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com