Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List" <dev@directory.apache.org>
Received-SPF: pass (nike.apache.org: domain of srivastavamohit91@gmail.com
 designates 209.85.212.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <52245422.2090209@gmail.com>
References: <52245422.2090209@gmail.com>
Date: Mon, 2 Sep 2013 22:41:13 +0530
Message-ID: 
 <CAL=jsQbA5yWO=vnihY35jpRXMW1Uwa5X4FbP=QeVN5s8o8mu0g@mail.gmail.com>
Subject: Re: [Mavibot] Need for a cache to replace the WeakReferences...
From: Mohit Srivastava <srivastavamohit91@gmail.com>
To: Apache Directory Developers List <dev@directory.apache.org>
Content-Type: multipart/alternative; boundary=089e0122f5f6ccada704e569a8e4

--089e0122f5f6ccada704e569a8e4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think we can use EhCache for now. Also modify it for our purposes.


On 2 September 2013 14:32, Emmanuel L=E9charny <elecharny@gmail.com> wrote:

> Hi guys,
>
> I have done some tests a week ago on Mavibot partition. It appears that
> we have a serious performance issue as soon as the JVM memory is
> completly eaten, when adding some entries. What happens is that we start
> to discard some pages as they are hold by WeakReferences, and the GC has
> to do quite a expensive processing to get back some free memory. This
> slows down the add operation so much (by at least a factor 10) that it's
> not possible to keep going with WeakReferences (or SoftReferences).
>
> At this point, the best solution would be to use a cache to replace this
> mechanism.
>
> There are different things we can cache :
> o PageIo : it's useless, as we still have to deserialize them afterward
> o Nodes and Leaves : They contain references to other Nodes or Leaves,
> and for Leaves, reference to data. Holding the upper Nodes would save us
> a lot of time
> o Data : the ultimate objects to cache : having the object in cache
> spares us the time it takes to searcj it in a Btree
>
> A few other things to consider :
> o we need a versionned cache : all the elements may be versionned.
> o each BTree may be cached, with different configuation (some BTree may
> be used rarely, other may be heavily used, requiring a more aggressive
> cache configuration)
> o the Cache must not use locks, but CAS (Compare-And-Swap) for better
> performance
>
> At this point, I suggest using EhCache as a first approach, in order to
> leverage the cache we already use in the server, but we may want to
> design our own LRU cache (something that should not be too complex to
> implement).
>
> Thoughts ?
>
> --
> Regards,
> Cordialement,
> Emmanuel L=E9charny
> www.iktek.com
>
>

--089e0122f5f6ccada704e569a8e4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I think we can use EhCache for now. Also modify it for our=
 purposes.</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quot=
e">On 2 September 2013 14:32, Emmanuel L=E9charny <span dir=3D"ltr">&lt;<a =
href=3D"mailto:elecharny@gmail.com" target=3D"_blank">elecharny@gmail.com</=
a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi guys,<br>
<br>
I have done some tests a week ago on Mavibot partition. It appears that<br>
we have a serious performance issue as soon as the JVM memory is<br>
completly eaten, when adding some entries. What happens is that we start<br=
>
to discard some pages as they are hold by WeakReferences, and the GC has<br=
>
to do quite a expensive processing to get back some free memory. This<br>
slows down the add operation so much (by at least a factor 10) that it&#39;=
s<br>
not possible to keep going with WeakReferences (or SoftReferences).<br>
<br>
At this point, the best solution would be to use a cache to replace this<br=
>
mechanism.<br>
<br>
There are different things we can cache :<br>
o PageIo : it&#39;s useless, as we still have to deserialize them afterward=
<br>
o Nodes and Leaves : They contain references to other Nodes or Leaves,<br>
and for Leaves, reference to data. Holding the upper Nodes would save us<br=
>
a lot of time<br>
o Data : the ultimate objects to cache : having the object in cache<br>
spares us the time it takes to searcj it in a Btree<br>
<br>
A few other things to consider :<br>
o we need a versionned cache : all the elements may be versionned.<br>
o each BTree may be cached, with different configuation (some BTree may<br>
be used rarely, other may be heavily used, requiring a more aggressive<br>
cache configuration)<br>
o the Cache must not use locks, but CAS (Compare-And-Swap) for better<br>
performance<br>
<br>
At this point, I suggest using EhCache as a first approach, in order to<br>
leverage the cache we already use in the server, but we may want to<br>
design our own LRU cache (something that should not be too complex to<br>
implement).<br>
<br>
Thoughts ?<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
--<br>
Regards,<br>
Cordialement,<br>
Emmanuel L=E9charny<br>
<a href=3D"http://www.iktek.com" target=3D"_blank">www.iktek.com</a><br>
<br>
</font></span></blockquote></div><br></div>

--089e0122f5f6ccada704e569a8e4--