jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tommaso Teofili <teof...@adobe.com>
Subject Re: Using DirectMemory as second level cache
Date Tue, 09 Jul 2013 09:45:54 GMT

On 08/lug/2013, at 12:54, Chetan Mehrotra wrote:

> Hi,
> 
> I tried to integrate Apache DirectMemory as Second level cache (L2).
> It uses Kryo  [2] for serializing/derserializing objects from the off
> heap memory managed by DirectMemory. Initial test based on this show
> quite a bit of saving in terms of memory taken
> 
> Below stats (sum of all MongoMK related cache) are taken when L2
> memory is enabled after restarting a CQ instance and some basic access
> done.
> 
> On Heap
>    No of entries    - 138288
>    Memory Taken - 217.3 MB
> Off Heap
>    No of entries    - 113012
>    Memory Taken - 42.8 MB
>    Hit Count 1257
>    Total Load Time 80253030
>    Avg Load Time 63844.8926014320
> 
> If I completely serialize the Document cache (the biggest of 3 )
> - Document Cache - On Heap 134.7 MB for 43935 entries
> - Document Cache - Off Heap 27.5 MB for 43935 entries. Takes 555.5 ms
> to deserialize all 43935 entries
> 
> So far I can make following observation
> -- Storing cache objects in serialized form provides a more compact storage
> -- Deserializtion cost is low
> -- Using Off heap memory would be helpful compared to on heap memory
> -- Even if we do not use DirectMemory we can look into storing the
> data in serialized form
> -- L2 cache can be added as an optional feature. Oak Core can still be
> used in absence of L2 cache
> 
> Caveats
> -- DirectMemory project is still in very early phase and I have to use
> the latest snapshot version with some local fixes to get it work
> 
> The required changes to be done can be checked at [3]
> 
> So would it be worthwhile to add this as an optional feature in Oak.
> If user require they can enable L2 cache otherwise we only use the
> Guava Cache as L1 cache
> 
> Thoughts?

I think it makes sense, probably as an optional feature with a pluggable implementation (can
be DM or some other mechanism).
Regarding DM I think I the community would be happy to support our use case (and I'd be happy
to help on that).

Regards,
Tommaso

> 
> Chetan Mehrotra
> 
> [1] http://directmemory.apache.org/
> [2] https://code.google.com/p/kryo/
> [3] https://github.com/chetanmeh/jackrabbit-oak/compare/apache:ab955ffd40e23a9f8aee87f9076bc045b643e35d...offheap


Mime
View raw message