jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Mehrotra <chetan.mehro...@gmail.com>
Subject Using DirectMemory as second level cache
Date Mon, 08 Jul 2013 10:54:02 GMT
Hi,

I tried to integrate Apache DirectMemory as Second level cache (L2).
It uses Kryo  [2] for serializing/derserializing objects from the off
heap memory managed by DirectMemory. Initial test based on this show
quite a bit of saving in terms of memory taken

Below stats (sum of all MongoMK related cache) are taken when L2
memory is enabled after restarting a CQ instance and some basic access
done.

On Heap
    No of entries    - 138288
    Memory Taken - 217.3 MB
Off Heap
    No of entries    - 113012
    Memory Taken - 42.8 MB
    Hit Count 1257
    Total Load Time 80253030
    Avg Load Time 63844.8926014320

If I completely serialize the Document cache (the biggest of 3 )
- Document Cache - On Heap 134.7 MB for 43935 entries
- Document Cache - Off Heap 27.5 MB for 43935 entries. Takes 555.5 ms
to deserialize all 43935 entries

So far I can make following observation
-- Storing cache objects in serialized form provides a more compact storage
-- Deserializtion cost is low
-- Using Off heap memory would be helpful compared to on heap memory
-- Even if we do not use DirectMemory we can look into storing the
data in serialized form
-- L2 cache can be added as an optional feature. Oak Core can still be
used in absence of L2 cache

Caveats
-- DirectMemory project is still in very early phase and I have to use
the latest snapshot version with some local fixes to get it work

The required changes to be done can be checked at [3]

So would it be worthwhile to add this as an optional feature in Oak.
If user require they can enable L2 cache otherwise we only use the
Guava Cache as L1 cache

Thoughts?

Chetan Mehrotra

[1] http://directmemory.apache.org/
[2] https://code.google.com/p/kryo/
[3] https://github.com/chetanmeh/jackrabbit-oak/compare/apache:ab955ffd40e23a9f8aee87f9076bc045b643e35d...offheap

Mime
View raw message