hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Booth <jaybo...@gmail.com>
Subject Re: Heap fragmentation
Date Thu, 20 Jan 2011 20:18:26 GMT
Do you even need to go unsafe?  Say you made the memstore a
closed-loop system..  get allocated 512MB or 1GB of buffers at
startup, and then whenever any buffer is flushed, it gets pushed onto
the bufferQueue.  New buffers only come from bufferQueue, no new
byte[] calls.  If there are no buffers available in the queue, and you
need to write, trigger a flush.  That means it'll be the same buffers,
which won't create new heap fragmentation as they'll presumably
migrate down to the bottom of old gen heap after the first major
compaction or two..  you're just passing references to them through
that queue.  I haven't look at the code but this is probably
reasonably pluggable with whatever safeguards you guys already have to
flush before allocing if you're close to max memory.

On Thu, Jan 20, 2011 at 3:14 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> Bigmemory serializes java objects then stores the serialized form in
> DirectByteBuffers, which isn't quite exactly what we want...
> Needless to say a number of us have been poking at the DBB and Unsafe
> APIs and have a few ideas :-)
> On Thu, Jan 20, 2011 at 10:39 AM, Todd Lipcon <todd@cloudera.com> wrote:
>> On Thu, Jan 20, 2011 at 10:34 AM, Vladimir Rodionov <vrodionov@carrieriq.com
>>> wrote:
>>> BigMemory anyone?
>>> http://www.terracotta.org/bigmemory
>> Yea, something like that, minus the commercial license ;-)
>> -Todd
>> ____
>>> From: Todd Lipcon [todd@cloudera.com]
>>> Sent: Thursday, January 20, 2011 9:33 AM
>>> To: dev
>>> Subject: Heap fragmentation
>>> I did some experiments to understand our full GC issues better last night.
>>> Here are the results:
>>> http://people.apache.org/~todd/hbase-fragmentation/
>>> <http://people.apache.org/~todd/hbase-fragmentation/>Basically my
>>> conclusion
>>> here is that (for these YCSB workloads) the memstore is way worse for heap
>>> fragmentation than the block cache.
>>> Also we now have some tools and reference for comparison if we make any
>>> changes to memstore to try to combat this issue.
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera

View raw message