hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: ConcurrentSkipListMap performance in the MemStore
Date Wed, 15 Jun 2011 02:19:10 GMT
Woah. That looks promising Jason.  Can you hack up TestMemStore.java
and have it see if above will even pass the CSLM hbase tests?

St.Ack

On Tue, Jun 14, 2011 at 5:23 PM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
> Here's the implementation we can presumably drop in and benchmark:
>
> https://github.com/mspiegel/lockfreeskiptree
>
> On Tue, Jun 14, 2011 at 4:56 PM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
>> Here's a paper on a ConcurrentSkipTree:
>>
>> http://www.cs.virginia.edu/~ms6ep/publications/michael-spiegel-dissertation.pdf
>>
>> On Tue, Jun 14, 2011 at 4:40 PM, Jason Rutherglen
>> <jason.rutherglen@gmail.com> wrote:
>>> Right, I think one naive approach would be to take CSLM, convert it to
>>> use AtomicIntegerArray instead of objects (basically emulate objects
>>> in blocks of ints), and of course not implement delete functionality.
>>>
>>> The first challenge is making an AtomicIntegerArray based linked list
>>> where the value is pointed to by an int into a byte block data
>>> structure.  The next is a 2nd AIA that emulates the internal 'Index'
>>> class of CSLM.
>>>
>>> Thoughts?
>>>
>>> On Tue, Jun 14, 2011 at 3:51 PM, Stack <stack@duboce.net> wrote:
>>>> Or do as Todd suggested a while back and pull in CSLM and hack it to
>>>> suit our needs; its public domain.
>>>> St.Ack
>>>>
>>>> On Tue, Jun 14, 2011 at 3:23 PM, Jason Rutherglen
>>>> <jason.rutherglen@gmail.com> wrote:
>>>>>> See this J: http://www.cloudera.com/blog/2011/03/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-3/
>>>>>
>>>>> Interesting.
>>>>>
>>>>> In Lucene for realtime search we have the same problem where the
>>>>> best/only option right now for the realtime terms dictionary is to use
>>>>> ConcurrentSkipListMap.  However the Node, key, value, and Index
>>>>> objects all are pointers that consume RAM and need to be cleaned up.
>>>>>
>>>>> Lucene right now during indexing does not generate much object pointer
>>>>> garbage.  We have the same problem in the two projects.
>>>>>
>>>>> Maybe there's a way to create an AtomicIntegerArray based system that
>>>>> uses int pointers to perform the same function as a the concurrent
>>>>> skip list.  It would greatly reduce garbage generation.
>>>>>
>>>>> On Tue, Jun 14, 2011 at 10:14 AM, Stack <stack@duboce.net> wrote:
>>>>>> See this J: http://www.cloudera.com/blog/2011/03/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-3/
>>>>>> St.Ack
>>>>>>
>>>>>> On Mon, Jun 13, 2011 at 10:58 PM, Jason Rutherglen
>>>>>> <jason.rutherglen@gmail.com> wrote:
>>>>>>> Is there an issue with garbage generated from using CSLM in the
MemStore?
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message