Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of jason.rutherglen@gmail.com
 designates 209.85.212.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=arWVSrX1uVpU6MsxVgIpL7llHonxQWEgO6o6xiDhkJhXri+bPY4M7eatvWODqg7ZlY
         dq6v6H+nyFcwy7VhgAxlg5+unFNj+545YbOVZUfKoP1dU5sM8WKn2F7pVdIt4Rroy97+
         2CQxDMXrtes+n+kK+IQNOHCiHCmsBuq+T4vtA=
MIME-Version: 1.0
In-Reply-To: <BANLkTimakaENMBjcLeZqm-med7RNSd5Bjg@mail.gmail.com>
References: <BANLkTinNp46-VOvmj2R2WoaDr8TF1MS-yA@mail.gmail.com>
	<BANLkTimakaENMBjcLeZqm-med7RNSd5Bjg@mail.gmail.com>
Date: Tue, 14 Jun 2011 15:23:40 -0700
Message-ID: <BANLkTin_qEXp_YDGbezLDcqxrsuUpzjMMA@mail.gmail.com>
Subject: Re: ConcurrentSkipListMap performance in the MemStore
From: Jason Rutherglen <jason.rutherglen@gmail.com>
To: dev@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

> See this J: http://www.cloudera.com/blog/2011/03/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-3/

Interesting.

In Lucene for realtime search we have the same problem where the
best/only option right now for the realtime terms dictionary is to use
ConcurrentSkipListMap.  However the Node, key, value, and Index
objects all are pointers that consume RAM and need to be cleaned up.

Lucene right now during indexing does not generate much object pointer
garbage.  We have the same problem in the two projects.

Maybe there's a way to create an AtomicIntegerArray based system that
uses int pointers to perform the same function as a the concurrent
skip list.  It would greatly reduce garbage generation.

On Tue, Jun 14, 2011 at 10:14 AM, Stack <stack@duboce.net> wrote:
> See this J: http://www.cloudera.com/blog/2011/03/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-3/
> St.Ack
>
> On Mon, Jun 13, 2011 at 10:58 PM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
>> Is there an issue with garbage generated from using CSLM in the MemStore?
>>
>