Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of ekoontz@hiro-tan.org designates
 99.96.246.181 as permitted sender)
Message-ID: <4D682359.5040802@hiro-tan.org>
Date: Fri, 25 Feb 2011 13:47:05 -0800
From: Eugene Koontz <ekoontz@hiro-tan.org>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
 rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7
MIME-Version: 1.0
To: dev@hbase.apache.org
Subject: Re: what's the roadmap of secondary index of hbase?
References: 
 <4A2E351F553A2C4D89148CC937FA81820584E5@SC-MBX02-1.TheFacebook.com>
 <441192.96424.qm@web65501.mail.ac4.yahoo.com>
 <4A2E351F553A2C4D89148CC937FA8182058541@SC-MBX02-1.TheFacebook.com>
In-Reply-To: 
 <4A2E351F553A2C4D89148CC937FA8182058541@SC-MBX02-1.TheFacebook.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 2/25/11 12:43 PM, Jonathan Gray wrote:
> Cool.  Plans for a design phase that we can collaborate on?
>
Hi Jon,

I'm thinking that we could use a coprocessor that watches the 
Write-Ahead Log (using the WAL-edit operations  
https://issues.apache.org/jira/browse/HBASE-3257 "Coprocessors: Extend 
server side integration API to include HLog operations"). This 
coprocessor would write these edits, perhaps filtering or transforming 
them, and enqueing the results in a global queue. A separate process 
would be responsible for pulling operations off the queue and using 
HBase client operations to do the insert into a secondary index table 
appropriate for that operation.
     Perhaps we could use some of the work that the Lily people have 
done with HBase indexing (see 
http://www.lilyproject.org/lily/about/playground/hbaseindexes.html) in 
order to do the edit->hbase operation transformations and the secondary 
index table creation.
     -Eugene