Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
From: "David G. Boney" <chair@austin-acm-sigkdd.org>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_940D9953-1669-43BB-8F22-59F60752E04B"
Subject: Bloom filter based scanner/filter - repost
Message-Id: <72D1C755-EF52-4889-AED1-778C978BFEB2@austin-acm-sigkdd.org>
Date: Tue, 15 Jan 2013 14:41:48 -0600
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))

--Apple-Mail=_940D9953-1669-43BB-8F22-59F60752E04B
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Sorry, I hit the send button before finishing the message.

I am building a data cube on top of HBase. All access to the data is by =
map/reduce jobs. I want to build a scanner where its first matching =
criteria is based on the set intersection of bloom filters, followed by =
additional matching criteria specified in the current filter =
architecture. First, I run a map/reduce job on table A. For every row I =
match in table A, I add the row key to a bloom filter. I then do a =
map/reduce job on table B, where the row keys are over the same domain =
as table A. I want to build a scanner that can use the builtin Bloom =
filters in HBase. When the scanner goes to get the block of data to =
which a row key based bloom filter is attached, it does a set =
intersection with the table A bloom filter to see if any of the keys =
from Table A are in the block. If so, the block is read in and the the =
scanner does addition matching on the rows according to the filter.

This is a simplification of my problem. I am trying to find out what the =
complexity of implementing such a feature would be in HBase.
-----------------
Sincerely,
David G. Boney
Chair, Austin ACM SIGKDD
chair@austin-acm-sigkdd.org
http://www.meetup.com/Austin-ACM-SIGKDD/
http://tech.groups.yahoo.com/group/austinsigkdd/


--Apple-Mail=_940D9953-1669-43BB-8F22-59F60752E04B--