Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com
 designates 65.55.34.91 as permitted sender)
Message-ID: <COL117-W3295223279AADDEAA4E8E48F540@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_d61993fb-fcb0-4063-a7eb-b4ca0c2a06cc_"
From: Michael Segel <michael_segel@hotmail.com>
To: <user@hbase.apache.org>
Subject: RE: Using external indexes in an HBase Map/Reduce job...
Date: Tue, 12 Oct 2010 11:13:37 -0500
Importance: Normal
In-Reply-To: <AANLkTi=UXpC4JFVx9hQsyi9H1VN48YJVFUswX-nyJiBq@mail.gmail.com>
References: 
 <COL117-W108E0BD11DADF3C00F36E18F540@phx.gbl>,<AANLkTi=UXpC4JFVx9hQsyi9H1VN48YJVFUswX-nyJiBq@mail.gmail.com>
MIME-Version: 1.0

--_d61993fb-fcb0-4063-a7eb-b4ca0c2a06cc_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Thanks for the reply...

That's not exactly what I'm looking for...

Suppose you have an exterior system which provides you the list of row keys=
 you want.=20
What ever that system is.

So you have a java list object and you want to do a M/R based on input from=
 a Java List.

What's the best way to do it?


> From: octo47@gmail.com
> Date: Tue=2C 12 Oct 2010 16:54:00 +0400
> Subject: Re: Using external indexes in an HBase Map/Reduce job...
> To: user@hbase.apache.org
>=20
> Hi Michael Segel.
>=20
> If I understand your question correctrly=2C you looking for optimal way
> for scanning
> index search results? If not=2C my answer below is not relevant :).
>=20
> 1. For mr joins or large index results scan bloom filters can be used
> like described here
> http://blog.rapleaf.com/dev/2009/09/25/batch-querying-with-cascading/
>=20
> 2. Another option: denormalize data in same or separate table.
> (depends on nature of object relations).
>=20
> 3. Random gets. For each row from solr issue random get. (for really
> small result sets or paging).
>=20
> 4. Put compacted data (latest data=2C small subset of data etc) into solr=
 index.
>=20
>=20
> 2010/10/12 Michael Segel <michael_segel@hotmail.com>:
> >
> > Hi=2C
> >
> > Now I realize that most everyone is sitting in NY=2C while some of us c=
an't leave our respective cities....
> >
> > Came across this problem and I was wondering how others solved it.
> >
> > Suppose you have a really large table with 1 billion rows of data.
> > Since HBase really doesn't have any indexes built in (Don't get me star=
ted about the contrib/transactional stuff...)=2C you're forced to use some =
sort of external index=2C or roll your own index table.
> >
> > The net result is that you end up with a list object that contains your=
 result set.
> >
> > So the question is... what's the best way to feed the list object in?
> >
> > One option I thought about is writing the object to a file and then usi=
ng it as the file in and then control the splitters. Not the most efficient=
 but it would work.
> >
> > Was trying to find a more 'elegant' solution and I'm sure that anyone u=
sing SOLR or LUCENE or whatever... had come across this problem too.
> >
> > Any suggestions?
> >
> > Thx
> >
> >
 		 	   		  =

--_d61993fb-fcb0-4063-a7eb-b4ca0c2a06cc_--