hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject RE: hbase mapreduce scan
Date Tue, 06 Apr 2010 11:34:53 GMT


J-D,

There's an alternative...

He could write a M/R that takes the input from a scan() , do something, reduce() and then
output the reduced set back to hbase in the form of a temp table.
(Even an in memory temp table) and then at the end pull the data out in to a hash table?

In theory this should be possible, but I haven't had time to play with in memory tables....

No?


Thx

-Mike

> Date: Mon, 5 Apr 2010 09:57:02 -0700
> Subject: Re: hbase mapreduce scan
> From: jdcryans@apache.org
> To: hbase-user@hadoop.apache.org
> 
> You want to put the result in a HashMap? MapReduce is a batch
> processing framework that runs multiple parallel JVMs over a cluster
> of machines so I don't see how you could simply output in a HashMap
> (unless you don't mind outputting on disk, then reading it back into a
> HashMap).
> 
> So I will guess that you want to do a live query against HBase, here
> MR won't help you since that is meant for bulk processing which
> usually takes more than a minute.
> 
> What you want to use is a Scan, using HTable. The unit tests have tons
> of example on how to use a scanner, look in the
> org.apache.hadoop.hbase.client package, so will find what you need.
> The main client package also contains some examples
> http://hadoop.apache.org/hbase/docs/r0.20.3/api/org/apache/hadoop/hbase/client/package-summary.html
> 
> J-D
> 
> On Sun, Apr 4, 2010 at 11:18 AM, Jürgen Jakobitsch <jakobitschj@punkt.at> wrote:
> > hi,
> >
> > i'm totally new to hbase and mapreduce and could really need some
> > pointer into the right direction for the following situation.
> >
> > i managed to run a basic mapreduce example - analog to Export.java
> > in the hbase.mapreduce package.
> >
> > what i need to achieve is the following :
> >
> > do a map/reduce scan on a hbase table and put the results
> > into a HashMap.
> >
> > could someone point me to an example.
> >
> > any help really appreciated
> >
> > wkr turnguard.com/turnguard
> >
> > --
> > punkt. netServices
> > ______________________________
> > Jürgen Jakobitsch
> > Codeography
> >
> > Lerchenfelder Gürtel 43 Top 5/2
> > A - 1160 Wien
> > Tel.: 01 / 897 41 22 - 29
> > Fax: 01 / 897 41 22 - 22
> >
> > netServices http://www.punkt.at
> >
> >
 		 	   		  
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message