hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <la...@apache.org>
Subject Re: Coprocessors
Date Thu, 25 Apr 2013 22:06:54 GMT
It doesn't. Based on your query and key-layout it only scans subranges of the keyspace and
these scans are parallelized across region servers.
I'll let James explain more (if he's listening) :)

-- Lars

 From: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) <skadambi@bloomberg.net>
To: user@hbase.apache.org 
Sent: Thursday, April 25, 2013 2:57 PM
Subject: Re: Coprocessors

Thanks Lars. I briefly looked into Phoenix but it appeared to do full-table scans to perform
the aggregation. The same goes with Impala. If you think otherwise, I'll look into it again.

----- Original Message -----
From: user@hbase.apache.org
To: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN), user@hbase.apache.org
At: Apr 25 2013 17:54:48

You might want to have a look at Phoenix (https://github.com/forcedotcom/phoenix), which does
that and more, and gives a SQL/JDBC interface.

-- Lars

From: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) <skadambi@bloomberg.net>
To: user@hbase.apache.org 
Sent: Thursday, April 25, 2013 2:44 PM
Subject: Coprocessors


This is my first post on the HBase user mailing list. 

I have the following scenario:
I've a HBase table of upto a billion keys. I'm looking to support an application where on
some user action, I'd need to fetch multiple columns for upto 250K keys and do some sort of
aggregation on it. Fetching all that data and doing the aggregation in my application takes
about a minute.

I'm looking to co-locate the aggregation logic with the region servers to
a. Distribute the aggregation
b. Avoid having to fetch large amounts of data over the network (this could potentially be

Neither observers nor aggregation endpoints work for this use case. Observers don't return
data back to the client while aggregation endpoints work in the context of scans not a multi-get
(Are these correct assumptions?).

I'm looking to write a service that runs alongside the region servers and acts a proxy b/w
my application and the region servers. 

I plan to use the logic in HBase client's HConnectionManager, to segment my request of 1M
rowkeys into sub-requests per region-server. These are sent over to the proxy which fetches
the data from the region server, aggregates locally and sends data back. Does this sound reasonable
or even a useful thing to pursue?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message