hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sujee Maniyam <su...@sujee.net>
Subject reading a row with lots of cells (wide-table) causing out-of-memory error
Date Wed, 02 Dec 2009 07:06:26 GMT
Hi all,

I have the following table
     user_id   => {  "ip_address",   "ref_url" }
column qualifiers are timestamps.  Created with default options
(BLOCKSIZE => '65536', ...etc)

so a typical row looks like:
     'user1'  => {
                           ip_address:t1  => value1
                           ip_address:t2  => value2
                           ref_url:t2 =>  value3
                     }
I have a few million rows in the table.  Trying to write a simple java client.

When I query for a user_id that has around 2-million 'values' (unique
timestamps) it is causing a region server to die with Out-of-memory
error.

code-snippet for client:

// ----------
// ---- http://pastebin.com/m75fc75d1

Get get = new Get(key);
Result r = table.get(get);

String[] families = {"ip_address", "ref_url"};
for (String family : families) {
   NavigableMap<byte[], byte[]> familyMap =
r.getFamilyMap(Bytes.toBytes(family));
   System.out.println(String.format("    %s #cells : %d", family,
familyMap.size()));
}
// ----------


I am curious to know...
1) is the above code doing some thing wrong?
2) does a row data has to completely fit  into memory?
3) I will want to iterate through all the cell values, wondering what
is the best way to do that?
4) if this is the limitation for 'wide tables', then I will redesign
to table to use composite keys ( row = userid + timestamp)

thanks so much for your help.
Sujee Maniyam

-- 
http://sujee.net

Mime
View raw message