hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From llpind <sonny_h...@hotmail.com>
Subject Loading large resultset into HBase
Date Fri, 15 May 2009 20:11:29 GMT

Hey all,

I'm loading data from a DB into HBase.  I have a single java process
iterating over a ResultSet.  After about 10,000 rows i do a BatchUpdate. 
I've changed the Heap size of both Hadoop & HBase to 2000.  

Setup:  0.19.1.  1 box with master and secondary.  3 boxes with
HRegionServer.

Problem 1:  The load seems to be unblananced:  
 Address Start Code Load 
1:60020 1242415770566 requests=0, regions=1, usedHeap=39, maxHeap=1777 
2:60020 1242415770417 requests=3, regions=3, usedHeap=52, maxHeap=1777 
3:60020 1242415770273 requests=1, regions=3, usedHeap=604, maxHeap=1777 
Total:  servers: 3   requests=4, regions=7 

Problem 2:  Around 10 Million rows, the upload starts to slow down.  

The upload is still going, so i'll update on what happens.  
-- 
View this message in context: http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23566568.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message