hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: OOME during frequent updates...
Date Wed, 09 Jun 2010 19:28:17 GMT
> Am getting back to using Java after a long time, guys.. So, give me a little more time
to ramp up to '10 :)

Welcome back!

>
> One more question then: What are the implications of running really large regions (like
around 4-8 gigs per region)? One implication I can think of is coarser grained control over
load (since a split will happen less frequently).. But with a large number of nodes, this
isnt that coarse-grained I guess?

I don't know anybody who's that high up, here we run at 1GB on our
table that has a few TBs. But yeah, at scale that stuff won't matter
as much, but with 8GB you could blow out your memory.

>
> We are trying to load 100's of terabytes eventually.. And running even 100s of regions
per RS seems like a big hit on the memory.

>From what I saw in your metrics dump, storing the actual regions was
costing you almost nothing. But, you will encounter problems when the
global size of all memstores is getting very big (a true random write
pattern will always get you there).

IMO, the biggest issue with the number of regions served per RS is
more about the actual data that is stored and retrieved WRT the
performance each of your node can deliver (capacity planning).

J-D

Mime
View raw message