hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Performance test results
Date Thu, 31 Mar 2011 17:27:57 GMT
Inline.

J-D

> I assume the block cache tunning key you talk about is
> "hfile.block.cache.size", right? If it is only 20% by default than
> what is the rest of the heap used for? Since there are no fancy
> operations like joins and since I'm not using memory tables the only
> thing I can think of is the memstore right? What is the recommended
> value for the block cache?

By default a max of 40% of the heap is reserved to MemStores, the rest
is used to answer queries, do compactions, flushes, etc. It's very
conservative, but people still find ways to OOME with very big cells
sometimes :)

>
> As for the regions layout, right now the table in discussion has 264
> regions more or less evenly distributed among the 5 region servers.
> Let me know what other information I can provide.

That's fine, but more important is the layout during the test. It can
be tricky to benchmark a "real life workload" if you just did them
import because it takes some time for the dust to settle. One example
among many others, the balancer only runs only every few minutes so if
you're doing a massive insert and then read, the load might only be on
two machines.

>
> The key space is as follows: I launch n threads, each thread writes
> keys that look like "streami_c" where "i" is the thread index (1-n)
> and "c" is a counter that goes up from 1 until I stop the test. I
> understand that each thread is only writing to the tail of its own
> keyspace so only "n" region files can be used, however if that was the
> limitation then adding more threads each with its own keyspace should
> have increased the throughput.

And can you tell by the start/stop keys that those threads do hit
different regions. I understand you wouldn't have to worry about that
too much in a real life scenario but since yours is artificial then
who knows how it ended up.

In order to speedup this discussion, feel free to drop by our IRC
channel on freenode, very often we're able to find issues much faster
using less time for everyone (and then report the findings here).

J-D

Mime
View raw message