hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Reconsidering the default block cache size because of HFileV2
Date Tue, 13 Dec 2011 01:47:39 GMT
Hey devs,

A thought just stuck me while I was writing down a more detailed block
caching documentation: with HFileV2, the indexes now live in the block
cache which means that those who upgrade may all of a sudden get
terrible cache hit ratios because of all that memory taken by the
indexes. This is somewhat mitigated by the fact that people don't
usually need to keep _all_ the index blocks in memory so in the end
we're more efficient.

Which brings me to a question: should we set hfile.block.cache.size
higher since indexes are now kept in the block cache? Currently it's
set to 20%.

Looking over my own production machines I see that the
storefileIndexSize is around 600-700MB so that's potentially how much
more data I'd have to block cache (more likely it's half of that
that's really being used actively).

What would be a good new default? 25%? 30%? How do we handle those
that will be pushed over the BC+memstore size limit because of that



View raw message