Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 209.85.213.169 as permitted sender)
MIME-Version: 1.0
Sender: jdcryans@gmail.com
Date: Mon, 12 Dec 2011 17:47:39 -0800
Message-ID: 
 <CAGpTDNdKvtZcg=8yn66pmq3phSvfp33=zknmSCEDZdCbmpGz9w@mail.gmail.com>
Subject: Reconsidering the default block cache size because of HFileV2
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: dev@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey devs,

A thought just stuck me while I was writing down a more detailed block
caching documentation: with HFileV2, the indexes now live in the block
cache which means that those who upgrade may all of a sudden get
terrible cache hit ratios because of all that memory taken by the
indexes. This is somewhat mitigated by the fact that people don't
usually need to keep _all_ the index blocks in memory so in the end
we're more efficient.

Which brings me to a question: should we set hfile.block.cache.size
higher since indexes are now kept in the block cache? Currently it's
set to 20%.

Looking over my own production machines I see that the
storefileIndexSize is around 600-700MB so that's potentially how much
more data I'd have to block cache (more likely it's half of that
that's really being used actively).

What would be a good new default? 25%? 30%? How do we handle those
that will be pushed over the BC+memstore size limit because of that
change?

Thx,

J-D