hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HBase Review Board (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3287) Add option to cache blocks on hfile write and evict blocks on hfile close
Date Tue, 30 Nov 2010 23:20:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965487#action_12965487

HBase Review Board commented on HBASE-3287:

Message from: "Jonathan Gray" <jgray@apache.org>

bq.  On 2010-11-30 09:57:27, Ryan Rawson wrote:
bq.  > branches/0.90/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java, line 765
bq.  > <http://review.cloudera.org/r/1261/diff/1/?file=17902#file17902line765>
bq.  >
bq.  >     why would you not want to evict blocks from the cache on close?
bq.  stack wrote:
bq.      I think this a good point.  Its different behavior but its behavior we should have
always had?  One less option too.
bq.  Ryan Rawson wrote:
bq.      I'm still confused why we are adding config for something that we should always be
doing it.  While we'll never be zero conf, I am not seeing the reason why we'd want to keep
things in the LRU.  
bq.      It would make more sense not to evict on a split, but evict every other time, since
a split will probably reopen the same hfiles and need those blocks again.

I think it makes sense to have undocumented configuration parameters.  The default behavior
is then "the way" but having a config option checked in the code at least gives the opportunity
to turn something on/off without making a code change and redeploying completely.  In the
unit test, I'm turning it on/off with the config parameter so I can verify it works as expected.

And although I've changed the default to true, I'm not convinced that it always makes sense
in all cases.

Ryan came up with example of the split, though that would override the config parameter. 
But I think there could be other situations where you don't want to as well.

In any case, I want to keep it configurable so I can turn it on/off between test runs and
see what, if any, difference these optimizations make and IMO there's very little cost associated
with using conf.getBoolean("some.undocumented.thing", true) vs. a hard-coded true (if there's
any possibility you might want to change the behavior).

- Jonathan

This is an automatically generated e-mail. To reply, visit:

> Add option to cache blocks on hfile write and evict blocks on hfile close
> -------------------------------------------------------------------------
>                 Key: HBASE-3287
>                 URL: https://issues.apache.org/jira/browse/HBASE-3287
>             Project: HBase
>          Issue Type: New Feature
>          Components: io, regionserver
>    Affects Versions: 0.90.0
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>             Fix For: 0.92.0
>         Attachments: HBASE-3287-FINAL-trunk.patch
> This issue is about adding configuration options to add/remove from the block cache when
creating/closing files.  For use cases with lots of flushing and compacting, this might be
desirable to prevent cache misses and maximize the effective utilization of total block cache
> The first option, {{hbase.rs.cacheblocksonwrite}}, will make it so we pre-cache blocks
as we are writing out new files.
> The second option, {{hbase.rs.evictblocksonclose}}, will make it so we evict blocks when
files are closed.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message