hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15240) Go Big BucketCache Fixes
Date Wed, 10 Feb 2016 14:19:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140843#comment-15140843

stack commented on HBASE-15240:

The 100000 upper limit per file has been in there since we first did reporting on blockcache:
  "HBASE-4089 blockCache contents report"

Prefetch is too slow. Takes ages to warm a big cache.

There is an NPE the odd time here abouts. Need to investigate:

diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
index 4ada262..834588e 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
@@ -1502,7 +1502,7 @@ public class HFileBlock implements Cacheable {
       // guaranteed to use hdfs checksum verification.
       boolean doVerificationThruHBaseChecksum = streamWrapper.shouldUseHBaseChecksum();
       FSDataInputStream is = streamWrapper.getStream(doVerificationThruHBaseChecksum);
+      if (is == null) throw new NullPointerException(toString());
       HFileBlock blk = readBlockDataInternal(is, offset,
                          uncompressedSize, pread,

Need below to be able to load a bunch into cache by prefetching:

diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
index f9d8167..c6fceea 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
@@ -154,11 +154,11 @@ public class BucketCache implements BlockCache, HeapSize {
   /** Cache access count (sequential ID) */
   private final AtomicLong accessCount = new AtomicLong(0);

-  private static final int DEFAULT_CACHE_WAIT_TIME = 50;
+  private static final int DEFAULT_CACHE_WAIT_TIME = 5000;
   // Used in test now. If the flag is false and the cache speed is very fast,
   // bucket cache will skip some blocks when caching. If the flag is true, we
   // will wait blocks flushed to IOEngine for some time when caching
-  boolean wait_when_cache = false;
+  boolean wait_when_cache = true;

   private final BucketCacheStats cacheStats = new BucketCacheStats();

Need to think about it.

Also needed these conffigs to do more work serializing into bucketcache.


Finally, we ask that blocks be 4k in configs but we are writing out 4.2k -- the header of
following block too. On the other end, fetching a block seems to be making for two seeks.

> Go Big BucketCache Fixes
> ------------------------
>                 Key: HBASE-15240
>                 URL: https://issues.apache.org/jira/browse/HBASE-15240
>             Project: HBase
>          Issue Type: Umbrella
>          Components: BucketCache
>            Reporter: stack
>            Assignee: stack
> Umbrella issue to which we will attach issues that prevent bucketcache going big; there's
a few.

This message was sent by Atlassian JIRA

View raw message