hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15240) Go Big BucketCache Fixes
Date Wed, 10 Feb 2016 14:19:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140843#comment-15140843
] 

stack commented on HBASE-15240:
-------------------------------

The 100000 upper limit per file has been in there since we first did reporting on blockcache:
  "HBASE-4089 blockCache contents report"

Prefetch is too slow. Takes ages to warm a big cache.

There is an NPE the odd time here abouts. Need to investigate:

{code}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
index 4ada262..834588e 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
@@ -1502,7 +1502,7 @@ public class HFileBlock implements Cacheable {
       // guaranteed to use hdfs checksum verification.
       boolean doVerificationThruHBaseChecksum = streamWrapper.shouldUseHBaseChecksum();
       FSDataInputStream is = streamWrapper.getStream(doVerificationThruHBaseChecksum);
-
+      if (is == null) throw new NullPointerException(toString());
       HFileBlock blk = readBlockDataInternal(is, offset,
                          onDiskSizeWithHeaderL,
                          uncompressedSize, pread,
{code}

Need below to be able to load a bunch into cache by prefetching:

{code}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
index f9d8167..c6fceea 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
@@ -154,11 +154,11 @@ public class BucketCache implements BlockCache, HeapSize {
   /** Cache access count (sequential ID) */
   private final AtomicLong accessCount = new AtomicLong(0);

-  private static final int DEFAULT_CACHE_WAIT_TIME = 50;
+  private static final int DEFAULT_CACHE_WAIT_TIME = 5000;
   // Used in test now. If the flag is false and the cache speed is very fast,
   // bucket cache will skip some blocks when caching. If the flag is true, we
   // will wait blocks flushed to IOEngine for some time when caching
-  boolean wait_when_cache = false;
+  boolean wait_when_cache = true;

   private final BucketCacheStats cacheStats = new BucketCacheStats();
{code}

Need to think about it.

Also needed these conffigs to do more work serializing into bucketcache.

{code}
<name>hbase.ui.blockcache.by.file.max
</name>
<value>2147483640</value>
</property>
<property>
<name>hbase.bucketcache.writer.queuelength
</name>
<value>256</value>
</property>
<property>
<name>hbase.bucketcache.writer.threads
</name>
<value>16</value>
</property>
{code}

Finally, we ask that blocks be 4k in configs but we are writing out 4.2k -- the header of
following block too. On the other end, fetching a block seems to be making for two seeks.
Investigate

> Go Big BucketCache Fixes
> ------------------------
>
>                 Key: HBASE-15240
>                 URL: https://issues.apache.org/jira/browse/HBASE-15240
>             Project: HBase
>          Issue Type: Umbrella
>          Components: BucketCache
>            Reporter: stack
>            Assignee: stack
>
> Umbrella issue to which we will attach issues that prevent bucketcache going big; there's
a few.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message