hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: hbase.bucketcache.bucket.sizes had set multiple of 1024 but still got "Invalid HFile block magic"
Date Tue, 28 Nov 2017 11:36:34 GMT
FYI

It has to be multiples of 256.

<property>
    <name>hbase.bucketcache.bucket.sizes</name>
    <value></value>
    <description>A comma-separated list of sizes for buckets for the
bucketcache.
    Can be multiple sizes. List block sizes in order from smallest to largest.
    The sizes you use will depend on your data access patterns.
    Must be a multiple of 256 else you will run into
    'java.io.IOException: Invalid HFile block magic' when you go to
read from cache.
    If you specify no values here, then you pick up the default bucketsizes set
    in code (See BucketAllocator#DEFAULT_BUCKET_SIZES).
  </description>
  </property>

-Anoop-

On Sat, Nov 25, 2017 at 7:53 PM, 苏国东 <greatwall070@126.com> wrote:
> Hi:
>
> Class BucketEntry calculate offset like as :
> long offset()
> { // Java has no unsigned numbers long o = ((long) offsetBase) & 0xFFFFFFFF; o +=
(((long) (offset1)) & 0xFF) << 32; return o << 8; }
> private void setOffset(long value)
> { assert (value & 0xFF) == 0; value >>= 8; offsetBase = (int) value; offset1
= (byte) (value >> 32); }
> If offset not multiple of 1024 , the method offset() will return wrong value. if bucketsizes
must be multiple of 1024, probably waste memory .
>
> 198211 is not multiple of 1024
>
>
> At 2017-11-25 20:52:10, "Weizhan Zeng" <qgweizhan@gmail.com> wrote:
>>Hi , guys
>>   In https://issues.apache.org/jira/browse/HBASE-16993 <https://issues.apache.org/jira/browse/HBASE-16993>
, I found that
>>
>>hbase.bucketcache.bucket.sizes must set multiple of 1024, But when I set
>>
>>  <property>
>>    <name>hbase.bucketcache.bucket.sizes</name>
>>    <value>6144,9216,41984,50176,58368,66560,99328,132096,198211,263168,394240,525312,1049600,2099200</value>
>>  </property>
>>
>>And I still got  error :
>>
>>
>>2017-11-25 20:37:37,222 ERROR [B.defaultRpcServer.handler=20,queue=1,port=60020] bucket.BucketCache:
Failed reading block d444ab4b244140c199f23a3870f59136_250591965 from bucket cache
>>java.io.IOException: Invalid HFile block magic: \x00\x00\x00\x00\x00\x00\x00\x00
>>       at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>>       at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>>       at org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:275)
>>       at org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:136)
>>       at org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:123)
>>       at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:428)
>>       at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
>>       at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:278)
>>       at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:418)
>>       at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
>>       at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:649)
>>       at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:599)
>>       at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:268)
>>       at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:173)
>>       at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350)
>>       at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:199)
>>       at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2077)
>>       at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5556)
>>       at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2574)
>>       at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2560)
>>       at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2541)
>>       at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6830)
>>       at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6809)
>>       at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2049)
>>       at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>>       at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
>>       at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>>       at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
>>       at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>>       at java.lang.Thread.run(Thread.java:748)
>>
>>
>>Is there anything I missed ?

Mime
View raw message