hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mukund murrali <mukundmurra...@gmail.com>
Subject Problem during hbase read
Date Wed, 05 Jul 2017 08:00:50 GMT
Hi,

Recently we faced an issue in our HBase Cluster and following is the
exception trace in client. Any idea what could be the cause and how to
rectify this?

RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
java.io.IOException: java.io.IOException: Could not reseek
StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b,
cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea-
45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
    at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:217)
    at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
enforceSeek(StoreFileScanner.java:408)
    at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
pollRealKV(KeyValueHeap.java:363)
    at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:123)
    at org.apache.hadoop.hbase.regionserver.StoreScanner.
next(StoreScanner.java:588)
    at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:150)
    at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
populateResult(HRegion.java:5731)
    at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
nextInternal(HRegion.java:5894)
    at org.apache.hadoop.hbase.regionserver.HRegion$
RegionScannerImpl.nextRaw(HRegion.java:5668)
    at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5645)
    at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5631)
    at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
    at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.
get(RSRpcServices.java:2049)
    at org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
    at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
RpcExecutor.java:133)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
    at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
    at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
HFileBlock.java:88)
    at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockDataInternal(HFileBlock.java:1665)
    at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockData(HFileBlock.java:1532)
    at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
HFileReaderV2.java:452)
    at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
    at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.seekTo(HFileReaderV2.java:649)
    at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
    at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
reseekTo(HalfStoreFileReader.java:293)
    at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseekAtOrAfter(StoreFileScanner.java:288)
    at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:200)
    ... 19 more

ThanksU
Regards,

Mukund Murrali

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message