hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Problem during hbase read
Date Wed, 05 Jul 2017 15:00:10 GMT
(What version of hbase?)

Unfortunately the exception does not print the file name. Can you figure
which file or region it is? Could try reading content w/ hfile tool to see
if it complains also.

Problem is:

On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
    at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)

I see the above exception in HBASE-13830 where Nils perhaps fixes it by
upgrading hbase (see end comment).

St.Ack



On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <mukundmurrali9@gmail.com>
wrote:

> Hi,
>
> Recently we faced an issue in our HBase Cluster and following is the
> exception trace in client. Any idea what could be the cause and how to
> rectify this?
>
> RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
> java.io.IOException: java.io.IOException: Could not reseek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b
> ,
> cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
> 1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea-
> 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
>     at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:217)
>     at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> enforceSeek(StoreFileScanner.java:408)
>     at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> pollRealKV(KeyValueHeap.java:363)
>     at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:123)
>     at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:588)
>     at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:150)
>     at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> populateResult(HRegion.java:5731)
>     at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> nextInternal(HRegion.java:5894)
>     at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.nextRaw(HRegion.java:5668)
>     at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5645)
>     at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5631)
>     at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
>     at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
>     at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> get(RSRpcServices.java:2049)
>     at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>     at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:133)
>     at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>     at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: On-disk size without header provided is
> 36901, but block header contains 859321901. Block offset: 440164791, data
> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
>     at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>     at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
> HFileBlock.java:88)
>     at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockDataInternal(HFileBlock.java:1665)
>     at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockData(HFileBlock.java:1532)
>     at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> HFileReaderV2.java:452)
>     at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
> loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
>     at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.seekTo(HFileReaderV2.java:649)
>     at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
>     at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
> reseekTo(HalfStoreFileReader.java:293)
>     at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseekAtOrAfter(StoreFileScanner.java:288)
>     at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:200)
>     ... 19 more
>
> ThanksU
> Regards,
>
> Mukund Murrali
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message