hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: How to set Timeout for get/scan operations without impacting others
Date Tue, 19 May 2015 01:29:04 GMT
bq. Caused by: java.io.IOException: Invalid HFile block magic:
\x00\x00\x00\x00\x00\x00\x00\x00

Looks like you have some corrupted HFile(s) in your cluster - which should
be fixed first.

Which hbase release are you using ?
Do you use data block encoding ?

You can use http://hbase.apache.org/book.html#_hfile_tool to do some
investigation.

Cheers

On Mon, May 18, 2015 at 6:19 PM, Fang, Mike <CHUFANG@paypal.com> wrote:

>  Hi Ted,
>
>
>
> Thanks for your information.
>
> My application queries the HBase, and for some of the queries it just hang
> there and throw exception after several minutes (5-8minutes). As a
> workaround, I try to set the timeout to a shorter time, so my app won’t
> hang for minutes but for several seconds.  I tried to set both the time out
> to 1000 (1s). but it still hang for several minutes.Is this expected?
>
>
>
> Appreciate it if you know how I could fix the exception.
>
>
>
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Could not seek StoreFileScanner[HFileScanner for
> reader
> reader=hdfs://xxx/hbase/data/data/default/xxx/af7898973c510425fabb7c814ac8ba04/EOUT_T_SRD/125acceb75d84724a089701c590a4d3d,
> compression=snappy, cacheConf=CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false]
> [cacheCompressed=false],
> firstKey=addrv#34005240#US,_28409,_822|addre/F|rval#null|cust#1158923121468951849|addre#1095283883|1/EOUT_T_SRD:~/1430982000000/Put,
> lastKey=addrv#38035AC7#US,_60449,_4684|addre/F|rval#null|cust#1335211720509289817|addre#697997140|1/EOUT_T_SRD:~/1430982000000/Put,
> avgKeyLen=122, avgValueLen=187, entries=105492830, length=6880313695,
> cur=null] to key addrv#34B97AEC#FR,_06110,_41 route des
> breguieres|addre/F|rval#/EOUT_T_SRD:/LATEST_TIMESTAMP/DeleteFamily/vlen=0/mvcc=0
>
>         at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:165)
>
>         at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317)
>
>         at
> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:176)
>
>         at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1847)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3716)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1890)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1876)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1853)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3090)
>
>         at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:28861)
>
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
>
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
>
>         at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>
>         at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>
>         at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>
>         at java.lang.Thread.run(Thread.java:724)
>
> Caused by: java.io.IOException: Failed to read compressed block at
> 1253175503, onDiskSizeWithoutHeader=66428, preReadHeaderSize=33,
> header.length=33, header bytes:
> DATABLKE\x00\x00&3\x00\x00\xC3\xC9\x00\x00\x00\x01r\xC4-\xDF\x01\x00\x00@
> \x00\x00\x00&P
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1451)
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314)
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:355)
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:494)
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:515)
>
>         at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:238)
>
>         at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:153)
>
>         ... 15 more
>
> Caused by: java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
>
>         at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
>
>         at
> org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:165)
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:239)
>
>         at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1448
>
>
>
> Thanks,
>
> Mike
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Monday, May 18, 2015 11:55 PM
> *To:* user@hbase.apache.org
> *Cc:* Fang, Mike; Dai, Kevin
> *Subject:* Re: How to set Timeout for get/scan operations without
> impacting others
>
>
>
> hbase.client.operation.timeout is used by HBaseAdmin operations, by RegionReplicaFlushHandler
> and by various HTable operations (including Get).
>
>
>
> hbase.rpc.timeout is for the RPC layer to define how long HBase client
> applications take for a remote call to time out. It uses pings to check
> connections but will eventually throw a TimeoutException.
>
>
>
> FYI
>
>
>
> On Sun, May 17, 2015 at 11:11 PM, Jianshi Huang <jianshi.huang@gmail.com>
> wrote:
>
> Hi,
>
> I need to set tight timeout for get/scan operations and I think HBase
> Client already support it.
>
> I found three related keys:
>
> - hbase.client.operation.timeout
> - hbase.rpc.timeout
> - hbase.client.retries.number
>
> What's the difference between hbase.client.operation.timeout and
> hbase.rpc.timeout?
> My understanding is that hbase.rpc.timeout has larger scope than hbase.
> client.operation.timeout, so setting hbase.client.operation.timeout  is
> safer. Am I correct?
>
> And any other property keys I can uses?
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message