hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuliang Jin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11625) Reading datablock throws "Invalid HFile block magic" and can not switch to hdfs checksum
Date Mon, 25 Aug 2014 06:12:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108789#comment-14108789
] 

Yuliang Jin commented on HBASE-11625:
-------------------------------------

We have encountered this issue this morning in a job which utilized frequently 'checkAndPut()'
method to put data to HBase (Version 0.94.6-cdh4.3.0) directly, and the stack trace says:

{noformat}
Mon Aug 25 05:04:06 CST 2014, org.apache.hadoop.hbase.client.HTable$3@30518bfc, java.io.IOException:
java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://dn:8020/hbase/.../.../.../.../,
compression=snappy, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
[cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false],
firstKey=.../.../1394762400000/Put, lastKey=.../.../1381640043000/Put, avgKeyLen=83, avgValueLen=14,
entries=323120857, length=6150040087, cur=.../.../1408874400000/Maximum/vlen=0/ts=0] to key
.../.../LATEST_TIMESTAMP/Maximum/vlen=0/ts=0
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:172)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:349)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:355)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:312)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:277)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:543)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:411)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:143)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3867)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3939)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3810)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3791)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3834)
	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4760)
	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4733)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2072)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428)
Caused by: java.io.IOException: Failed to read compressed block at 5908855614, onDiskSizeWithoutHeader=2995,
preReadHeaderSize=0, header.length=3028, header bytes: \x00\x9EY\x03ld017766ac516715d3925db24b473
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1871)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1703)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:338)
	at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:480)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:530)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:236)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:161)
	... 20 more
Caused by: java.io.IOException: Invalid HFile block magic: \x00\x9EY\x03ld01
	at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
	at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:256)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1867)
	... 27 more
{noformat}

A major compaction on the problematic region fixed the problem.

> Reading datablock throws "Invalid HFile block magic" and can not switch to hdfs checksum

> -----------------------------------------------------------------------------------------
>
>                 Key: HBASE-11625
>                 URL: https://issues.apache.org/jira/browse/HBASE-11625
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile
>    Affects Versions: 0.94.21, 0.98.4, 0.98.5
>            Reporter: qian wang
>         Attachments: 2711de1fdf73419d9f8afc6a8b86ce64.gz
>
>
> when using hbase checksum,call readBlockDataInternal() in hfileblock.java, it could happen
file corruption but it only can switch to hdfs checksum inputstream till validateBlockChecksum().
If the datablock's header corrupted when b = new HFileBlock(),it throws the exception "Invalid
HFile block magic" and the rpc call fail



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message