hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5885) Invalid HFile block magic on Local file System
Date Sat, 28 Apr 2012 01:06:48 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264178#comment-13264178
] 

Todd Lipcon commented on HBASE-5885:
------------------------------------

{quote}
I would see this as the final solution. Anything else would either be much bigger and more
invasive or would rely on hadoop changes which is unlikely.
Inside local FS they seek to the next boundary of checksums. However when not reading checksums
it gives the wrong data when a read is issued.
{quote}

Please file an upstream bug in the hadoop project for this as well - we should absolutely
fix this. I agree a workaround in the near term makes sense, but if LocalFileSystem is broken,
we should fix it!
                
> Invalid HFile block magic on Local file System
> ----------------------------------------------
>
>                 Key: HBASE-5885
>                 URL: https://issues.apache.org/jira/browse/HBASE-5885
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.0, 0.96.0
>            Reporter: Elliott Clark
>            Assignee: Elliott Clark
>            Priority: Blocker
>             Fix For: 0.94.0
>
>         Attachments: HBASE-5885-94-0.patch, HBASE-5885-94-1.patch, HBASE-5885-trunk-0.patch,
HBASE-5885-trunk-1.patch
>
>
> ERROR: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=7, exceptions:
> Thu Apr 26 11:19:18 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a,
java.io.IOException: java.io.IOException: Could not iterate StoreFileScanner[HFileScanner
for reader reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268,
compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
[cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false],
firstKey=0000000001/info:data/1335463981520/Put, lastKey=0002588100/info:data/1335463902296/Put,
avgKeyLen=30, avgValueLen=1000, entries=1215085, length=1264354417, cur=0000000248/info:data/1335463994457/Put/vlen=1000/ts=0]
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
> 	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> 	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
> Caused by: java.io.IOException: Invalid HFile block magic: \xEC\xD5\x9D\xB4\xC2bfo
> 	at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
> 	at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:254)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
> 	... 12 more
> Thu Apr 26 11:19:19 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a,
java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
> 	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> 	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
> Caused by: java.lang.IllegalArgumentException
> 	at java.nio.Buffer.position(Buffer.java:216)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:406)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
> 	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
> 	at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
> 	... 5 more
> On latest 0.94 branch I spun up a new standalone hbase.  Then I started a performance
evaluation run hbase/bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred randomWrite
10
> after that completed I tried a scan of TestTable.  The scan got the above error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message