hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From test gmail test <meeran.gladiat...@gmail.com>
Subject Re: Could not iterate StoreFileScanner - during compaction
Date Mon, 06 Jul 2020 08:54:16 GMT
Hi Sean,



printblocks output - https://pastebin.com/EYUpi6LL

blockheaders output - https://pastebin.com/TJBqgwsp



We are yet to test it on HBase-2.2. Will upgrade the cluster and let you know. Thanks for
the help.
 

Regards,

Meeran




---- On Sat, 04 Jul 2020 05:26:46 +0530 Sean Busbey <busbey@apache.org> wrote ----



File attachments won't work on the mailing list. Can you put the files on 
some hosting service? 
 
Can you reproduce the problem on hbase 2.2? HBase 2.1 has been EOM since 
May. 
 
 
On Fri, Jul 3, 2020, 18:20 Mohamed Meeran <mailto:meeran.gladiatorz@gmail.com> 
wrote: 
 
> Hi, 
> 
> We are using HBase-2.1.9 (Hadoop-3.1.3) in our setup. In the logs, we see 
> major compaction failed for some of the regions with the following error 
> logs. 
> 
> Caused by: java.io.IOException: Could not iterate 
> StoreFileScanner[HFileScanner for reader 
> reader=hdfs://TestCluster/hbasedata/data/Test/Test/6472f3839fc9b0a1d4b64e182043bc52/hb/2ec37243628b4a03ae3d937da4c27081,

> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=332, 
> currentSize=485.88 MB, freeSize=333.32 MB, maxSize=819.20 MB, 
> heapSize=485.88 MB, minSize=778.24 MB, minFactor=0.95, multiSize=389.12 MB, 
> multiFactor=0.5, singleSize=194.56 MB, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, 
> cacheDataCompressed=false, prefetchOnOpen=false, 
> firstKey=Optional[10259783_1010157000000008129/hb:B/1490097103780/Put/seqid=0], 
> lastKey=Optional[10260211_1009658000000470017/hb:H/1490097295354/Put/seqid=0], 
> avgKeyLen=43, avgValueLen=213357, entries=10134, length=2163318554, 
> cur=10259783_1010157000000008851/hb:B/1490097148981/Put/vlen=16591695/seqid=0] 
> at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:217)

> at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120) 
> at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654) 
> at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153) 
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6593)

> at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6757)

> at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6527)

> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3158) 
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3407) 
> ... 5 more 
> Caused by: java.io.IOException: Invalid onDisksize=-969694035: expected to 
> be at least 33 and at most 2147483647, or -1 
> at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.checkAndGetSizeAsInt(HFileBlock.java:1673)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1746)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1610)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1496)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readNextDataBlock(HFileReaderImpl.java:931)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.isNextBlock(HFileReaderImpl.java:1064)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.positionForNextBlock(HFileReaderImpl.java:1058)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1076)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)

> at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)

> ... 13 more 
> 
> We analysed a file using the hfile tool. Attaching the output for 
> printblocks and printblockheaders. 
> 
> Any help to fix this would be greatly appreciated. 
> -- 
> Thanks in advance, 
>      Meeran 
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message