hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meeran <meeran.gladiat...@gmail.com>
Subject Re: Could not iterate StoreFileScanner - during compaction
Date Wed, 08 Jul 2020 07:44:36 GMT
Hi Sean,



We upgraded the cluster to latest stable version HBase-2.2.4. We are still facing the issue.
Any help on this please?



Thanks,

Meeran



---- On Mon, 06 Jul 2020 14:24:16 +0530 test gmail test <meeran.gladiatorz@gmail.com>
wrote ----


Hi Sean,



printblocks output - https://pastebin.com/EYUpi6LL

blockheaders output - https://pastebin.com/TJBqgwsp



We are yet to test it on HBase-2.2. Will upgrade the cluster and let you know. Thanks for
the help.

 

Regards,

Meeran





---- On Sat, 04 Jul 2020 05:26:46 +0530 Sean Busbey <mailto:busbey@apache.org> wrote
----











File attachments won't work on the mailing list. Can you put the files on 
some hosting service? 
 
Can you reproduce the problem on hbase 2.2? HBase 2.1 has been EOM since 
May. 
 
 
On Fri, Jul 3, 2020, 18:20 Mohamed Meeran <mailto:meeran.gladiatorz@gmail.com> 
wrote: 
 
> Hi, 
> 
> We are using HBase-2.1.9 (Hadoop-3.1.3) in our setup. In the logs, we see 
> major compaction failed for some of the regions with the following error 
> logs. 
> 
> Caused by: java.io.IOException: Could not iterate 
> StoreFileScanner[HFileScanner for reader 
> reader=hdfs://TestCluster/hbasedata/data/Test/Test/6472f3839fc9b0a1d4b64e182043bc52/hb/2ec37243628b4a03ae3d937da4c27081,

> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=332, 
> currentSize=485.88 MB, freeSize=333.32 MB, maxSize=819.20 MB, 
> heapSize=485.88 MB, minSize=778.24 MB, minFactor=0.95, multiSize=389.12 MB, 
> multiFactor=0.5, singleSize=194.56 MB, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, 
> cacheDataCompressed=false, prefetchOnOpen=false, 
> firstKey=Optional[10259783_1010157000000008129/hb:B/1490097103780/Put/seqid=0], 
> lastKey=Optional[10260211_1009658000000470017/hb:H/1490097295354/Put/seqid=0], 
> avgKeyLen=43, avgValueLen=213357, entries=10134, length=2163318554, 
> cur=10259783_1010157000000008851/hb:B/1490097148981/Put/vlen=16591695/seqid=0] 
> at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:217)

> at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120) 
> at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654) 
> at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153) 
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6593)

> at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6757)

> at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6527)

> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3158) 
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3407) 
> ... 5 more 
> Caused by: java.io.IOException: Invalid onDisksize=-969694035: expected to 
> be at least 33 and at most 2147483647, or -1 
> at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.checkAndGetSizeAsInt(HFileBlock.java:1673)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1746)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1610)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1496)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readNextDataBlock(HFileReaderImpl.java:931)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.isNextBlock(HFileReaderImpl.java:1064)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.positionForNextBlock(HFileReaderImpl.java:1058)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1076)

> at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)

> at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)

> ... 13 more 
> 
> We analysed a file using the hfile tool. Attaching the output for 
> printblocks and printblockheaders. 
> 
> Any help to fix this would be greatly appreciated. 
> -- 
> Thanks in advance, 
>      Meeran 
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message