hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj kr <pankaj...@huawei.com>
Subject RE: IndexOutOfBoundsException during retrieving region split point
Date Mon, 06 Jun 2016 07:42:34 GMT
Thanks Ted for replying. 
Yeah, We have a plan to upgrade. But currently I want to know the reason behind this. I tried
to reproduce this in our test environment but didn’t happen.

in HFilePrettyPrinter output "numDataIndexLevels=2", so there were multilevel data index.
Is which circumstances this problem can happen?

Regards,
Pankaj

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: Saturday, June 04, 2016 12:16 AM
To: user@hbase.apache.org
Cc: bhupendra jain; Sharanabasappa G Keriwaddi
Subject: Re: IndexOutOfBoundsException during retrieving region split point

1.0.0 is quite old.

Is it possible to upgrade to 1.1 or 1.2 release ?

Thanks

On Fri, Jun 3, 2016 at 8:12 AM, Pankaj kr <pankaj.kr@huawei.com> wrote:

> Hi,
>
> We met a weird scenario in our production environment.
> IndexOutOfBoundsException is thrown while retrieving mid key of the 
> storefile after region compaction.
>
> Log Snippet :
> -----------------
> 2016-05-30 01:41:58,484 | INFO  |
> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | 
> Completed compaction of 1 (all) file(s) in CF of 
> User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.
> into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for 
> store is
> 22.0 G. This selection was in queue for 0sec, and took 6mins, 25sec to 
> execute. |
> org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HS
> tore.java:1356)
> 2016-05-30 01:41:58,485 | INFO  |
> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | 
> Completed compaction: Request = 
> regionName=User_Namespace:User_Table,100050007010803_20140126_30801071
> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3
> c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, 
> time=295643974900644; duration=6mins, 25sec |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn
> er.run(CompactSplitThread.java:544)
> 2016-05-30 01:41:58,529 | ERROR |
> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | 
> Compaction failed Request = 
> regionName=User_Namespace:User_Table,100050007010803_20140126_30801071
> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3
> c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6,
> time=295643974900644 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn
> er.run(CompactSplitThread.java:563)
> java.lang.IndexOutOfBoundsException
>                 at java.nio.Buffer.checkIndex(Buffer.java:540)
>                 at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
>                 at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
>                 at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
>                 at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512)
>                 at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480)
>                 at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685)
>                 at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
>                 at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986)
>                 at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914)
>                 at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240)
>                 at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552)
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>                 at java.lang.Thread.run(Thread.java:745)
>
> -----------------
> Observation:
> >> HFilePrettyPrinter also print the message "Unable to retrieve the
> midkey" for the mid key.
> >> HDFS fsck report the hfile healthy.
>
> Though successful region compaction were also there, only few region 
> compaction failed with same error.
>
> Have anyone faced this issue? Any help will be appreciated.
> HBase version is 1.0.0.
>
> Regards,
> Pankaj
>
Mime
View raw message