hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj kr <pankaj...@huawei.com>
Subject RE: IndexOutOfBoundsException during retrieving region split point
Date Mon, 06 Jun 2016 08:49:35 GMT
Thanks Anoop for replying...
Yes, in our test environment also numDataIndexLevels=2.

In production environment there were successful region split after compaction, only few region
split failed with same error.

Regards,
Pankaj

-----Original Message-----
From: Anoop John [mailto:anoop.hbase@gmail.com] 
Sent: Monday, June 06, 2016 3:49 PM
To: user@hbase.apache.org
Subject: Re: IndexOutOfBoundsException during retrieving region split point

In ur test env also u have numDataIndexLevels=2?  Or it is 1 only?

-Anoop-

On Mon, Jun 6, 2016 at 1:12 PM, Pankaj kr <pankaj.kr@huawei.com> wrote:
> Thanks Ted for replying.
> Yeah, We have a plan to upgrade. But currently I want to know the reason behind this.
I tried to reproduce this in our test environment but didn’t happen.
>
> in HFilePrettyPrinter output "numDataIndexLevels=2", so there were multilevel data index.
Is which circumstances this problem can happen?
>
> Regards,
> Pankaj
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Saturday, June 04, 2016 12:16 AM
> To: user@hbase.apache.org
> Cc: bhupendra jain; Sharanabasappa G Keriwaddi
> Subject: Re: IndexOutOfBoundsException during retrieving region split 
> point
>
> 1.0.0 is quite old.
>
> Is it possible to upgrade to 1.1 or 1.2 release ?
>
> Thanks
>
> On Fri, Jun 3, 2016 at 8:12 AM, Pankaj kr <pankaj.kr@huawei.com> wrote:
>
>> Hi,
>>
>> We met a weird scenario in our production environment.
>> IndexOutOfBoundsException is thrown while retrieving mid key of the 
>> storefile after region compaction.
>>
>> Log Snippet :
>> -----------------
>> 2016-05-30 01:41:58,484 | INFO  |
>> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | 
>> Completed compaction of 1 (all) file(s) in CF of 
>> User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.
>> into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for 
>> store is
>> 22.0 G. This selection was in queue for 0sec, and took 6mins, 25sec 
>> to execute. | 
>> org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(H
>> S
>> tore.java:1356)
>> 2016-05-30 01:41:58,485 | INFO  |
>> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | 
>> Completed compaction: Request =
>> regionName=User_Namespace:User_Table,100050007010803_20140126_3080107
>> 1
>> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d
>> 3 c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, 
>> time=295643974900644; duration=6mins, 25sec | 
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRun
>> n
>> er.run(CompactSplitThread.java:544)
>> 2016-05-30 01:41:58,529 | ERROR |
>> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | 
>> Compaction failed Request =
>> regionName=User_Namespace:User_Table,100050007010803_20140126_3080107
>> 1
>> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d
>> 3 c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6,
>> time=295643974900644 |
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRun
>> n
>> er.run(CompactSplitThread.java:563)
>> java.lang.IndexOutOfBoundsException
>>                 at java.nio.Buffer.checkIndex(Buffer.java:540)
>>                 at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
>>                 at
>> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
>>                 at
>> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
>>                 at
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512)
>>                 at
>> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480)
>>                 at
>> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685)
>>                 at
>> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
>>                 at
>> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986)
>>                 at
>> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
>>                 at
>> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914)
>>                 at
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240)
>>                 at
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552)
>>                 at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>                 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>                 at java.lang.Thread.run(Thread.java:745)
>>
>> -----------------
>> Observation:
>> >> HFilePrettyPrinter also print the message "Unable to retrieve the
>> midkey" for the mid key.
>> >> HDFS fsck report the hfile healthy.
>>
>> Though successful region compaction were also there, only few region 
>> compaction failed with same error.
>>
>> Have anyone faced this issue? Any help will be appreciated.
>> HBase version is 1.0.0.
>>
>> Regards,
>> Pankaj
>>
Mime
View raw message