hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: IndexOutOfBoundsException during retrieving region split point
Date Fri, 03 Jun 2016 16:16:16 GMT
1.0.0 is quite old.

Is it possible to upgrade to 1.1 or 1.2 release ?

Thanks

On Fri, Jun 3, 2016 at 8:12 AM, Pankaj kr <pankaj.kr@huawei.com> wrote:

> Hi,
>
> We met a weird scenario in our production environment.
> IndexOutOfBoundsException is thrown while retrieving mid key of the
> storefile after region compaction.
>
> Log Snippet :
> -----------------
> 2016-05-30 01:41:58,484 | INFO  |
> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 |
> Completed compaction of 1 (all) file(s) in CF of
> User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.
> into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for store is
> 22.0 G. This selection was in queue for 0sec, and took 6mins, 25sec to
> execute. |
> org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HStore.java:1356)
> 2016-05-30 01:41:58,485 | INFO  |
> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 |
> Completed compaction: Request =
> regionName=User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.,
> storeName=CF, fileCount=1, fileSize=44.0 G, priority=6,
> time=295643974900644; duration=6mins, 25sec |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:544)
> 2016-05-30 01:41:58,529 | ERROR |
> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 |
> Compaction failed Request =
> regionName=User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.,
> storeName=CF, fileCount=1, fileSize=44.0 G, priority=6,
> time=295643974900644 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:563)
> java.lang.IndexOutOfBoundsException
>                 at java.nio.Buffer.checkIndex(Buffer.java:540)
>                 at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
>                 at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
>                 at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
>                 at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512)
>                 at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480)
>                 at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685)
>                 at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
>                 at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986)
>                 at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914)
>                 at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240)
>                 at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552)
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>                 at java.lang.Thread.run(Thread.java:745)
>
> -----------------
> Observation:
> >> HFilePrettyPrinter also print the message "Unable to retrieve the
> midkey" for the mid key.
> >> HDFS fsck report the hfile healthy.
>
> Though successful region compaction were also there, only few region
> compaction failed with same error.
>
> Have anyone faced this issue? Any help will be appreciated.
> HBase version is 1.0.0.
>
> Regards,
> Pankaj
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message