phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <>
Subject [jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
Date Thu, 01 Jun 2017 05:01:04 GMT


Lars Hofhansl commented on PHOENIX-3797:

Sorry for the many comments. It turns out the repair scanner is trying to rebuild the local
index during the compaction while scanning along the main column families, and that's why
the index writes can be out of order.

The idea is pretty clever! []

And hence your new idea of writing the index update through the region, so that they will
get sorted.
So that can work. For large regions that would lead to a lot of HFiles, though (for a 10g
region with 256mb flush size it would lead to 40 files after the major compaction).

I think my idea of rewriting the index data from the merged index data itself is pretty tricky,

Here's yet another idea: Can we hook a scanner right above the HFiles? That scanner would
rewrite the keys based on the new region startkey. So now the store scanner for the index
would do the right thing (merge sort between the values from the HFile scanners).

> Local Index - Compaction fails on table with local index due to non-increasing bloom
> -----------------------------------------------------------------------------------------
>                 Key: PHOENIX-3797
>                 URL:
>             Project: Phoenix
>          Issue Type: Bug
>         Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied. HBase 0.98.23-hadoop2
>            Reporter: Mujtaba Chohan
>            Assignee: Ankit Singhal
>            Priority: Blocker
>             Fix For: 4.11.0
>         Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR [RS:0;host:59455-smallCompactions-1492644947594] regionserver.CompactSplitThread:
Compaction failed Request = regionName=FHA,00Dxx0000001gES005001xx000003DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M), priority=7, time=7442973347247614
> Non-increasing Bloom keys: 00Dxx0000001gES005001xx000003DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx0000001gER001001xx000003DGPb01739544DCtf
   after 00Dxx0000001gES005001xx000003DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx0000001gER001001xx000003DGPa017115434KTM
> 	at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(
> 	at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(
> 	at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(
> 	at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(
> 	at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(
> 	at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(
> 	at org.apache.hadoop.hbase.regionserver.HStore.compact(
> 	at org.apache.hadoop.hbase.regionserver.HRegion.compact(
> 	at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(
> 	at org.apache.hadoop.hbase.regionserver.CompactSplitThread$
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(
> 	at java.util.concurrent.ThreadPoolExecutor$
> 	at
> {noformat}

This message was sent by Atlassian JIRA

View raw message