Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: Pankaj kr <pankaj.kr@huawei.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>,
        "user@hbase.apache.org"
	<user@hbase.apache.org>
Subject: Region comapction failed 
Thread-Topic: Region comapction failed 
Thread-Index: AdJtinVkQOyKJFlaSUGkANGhzcN1Pw==
Date: Fri, 13 Jan 2017 10:47:40 +0000
Message-ID: <74ECFFA8DC3B6847888649793C770FE0A2E46CAE@blreml501-mbb>
Accept-Language: en-US, zh-CN
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_74ECFFA8DC3B6847888649793C770FE0A2E46CAEblreml501mbb_"
MIME-Version: 1.0
archived-at: Fri, 13 Jan 2017 10:51:29 -0000

--_000_74ECFFA8DC3B6847888649793C770FE0A2E46CAEblreml501mbb_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

We met a weird issue in our production environment.

Region compaction is always failing with  following errors,

1.
2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompa=
ctions-1483858654825 | Compaction failed Request =3D regionName=3DXXXX., st=
oreName=3DXYZ, fileCount=3D6, fileSize=3D100.7 M (3.2 M, 20.8 M, 15.1 M, 20=
.9 M, 21.0 M, 19.7 M), priority=3D-5, time=3D1747414906352088 | org.apache.=
hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(=
CompactSplitThread.java:562)
java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a colum=
n actually smaller than the previous column:  XXXXXXX
                at org.apache.hadoop.hbase.regionserver.ScanWildcardColumnT=
racker.checkVersions(ScanWildcardColumnTracker.java:114)
                at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.ma=
tch(ScanQueryMatcher.java:457)
                at org.apache.hadoop.hbase.regionserver.StoreScanner.next(S=
toreScanner.java:551)
                at org.apache.hadoop.hbase.regionserver.compactions.Compact=
or.performCompaction(Compactor.java:328)
                at org.apache.hadoop.hbase.regionserver.compactions.Default=
Compactor.compact(DefaultCompactor.java:104)
                at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$=
DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
                at org.apache.hadoop.hbase.regionserver.HStore.compact(HSto=
re.java:1243)
                at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRe=
gion.java:1895)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread$=
CompactionRunner.doCompaction(CompactSplitThread.java:546)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread$=
CompactionRunner.run(CompactSplitThread.java:583)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(Thread=
PoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecuto

2.
2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompa=
ctions-1483686810953 | Compaction failed Request =3D regionName=3DYYYYYY, s=
toreName=3DABC, fileCount=3D6, fileSize=3D125.3 M (20.9 M, 20.9 M, 20.9 M, =
20.9 M, 20.9 M, 20.9 M), priority=3D-68, time=3D1748294500157323 | org.apac=
he.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompacti=
on(CompactSplitThread.java:562)
java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX afte=
r XXXXXXXXXXXX
                at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.ap=
pendGeneralBloomfilter(StoreFile.java:911)
                at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.ap=
pend(StoreFile.java:947)
                at org.apache.hadoop.hbase.regionserver.compactions.Compact=
or.performCompaction(Compactor.java:337)
                at org.apache.hadoop.hbase.regionserver.compactions.Default=
Compactor.compact(DefaultCompactor.java:104)
                at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$=
DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
                at org.apache.hadoop.hbase.regionserver.HStore.compact(HSto=
re.java:1243)
                at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRe=
gion.java:1895)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread$=
CompactionRunner.doCompaction(CompactSplitThread.java:546)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread$=
CompactionRunner.run(CompactSplitThread.java:583)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(Thread=
PoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa=
dPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)

HBase version : 1.0.2

We have verified all the HFiles in the store using HFilePrettyPrinter with =
"k" (checkrow), all report is normal. Full scan is also successful.
We don't have the access to the actual data and may be customer wont agree =
to  share that .

Have anyone faced this issue, any pointers will be much appreciated.

Thanks & Regards,
Pankaj

--_000_74ECFFA8DC3B6847888649793C770FE0A2E46CAEblreml501mbb_--