hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Region comapction failed
Date Mon, 16 Jan 2017 11:36:25 GMT
This is likely a new bug. 

Please consider opening a JIRA. 

> On Jan 16, 2017, at 2:22 AM, Pankaj kr <pankaj.kr@huawei.com> wrote:
> 
> Yeah, HBASE-13123 already available.
> 
> While going through HFilePrettyPrinter result with -e option, we observed some keys doesn't
contain complete column_family:qualifier information. Normally in HFIle Keys are formed as,
>    K: r1/CF:Qualifier/1484302946516/Put/vlen=1/seqid=0
> 
> But some records,
>    K: r1/Unknown_Qualifier/1484302946516/Put/vlen=1/seqid=0
> Here there is no column family detail, normally it is like column family name and the
colon (:)    .
> 
> It's strange, how key is written in wrong format in HFile.
> 
> 
> Regards,
> Pankaj
> 
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com] 
> Sent: Saturday, January 14, 2017 9:41 AM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Region comapction failed
> 
> w.r.t. #2, I did a quick search for bloom related fixes.
> 
> I found HBASE-13123 but it was in 1.0.2
> 
> Planning to spend more time in the next few days.
> 
>> On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr <pankaj.kr@huawei.com> wrote:
>> 
>> Thanks Ted for replying.
>> 
>> Actually issue happened in production environment and there are many 
>> HFiles in that store (can't get the file). As we don't log the file 
>> name which is corrupted, Is there anyway to get the corrupted  file name?
>> 
>> Block encoding is "NONE", table schema has bloom filter as "ROW", 
>> compression type is "Snappy" and durability is SKIP_WAL.
>> 
>> 
>> Regards,
>> Pankaj
>> 
>> 
>> -----Original Message-----
>> From: Ted Yu [mailto:yuzhihong@gmail.com]
>> Sent: Friday, January 13, 2017 10:30 PM
>> To: dev@hbase.apache.org
>> Cc: user@hbase.apache.org
>> Subject: Re: Region comapction failed
>> 
>> In the second case, the error happened when writing hfile. Can you 
>> track down the path of the new file so that further investigation can be done ?
>> 
>> Does the table use any encoding ?
>> 
>> Thanks
>> 
>>> On Jan 13, 2017, at 2:47 AM, Pankaj kr <pankaj.kr@huawei.com> wrote:
>>> 
>>> Hi,
>>> 
>>> We met a weird issue in our production environment.
>>> 
>>> Region compaction is always failing with  following errors,
>>> 
>>> 1.
>>> 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:
>> PORT-longCompactions-1483858654825 | Compaction failed Request = 
>> regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, 
>> 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, 
>> time=1747414906352088 | 
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$
>> CompactionRunner.doCompaction(CompactSplitThread.java:562)
>>> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into 
>>> a
>> column actually smaller than the previous column:  XXXXXXX
>>>               at org.apache.hadoop.hbase.regionserver.
>> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.
>> java:114)
>>>               at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
>> match(ScanQueryMatcher.java:457)
>>>               at org.apache.hadoop.hbase.regionserver.StoreScanner.
>> next(StoreScanner.java:551)
>>>               at org.apache.hadoop.hbase.regionserver.compactions.
>> Compactor.performCompaction(Compactor.java:328)
>>>               at org.apache.hadoop.hbase.regionserver.compactions.
>> DefaultCompactor.compact(DefaultCompactor.java:104)
>>>               at org.apache.hadoop.hbase.regionserver.
>> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
>> java:133)
>>>               at 
>>> org.apache.hadoop.hbase.regionserver.HStore.compact(
>> HStore.java:1243)
>>>               at 
>>> org.apache.hadoop.hbase.regionserver.HRegion.compact(
>> HRegion.java:1895)
>>>               at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:546)
>>>               at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
>>>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>>               at java.util.concurrent.ThreadPoolExecuto
>>> 
>>> 2.
>>> 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:
>> PORT-longCompactions-1483686810953 | Compaction failed Request = 
>> regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 
>> M,
>> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
>> time=1748294500157323 | org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:562)
>>> java.io.IOException: Non-increasing Bloom keys: 
>>> XXXXXXXXXXXXXXXXXXXXXX
>> after XXXXXXXXXXXX
>>>               at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
>> appendGeneralBloomfilter(StoreFile.java:911)
>>>               at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
>> append(StoreFile.java:947)
>>>               at org.apache.hadoop.hbase.regionserver.compactions.
>> Compactor.performCompaction(Compactor.java:337)
>>>               at org.apache.hadoop.hbase.regionserver.compactions.
>> DefaultCompactor.compact(DefaultCompactor.java:104)
>>>               at org.apache.hadoop.hbase.regionserver.
>> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
>> java:133)
>>>               at 
>>> org.apache.hadoop.hbase.regionserver.HStore.compact(
>> HStore.java:1243)
>>>               at 
>>> org.apache.hadoop.hbase.regionserver.HRegion.compact(
>> HRegion.java:1895)
>>>               at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:546)
>>>               at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
>>>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>>               at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>>               at java.lang.Thread.run(Thread.java:745)
>>> 
>>> HBase version : 1.0.2
>>> 
>>> We have verified all the HFiles in the store using 
>>> HFilePrettyPrinter
>> with "k" (checkrow), all report is normal. Full scan is also successful.
>>> We don't have the access to the actual data and may be customer wont
>> agree to  share that .
>>> 
>>> Have anyone faced this issue, any pointers will be much appreciated.
>>> 
>>> Thanks & Regards,
>>> Pankaj
>> 

Mime
View raw message