hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Meta vs Stores / first and last keys and corrupted HFiles.
Date Tue, 27 Aug 2013 01:26:46 GMT
I have uploaded a patch in HBASE-9346 to add some check into HBCK. Working
pretty well.

To solve the issue, I "simply" merged the culprit region with the next one,
so there is no more issues with the region boundaries. Now I will major
compact and check the result. Then I will take a look at the NPE below...

2013/8/26 Jean-Marc Spaggiari <jean-marc@spaggiari.org>

> I have created HBASE-9346<https://issues.apache.org/jira/browse/HBASE-9346>for
the boundaries issues.
>
>
> 2013/8/26 Jean-Marc Spaggiari <jean-marc@spaggiari.org>
>
>> You're right. I figured that after. So I removed the oldlogs (might be a
>> left over, was there for 2 days even after multiple stop/restart) but I'm
>> still getting this error on the region:
>>
>>
>> Exception in thread "main" java.lang.NullPointerException
>>     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>>     at
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>>     at
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>>     at
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>>
>> Even if now I'm able to get the keys information:
>>     firstKey=\x16\xCE\x8E\xCF\x00\x00\x00\x00...
>>     lastKey=\x19\xA0Ma\x00\x00\x00\x00...
>>
>> This is another region that the one where I have overlaps....
>>
>> JM
>>
>>
>> 2013/8/26 Sergey Shelukhin <sergey@hortonworks.com>
>>
>>> Um, I may be missing something here, but is it really an HFile?
>>> Looks like an hlog to me.
>>>
>>> org.apache.hadoop.hbase.io.
>>> hfile.CorruptHFileException: Problem reading
>>> HFile Trailer from file
>>>
>>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>>>
>>>
>>> On Mon, Aug 26, 2013 at 12:31 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>> > Hi,
>>> >
>>> > Don't ask me how, but I have one table in a pretty strange state.
>>> >
>>> > First, seems that I have one corrupted HFile (at least).
>>> >
>>> > FirstKey return null. (Same for StopKey since header is corrupted).
>>> >
>>> > Exception in thread "main" java.lang.NullPointerException
>>> >     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>>> >     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>>> >
>>> > Because the header is not correct.
>>> >
>>> > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
>>> > HFile Trailer from file
>>> >
>>> >
>>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>>> >     at
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595)
>>> >     at
>>> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>>> >     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>>> > Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
>>> > 16275367 (expected to be between 1 and 2)
>>> >     at
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:771)
>>> >     at
>>> >
>>> >
>>> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:323)
>>> >     at
>>> >
>>> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:549)
>>> >     ... 5 more
>>> >
>>> > HBCK don't detect that:
>>> > 0 inconsistencies detected.
>>> > Status: OK
>>> >
>>> > Also, I have an issue with the first key / last key reported by the
>>> stores
>>> > overlapping when the one reported by the META are not.
>>> >
>>> > Store:
>>> >     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00...
>>> >     lastKey=\xFF\xFF\xFF\xFE\x00\x00\x00\x00...
>>> > Meta:
>>> >     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00
>>> >     lastKey=\xF5\x9B@}\x00\x00\x00\x00...
>>> >
>>> >
>>> > So, few things.
>>> >
>>> > 1) We should add something into HBCK to check the HFile format again
>>> > corruption.
>>> > 2) We should add something into HBCK to validate META regions
>>> boundaries
>>> > against Store Files
>>> > 3) How can I repaire my HFile? ;)
>>> >
>>> > I'm already working on #2 and will have something ready soon. Then I
>>> will
>>> > most probably more to #1. But I only have detection dont for now. I'm
>>> not
>>> > sure exactly what are the correct steps to repair...
>>> >
>>> > JM
>>> >
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message