hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Meta vs Stores / first and last keys and corrupted HFiles.
Date Mon, 26 Aug 2013 23:44:19 GMT
I have created HBASE-9346
<https://issues.apache.org/jira/browse/HBASE-9346>for the boundaries
issues.

2013/8/26 Jean-Marc Spaggiari <jean-marc@spaggiari.org>

> You're right. I figured that after. So I removed the oldlogs (might be a
> left over, was there for 2 days even after multiple stop/restart) but I'm
> still getting this error on the region:
>
>
> Exception in thread "main" java.lang.NullPointerException
>     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>     at
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>     at
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>     at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>     at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>     at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>
> Even if now I'm able to get the keys information:
>     firstKey=\x16\xCE\x8E\xCF\x00\x00\x00\x00...
>     lastKey=\x19\xA0Ma\x00\x00\x00\x00...
>
> This is another region that the one where I have overlaps....
>
> JM
>
>
> 2013/8/26 Sergey Shelukhin <sergey@hortonworks.com>
>
>> Um, I may be missing something here, but is it really an HFile?
>> Looks like an hlog to me.
>>
>> org.apache.hadoop.hbase.io.
>> hfile.CorruptHFileException: Problem reading
>> HFile Trailer from file
>>
>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>>
>>
>> On Mon, Aug 26, 2013 at 12:31 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>> > Hi,
>> >
>> > Don't ask me how, but I have one table in a pretty strange state.
>> >
>> > First, seems that I have one corrupted HFile (at least).
>> >
>> > FirstKey return null. (Same for StopKey since header is corrupted).
>> >
>> > Exception in thread "main" java.lang.NullPointerException
>> >     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>> >     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>> >
>> > Because the header is not correct.
>> >
>> > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
>> > HFile Trailer from file
>> >
>> >
>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>> >     at
>> > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595)
>> >     at
>> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>> >     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>> > Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
>> > 16275367 (expected to be between 1 and 2)
>> >     at
>> >
>> org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:771)
>> >     at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:323)
>> >     at
>> > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:549)
>> >     ... 5 more
>> >
>> > HBCK don't detect that:
>> > 0 inconsistencies detected.
>> > Status: OK
>> >
>> > Also, I have an issue with the first key / last key reported by the
>> stores
>> > overlapping when the one reported by the META are not.
>> >
>> > Store:
>> >     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00...
>> >     lastKey=\xFF\xFF\xFF\xFE\x00\x00\x00\x00...
>> > Meta:
>> >     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00
>> >     lastKey=\xF5\x9B@}\x00\x00\x00\x00...
>> >
>> >
>> > So, few things.
>> >
>> > 1) We should add something into HBCK to check the HFile format again
>> > corruption.
>> > 2) We should add something into HBCK to validate META regions boundaries
>> > against Store Files
>> > 3) How can I repaire my HFile? ;)
>> >
>> > I'm already working on #2 and will have something ready soon. Then I
>> will
>> > most probably more to #1. But I only have detection dont for now. I'm
>> not
>> > sure exactly what are the correct steps to repair...
>> >
>> > JM
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message