Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of sergey@hortonworks.com
 designates 74.125.82.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPQV63WRYm8bKhqz2e57_tqpfnMH4onwKW4ajLpcbVNjruvk7g@mail.gmail.com>
References: 
 <CAPQV63WRYm8bKhqz2e57_tqpfnMH4onwKW4ajLpcbVNjruvk7g@mail.gmail.com>
Date: Mon, 26 Aug 2013 16:04:23 -0700
Message-ID: 
 <CAHXxaiDtaf3Yp18jw1RiWYKk8j4nHVyQ7w8X_UjvvCa3CZSnhg@mail.gmail.com>
Subject: Re: Meta vs Stores / first and last keys and corrupted HFiles.
From: Sergey Shelukhin <sergey@hortonworks.com>
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=089e0102fb5cf0e65304e4e1c6fa

--089e0102fb5cf0e65304e4e1c6fa
Content-Type: text/plain; charset=US-ASCII

Um, I may be missing something here, but is it really an HFile?
Looks like an hlog to me.

org.apache.hadoop.hbase.io.
hfile.CorruptHFileException: Problem reading
HFile Trailer from file
hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526


On Mon, Aug 26, 2013 at 12:31 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> Hi,
>
> Don't ask me how, but I have one table in a pretty strange state.
>
> First, seems that I have one corrupted HFile (at least).
>
> FirstKey return null. (Same for StopKey since header is corrupted).
>
> Exception in thread "main" java.lang.NullPointerException
>     at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
>     at
>
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
>     at
>
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
>
> Because the header is not correct.
>
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> HFile Trailer from file
>
> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595)
>     at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
>     at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
> 16275367 (expected to be between 1 and 2)
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:771)
>     at
>
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:323)
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:549)
>     ... 5 more
>
> HBCK don't detect that:
> 0 inconsistencies detected.
> Status: OK
>
> Also, I have an issue with the first key / last key reported by the stores
> overlapping when the one reported by the META are not.
>
> Store:
>     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00...
>     lastKey=\xFF\xFF\xFF\xFE\x00\x00\x00\x00...
> Meta:
>     firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00
>     lastKey=\xF5\x9B@}\x00\x00\x00\x00...
>
>
> So, few things.
>
> 1) We should add something into HBCK to check the HFile format again
> corruption.
> 2) We should add something into HBCK to validate META regions boundaries
> against Store Files
> 3) How can I repaire my HFile? ;)
>
> I'm already working on #2 and will have something ready soon. Then I will
> most probably more to #1. But I only have detection dont for now. I'm not
> sure exactly what are the correct steps to repair...
>
> JM
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

--089e0102fb5cf0e65304e4e1c6fa--