Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of theo.bertozzi@gmail.com
 designates 209.85.213.45 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALte62wZRxidZHr8aDz3mZ7jPC7eEAKRRHXkWRcN1GMqT9Qv5g@mail.gmail.com>
References: <53287DF001E4042800392A3F_0_66113@p057>
 <CALte62wZRxidZHr8aDz3mZ7jPC7eEAKRRHXkWRcN1GMqT9Qv5g@mail.gmail.com>
From: Matteo Bertozzi <theo.bertozzi@gmail.com>
Date: Tue, 18 Mar 2014 17:33:41 +0000
Message-ID: 
 <CAFb6HiL9XuT972tYOXULZ-XfAc0sacp8t3UCLnP2vB1W3eDHnA@mail.gmail.com>
Subject: Re: Corrupt HBase table
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf3011d90b14143d04f4e4f1ea

--20cf3011d90b14143d04f4e4f1ea
Content-Type: text/plain; charset=ISO-8859-1

can you try to read the file by using:
org.apache.hadoop.hbase.io.hfile.HFile -f FILE_PATH -s -e

to verify if the problem is reading the block from cache or the data on
disk is corrupted in some way?
The stacktrace is showing a file size > 255GB and a Key of 16M which I
doubt being correct.


Matteo


On Tue, Mar 18, 2014 at 5:26 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> The stack trace doesn't reveal too much other than the error occurred at
> the beginning of a scan.
>
> Other experts would have more insights than I do.
>
> Cheers
>
>
> On Tue, Mar 18, 2014 at 10:10 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN)
> <
> skadambi@bloomberg.net> wrote:
>
> > Thanks Ted. No, we don't use data block encoding. This is 0.96. It
> appears
> > multiple HFiles are affected. Does the stack trace give some context on
> > what is that it's attempting to do at the lowest level?
> >
> >
> > ----- Original Message -----
> > From: yuzhihong@gmail.com
> > To: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) <skadambi@bloomberg.net>,
> > user@hbase.apache.org
> > At: Mar 18 2014 13:03:20
> >
> > Are you using Data Block Encoding for this table ?
> >
> > Can you sideline the HFile and see if other HFiles were affected ?
> >
> > I assume you use 0.94.x
> >
> > Cheers
> >
> >
> > On Tue, Mar 18, 2014 at 7:14 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN)
> > <skadambi@bloomberg.net> wrote:
> >
> >> Hi Folks:
> >>
> >> We've a table that we're unable to make online. A scan on this table
> >> yields the following stack trace. hbck doesn't report any
> inconsistencies.
> >> I went ahead and did a repair, which didn't make a difference. Any tips?
> >>
> >> ERROR: java.io.IOException: Invalid currKeyLen 16777280 or currValueLen
> >> 64. Block offset: 274609471488, block length: 16415, position: 0
> (without
> >> header).
> >>         at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2213)
> >>         at
> >> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
> >> Caused by: java.lang.IllegalStateException: Invalid currKeyLen 16777280
> >> or currValueLen 64. Block offset: 274609471488, block length: 16415,
> >> position: 0 (without header).
> >>         at
> >>
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:820)
> >>         at
> >>
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.updateCurrBlock(HFileReaderV2.java:787)
> >>         at
> >>
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:749)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:232)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:145)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:168)
> >>         at
> >> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1642)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3460)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1768)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1760)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1737)
> >>         at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2996)
> >>         at
> >>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
> >>         at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
> >>         ... 1 more
> >>
> >>
> >
>

--20cf3011d90b14143d04f4e4f1ea--