Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 209.85.160.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <53287DF001E4042800392A3F_0_66113@p057>
References: <53287DF001E4042800392A3F_0_66113@p057>
Date: Tue, 18 Mar 2014 10:26:48 -0700
Message-ID: 
 <CALte62wZRxidZHr8aDz3mZ7jPC7eEAKRRHXkWRcN1GMqT9Qv5g@mail.gmail.com>
Subject: Re: Corrupt HBase table
From: Ted Yu <yuzhihong@gmail.com>
To: Sudarshan Kadambi <skadambi@bloomberg.net>
Cc: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=20cf3011dff54749bf04f4e4d74e

--20cf3011dff54749bf04f4e4d74e
Content-Type: text/plain; charset=ISO-8859-1

The stack trace doesn't reveal too much other than the error occurred at
the beginning of a scan.

Other experts would have more insights than I do.

Cheers


On Tue, Mar 18, 2014 at 10:10 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) <
skadambi@bloomberg.net> wrote:

> Thanks Ted. No, we don't use data block encoding. This is 0.96. It appears
> multiple HFiles are affected. Does the stack trace give some context on
> what is that it's attempting to do at the lowest level?
>
>
> ----- Original Message -----
> From: yuzhihong@gmail.com
> To: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) <skadambi@bloomberg.net>,
> user@hbase.apache.org
> At: Mar 18 2014 13:03:20
>
> Are you using Data Block Encoding for this table ?
>
> Can you sideline the HFile and see if other HFiles were affected ?
>
> I assume you use 0.94.x
>
> Cheers
>
>
> On Tue, Mar 18, 2014 at 7:14 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN)
> <skadambi@bloomberg.net> wrote:
>
>> Hi Folks:
>>
>> We've a table that we're unable to make online. A scan on this table
>> yields the following stack trace. hbck doesn't report any inconsistencies.
>> I went ahead and did a repair, which didn't make a difference. Any tips?
>>
>> ERROR: java.io.IOException: Invalid currKeyLen 16777280 or currValueLen
>> 64. Block offset: 274609471488, block length: 16415, position: 0 (without
>> header).
>>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2213)
>>         at
>> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
>> Caused by: java.lang.IllegalStateException: Invalid currKeyLen 16777280
>> or currValueLen 64. Block offset: 274609471488, block length: 16415,
>> position: 0 (without header).
>>         at
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:820)
>>         at
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.updateCurrBlock(HFileReaderV2.java:787)
>>         at
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:749)
>>         at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:232)
>>         at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:145)
>>         at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:168)
>>         at
>> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1642)
>>         at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3460)
>>         at
>> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1768)
>>         at
>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1760)
>>         at
>> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1737)
>>         at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2996)
>>         at
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
>>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
>>         ... 1 more
>>
>>
>

--20cf3011dff54749bf04f4e4d74e--