hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohit Nigam" <rni...@decarta.com>
Subject RE: corrupt .logs block
Date Fri, 12 Aug 2011 17:49:59 GMT
Hi
We are having a issue while running the job as it hangs forever at 99%
(reducer) inserting into the table , hbase  hbck -details  finds
inconsistency in the table I am trying to insert data as :--

Chain of regions in table NAM_CLUSTERKEYS3 is broken; edges does not
contain
00000000|00000000000000000USA|0000240b|000000000000000LAKE,|002ff027|000
0000000CONSULTING|006e8f90|000000000000000000NY:0
I tried hbase hbck -fix  but does not help.

Tried scanning  the '.META' table

scan
'.META.',{STARTROW=>'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000
240b|000000000000000LAKE',LIMIT=>1}

Getting this error:--


ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying
to contact region server doop6. for region .META.,,1, row
'NAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|000000000000000
LAKE', but failed after 7 attempts.
Exceptions:
java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
        at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)
        at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)
        at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)
        at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)
        at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContainingKey(HFile.j
ava:918)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:
1296)
        at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(Stor
eFileScanner.java:136)
        at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScan
ner.java:96)
        at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.ja
va:77)
        at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1341)
        at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegio
n.java:2269)
        at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(
HRegion.java:1126)
        at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:111
8)
        at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:110
2)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1781)
        ... 5 more

java.io.IOException: java.io.IOException:
java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:992)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE
(HRegionServer.java:981)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSe
rver.java:1783)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Caused by: java.lang.IllegalArgumentException: No 44 in
<WKNAM_CLUSTERKEYS3,00000000|00000000000000000USA|0000240b|0000000000000
00LAK????????>, length=58, offset=27
        at
org.apache.hadoop.hbase.KeyValue.getRequiredDelimiterInReverse(KeyValue.
java:1281)
        at
org.apache.hadoop.hbase.KeyValue$MetaKeyComparator.compareRows(KeyValue.
java:1827)
        at
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:186
6)
        at
org.apache.hadoop.hbase.util.Bytes.binarySearch(Bytes.java:1159)
        at
org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.blockContainingKey(HFi
le.java:1618)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.blockContaini

How can I fix the problem in the table . Any help would be appreciated. 

Rohit

-----Original Message-----
From: Geoff Hendrey 
Sent: Thursday, August 11, 2011 2:22 PM
To: Jinsong Hu; user@hbase.apache.org
Subject: RE: corrupt .logs block

Hey -

Our table behaves fine until we try to do a mapreduce job that reads and
writes from the table. When we try to retrieve keys from the afflicted
regions, the job just hangs forever. It's interesting because we never
get timeouts of any sort. This is different than other failures we've
seen in which we'd get expired leases. This is a critical bug for us
because it is preventing the launch of a product databuild which I have
to complete in the next week.

Does anyone have any suggestions as to how I can bring the afflicted
regions online? Worst case, delete the regions?

-geoff

-----Original Message-----
From: Jinsong Hu [mailto:jinsong_hu@hotmail.com] 
Sent: Thursday, August 11, 2011 11:47 AM
To: user@hbase.apache.org
Cc: Search
Subject: Re: corrupt .logs block

I run into same issue. I tried check_meta.rb --fix and add_table.rb, and

still get the same hbck "inconsistent" table,
however, I am able to do a rowcount for the table and there is no
problem.

Jimmy


--------------------------------------------------
From: "Geoff Hendrey" <ghendrey@decarta.com>
Sent: Thursday, August 11, 2011 10:36 AM
To: <user@hbase.apache.org>
Cc: "Search" <Search@decarta.com>
Subject: RE: corrupt .logs block

> so I delete the corrpupt .logs files. OK, fine no more issue there.
But a 
> handful of regions in a very large table (2000+ regions) are offline 
> (".META." says offline=true).
>
> How do I go about trying to get the region online, and how come
restarting 
> hbase has no effect (region still offline).
>
> Tried 'hbck -fix', no effect. Hbck simply lists the table as 
> "inconsistent".
>
> Would appreciate any advice on how to resolve this.
>
> Thanks,
> geoff
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
> Sent: Monday, August 08, 2011 4:25 PM
> To: user@hbase.apache.org
> Subject: Re: corrupt .logs block
>
> Well, if its a log no longer used, then you could just delete it.
> That'll get rid of the fsck complaint (True, logs are not per table so
> to be safe you'd need to flush all tables -- this would get all edits
> that the log could be carrying out into the filesystem into hfiles).
>
> St.Ack
>
> On Mon, Aug 8, 2011 at 4:20 PM, Geoff Hendrey <ghendrey@decarta.com> 
> wrote:
>> Ah. Thanks for that. No, I don't need the log anymore. I am aware of
how
>> to flush a table from the hbase shell. But since the "fsck /" tells
me a
>> log file is corrupt, but not which table the corruption pertains to,
>> does this mean I have to flush all my tables (I have a lot of
tables).
>>
>> -geoff
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>> Stack
>> Sent: Monday, August 08, 2011 4:09 PM
>> To: user@hbase.apache.org
>> Subject: Re: corrupt .logs block
>>
>> On Sat, Aug 6, 2011 at 12:12 PM, Geoff Hendrey <ghendrey@decarta.com>
>> wrote:
>>> I've got a corrupt HDFS block in a region server's ".logs"
directory.
>>
>> You see this when you do hdfs fsck?  Is the log still needed?  You
>> could do a flush across the cluster and that should do away with your
>> dependency on this log.
>>
>> St.Ack
>>
> 

Mime
View raw message