hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geoff Hendrey" <ghend...@decarta.com>
Subject RE: corrupt .logs block
Date Fri, 12 Aug 2011 05:24:20 GMT
Thanks, 

check_meta.rb stack traces with NPE...

[hroot@doop10 bin]$ hbase org.jruby.Main check_meta.rb
Writables.java:75:in
`org.apache.hadoop.hbase.util.Writables.getWritable':
java.lang.NullPointerException: null (NativeException)
        from Writables.java:119:in
`org.apache.hadoop.hbase.util.Writables.getHRegionInfo'
        from null:-1:in `sun.reflect.GeneratedMethodAccessor6.invoke'
        from DelegatingMethodAccessorImpl.java:43:in
`sun.reflect.DelegatingMethodAccessorImpl.invoke'
        from Method.java:616:in `java.lang.reflect.Method.invoke'
        from JavaMethod.java:196:in
`org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling'
        from JavaMethod.java:182:in
`org.jruby.javasupport.JavaMethod.invoke_static'
        from JavaClass.java:371:in
`org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute'
        from SimpleCallbackMethod.java:81:in
`org.jruby.internal.runtime.methods.SimpleCallbackMethod.call'
         ... 16 levels...
        from Main.java:183:in `org.jruby.Main.runInterpreter'
        from Main.java:120:in `org.jruby.Main.run'
        from Main.java:95:in `org.jruby.Main.main'
Complete Java stackTrace
java.lang.NullPointerException
        at
org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
        at
org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119
)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.
java:196)
        at
org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182)
        at
org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass.ja
va:371)
        at
org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCallb
ackMethod.java:81)
        at
org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571)
        at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:20
7)
        at
org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java:1
254)
        at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:28
6)
        at
org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
        at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:20
1)
        at
org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:1793)
        at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:38
7)
        at
org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
        at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:20
1)
        at
org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628)
        at
org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:35
6)
        at
org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164)
        at org.jruby.Ruby.eval(Ruby.java:278)
        at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306)
        at org.jruby.Main.runInterpreter(Main.java:238)
        at org.jruby.Main.runInterpreter(Main.java:183)
        at org.jruby.Main.run(Main.java:120)
        at org.jruby.Main.main(Main.java:95)

-----Original Message-----
From: Jinsong Hu [mailto:jinsong_hu@hotmail.com] 
Sent: Thursday, August 11, 2011 3:18 PM
To: user@hbase.apache.org
Subject: Re: corrupt .logs block

as I said, run "hbase org.jruby.Main add_table.rb <table_name>" first,
then 
run "hbase org.jruby.Main check_meta.rb --fix"
then restart hbase.

It doesn't completely solve problem for me, as hbck still complains.
but at least it recovers all data and I can do full rowcount for the
table.


Jimmy.

--------------------------------------------------
From: "Geoff Hendrey" <ghendrey@decarta.com>
Sent: Thursday, August 11, 2011 2:21 PM
To: "Jinsong Hu" <jinsong_hu@hotmail.com>; <user@hbase.apache.org>
Subject: RE: corrupt .logs block

> Hey -
>
> Our table behaves fine until we try to do a mapreduce job that reads
and
> writes from the table. When we try to retrieve keys from the afflicted
> regions, the job just hangs forever. It's interesting because we never
> get timeouts of any sort. This is different than other failures we've
> seen in which we'd get expired leases. This is a critical bug for us
> because it is preventing the launch of a product databuild which I
have
> to complete in the next week.
>
> Does anyone have any suggestions as to how I can bring the afflicted
> regions online? Worst case, delete the regions?
>
> -geoff
>
> -----Original Message-----
> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> Sent: Thursday, August 11, 2011 11:47 AM
> To: user@hbase.apache.org
> Cc: Search
> Subject: Re: corrupt .logs block
>
> I run into same issue. I tried check_meta.rb --fix and add_table.rb,
and
>
> still get the same hbck "inconsistent" table,
> however, I am able to do a rowcount for the table and there is no
> problem.
>
> Jimmy
>
>
> --------------------------------------------------
> From: "Geoff Hendrey" <ghendrey@decarta.com>
> Sent: Thursday, August 11, 2011 10:36 AM
> To: <user@hbase.apache.org>
> Cc: "Search" <Search@decarta.com>
> Subject: RE: corrupt .logs block
>
>> so I delete the corrpupt .logs files. OK, fine no more issue there.
> But a
>> handful of regions in a very large table (2000+ regions) are offline
>> (".META." says offline=true).
>>
>> How do I go about trying to get the region online, and how come
> restarting
>> hbase has no effect (region still offline).
>>
>> Tried 'hbck -fix', no effect. Hbck simply lists the table as
>> "inconsistent".
>>
>> Would appreciate any advice on how to resolve this.
>>
>> Thanks,
>> geoff
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> Stack
>> Sent: Monday, August 08, 2011 4:25 PM
>> To: user@hbase.apache.org
>> Subject: Re: corrupt .logs block
>>
>> Well, if its a log no longer used, then you could just delete it.
>> That'll get rid of the fsck complaint (True, logs are not per table
so
>> to be safe you'd need to flush all tables -- this would get all edits
>> that the log could be carrying out into the filesystem into hfiles).
>>
>> St.Ack
>>
>> On Mon, Aug 8, 2011 at 4:20 PM, Geoff Hendrey <ghendrey@decarta.com>
>> wrote:
>>> Ah. Thanks for that. No, I don't need the log anymore. I am aware of
> how
>>> to flush a table from the hbase shell. But since the "fsck /" tells
> me a
>>> log file is corrupt, but not which table the corruption pertains to,
>>> does this mean I have to flush all my tables (I have a lot of
> tables).
>>>
>>> -geoff
>>>
>>> -----Original Message-----
>>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>>> Stack
>>> Sent: Monday, August 08, 2011 4:09 PM
>>> To: user@hbase.apache.org
>>> Subject: Re: corrupt .logs block
>>>
>>> On Sat, Aug 6, 2011 at 12:12 PM, Geoff Hendrey
<ghendrey@decarta.com>
>>> wrote:
>>>> I've got a corrupt HDFS block in a region server's ".logs"
> directory.
>>>
>>> You see this when you do hdfs fsck?  Is the log still needed?  You
>>> could do a flush across the cluster and that should do away with
your
>>> dependency on this log.
>>>
>>> St.Ack
>>>
>>
> 

Mime
View raw message