hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: wrong region exception
Date Tue, 31 May 2011 19:57:10 GMT
Try adding this change:

Index: bin/check_meta.rb
===================================================================
--- bin/check_meta.rb   (revision 1129468)
+++ bin/check_meta.rb   (working copy)
@@ -127,11 +127,13 @@
 scan = Scan.new()
 scanner = metatable.getScanner(scan)
 oldHRI = nil
-bad = nil
+bad = 0
 while (result = scanner.next())
   rowid = Bytes.toString(result.getRow())
   rowidStr = java.lang.String.new(rowid)
   bytes = result.getValue(HConstants::CATALOG_FAMILY,
HConstants::REGIONINFO_QUALIFIER)
+  next if not bytes
+  next if bytes.length == 0
   hri = Writables.getHRegionInfo(bytes)
   if oldHRI
     if oldHRI.isOffline() && Bytes.equals(oldHRI.getStartKey(),
hri.getStartKey())


You might print out the result you have if the qualifier is null just
to see what are the rows missing an HRegionInfo.

St.Ack

On Tue, May 31, 2011 at 9:02 AM, Robert Gonzalez
<Robert.Gonzalez@maxpointinteractive.com> wrote:
> I'm trying my "nuclear" option: basically copy the data from the old db to a new one,
skipping over bad regions.  The bad news is that it is taking forever.
>
>
> I get a stack trace just trying to run check_meta.rb:
>
> maxpoint@c1-m02:/usr/lib/hbase/bin$ ./hbase org.jruby.Main check_meta.rb
> Writables.java:75:in `org.apache.hadoop.hbase.util.Writables.getWritable': java.lang.NullPointerException:
null (NativeException)
>        from Writables.java:119:in `org.apache.hadoop.hbase.util.Writables.getHRegionInfo'
>        from NativeMethodAccessorImpl.java:-2:in `sun.reflect.NativeMethodAccessorImpl.invoke0'
>        from NativeMethodAccessorImpl.java:39:in `sun.reflect.NativeMethodAccessorImpl.invoke'
>        from DelegatingMethodAccessorImpl.java:25:in `sun.reflect.DelegatingMethodAccessorImpl.invoke'
>        from Method.java:597:in `java.lang.reflect.Method.invoke'
>        from JavaMethod.java:196:in `org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling'
>        from JavaMethod.java:182:in `org.jruby.javasupport.JavaMethod.invoke_static'
>        from JavaClass.java:371:in `org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute'
>         ... 17 levels...
>        from Main.java:183:in `org.jruby.Main.runInterpreter'
>        from Main.java:120:in `org.jruby.Main.run'
>        from Main.java:95:in `org.jruby.Main.main'
> Complete Java stackTrace
> java.lang.NullPointerException
>        at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
>        at org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:119)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:196)
>        at org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182)
>        at org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass.java:371)
>        at org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCallbackMethod.java:81)
>        at org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571)
>        at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:207)
>        at org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java:1254)
>        at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:286)
>        at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
>        at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
>        at org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:1793)
>        at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:387)
>        at org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533)
>        at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:201)
>        at org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628)
>        at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:356)
>        at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164)
>        at org.jruby.Ruby.eval(Ruby.java:278)
>        at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306)
>        at org.jruby.Main.runInterpreter(Main.java:238)
>        at org.jruby.Main.runInterpreter(Main.java:183)
>        at org.jruby.Main.run(Main.java:120)
>        at org.jruby.Main.main(Main.java:95)
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Friday, May 27, 2011 12:43 AM
> To: user@hbase.apache.org
> Subject: Re: wrong region exception
>
> Robert:
>
> Looks like script already exists.  Check bin/check_meta.rb.  If you pass it --fix it
should plug the hole.  Read the head of the script for how to run it.
>
> Good luck,
> St.Ack
>
> On Thu, May 26, 2011 at 1:06 PM, Robert Gonzalez <Robert.Gonzalez@maxpointinteractive.com>
wrote:
>> I sent the meta.txt to your saint.ack@gmail .com account due to the attachment.
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>> Stack
>> Sent: Thursday, May 26, 2011 1:35 PM
>> To: user@hbase.apache.org
>> Subject: Re: wrong region exception
>>
>> On Thu, May 26, 2011 at 8:06 AM, Robert Gonzalez <Robert.Gonzalez@maxpointinteractive.com>
wrote:
>>> It looks like an entire region is missing, here is the online table:
>>>
>>> urlhashv2,7FF1A5BF839C37078083B4F8267008F6,1303028235302.b0d55566fd0e
>>> 0 2ae98541618396aa7b1.
>>> c1-s03.atxd.maxpointinteractive.com:60030
>>> 7FF1A5BF839C37078083B4F8267008F6
>>> 80116D7E506D87ED39EAFFE784B5B590
>>> urlhashv2,8031483E0B3B7F587020FCBB764272D9,1305226123483.3ed065ad87f8
>>> 9 aece6b994dd31b42b2a.      c1-s33.atxd.maxpointinteractive.com:60030
>>> 8031483E0B3B7F587020FCBB764272D9
>>> 8041346D0B05617FA4B9152BFE9B18B9
>>>
>>> One ends at 80116D7E506D87ED39EAFFE784B5B590, but the next one doesn't start
there.
>>>
>>
>> So, make sure you actually have a hole.  Dump out your meta table:
>>
>> echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt
>>
>> Then look ensure that there is a hole between the above regions (compare start and
end keys... the end key of one region needs to match the start key of the next).
>>
>> If indeed a hole, you need to do a little surgery inserting a new missing region
(hbck should fix this but it doesn't have the smarts just yet).
>>
>> Basically, you create a new region with start and end keys to fill the hole then
you insert it into .META. and then assign it.  There are some scripts in our bin directory
that do various parts of this.  I'm pretty sure its beyond any but a few figuring this mess
out so if you do the above foot work and provide a few more details, I'll hack up something
for you (and hopefully something generalized to be use by others later, and later to be integrated
into hbck).
>>
>> St.Ack
>>
>

Mime
View raw message