hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Gonzalez <Robert.Gonza...@maxpointinteractive.com>
Subject RE: wrong region exception
Date Tue, 31 May 2011 22:34:35 GMT
The script doesn't work because it attempts to fix the hole by finding a region in the hdfs
filesystem that fills the hole.  But in this case there is no such file.  The hole is just
there.

-----Original Message-----
From: Robert Gonzalez [mailto:Robert.Gonzalez@maxpointinteractive.com] 
Sent: Tuesday, May 31, 2011 5:20 PM
To: 'user@hbase.apache.org'
Subject: RE: wrong region exception

The script ran without the previous problem, but it did not fix the problem.  When I ran hbck
or check_meta.rb again they indicated that the problem was still there.  Do I need to do something
else in preparation before running check_meta?

Thanks,

Robert


-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Tuesday, May 31, 2011 2:57 PM
To: user@hbase.apache.org
Subject: Re: wrong region exception

Try adding this change:

Index: bin/check_meta.rb
===================================================================
--- bin/check_meta.rb   (revision 1129468)
+++ bin/check_meta.rb   (working copy)
@@ -127,11 +127,13 @@
 scan = Scan.new()
 scanner = metatable.getScanner(scan)
 oldHRI = nil
-bad = nil
+bad = 0
 while (result = scanner.next())
   rowid = Bytes.toString(result.getRow())
   rowidStr = java.lang.String.new(rowid)
   bytes = result.getValue(HConstants::CATALOG_FAMILY,
HConstants::REGIONINFO_QUALIFIER)
+  next if not bytes
+  next if bytes.length == 0
   hri = Writables.getHRegionInfo(bytes)
   if oldHRI
     if oldHRI.isOffline() && Bytes.equals(oldHRI.getStartKey(),
hri.getStartKey())


You might print out the result you have if the qualifier is null just to see what are the
rows missing an HRegionInfo.

St.Ack

On Tue, May 31, 2011 at 9:02 AM, Robert Gonzalez <Robert.Gonzalez@maxpointinteractive.com>
wrote:
> I'm trying my "nuclear" option: basically copy the data from the old db to a new one,
skipping over bad regions.  The bad news is that it is taking forever.
>
>
> I get a stack trace just trying to run check_meta.rb:
>
> maxpoint@c1-m02:/usr/lib/hbase/bin$ ./hbase org.jruby.Main 
> check_meta.rb Writables.java:75:in
> `org.apache.hadoop.hbase.util.Writables.getWritable': 
> java.lang.NullPointerException: null (NativeException)
>        from Writables.java:119:in `org.apache.hadoop.hbase.util.Writables.getHRegionInfo'
>        from NativeMethodAccessorImpl.java:-2:in `sun.reflect.NativeMethodAccessorImpl.invoke0'
>        from NativeMethodAccessorImpl.java:39:in `sun.reflect.NativeMethodAccessorImpl.invoke'
>        from DelegatingMethodAccessorImpl.java:25:in `sun.reflect.DelegatingMethodAccessorImpl.invoke'
>        from Method.java:597:in `java.lang.reflect.Method.invoke'
>        from JavaMethod.java:196:in `org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling'
>        from JavaMethod.java:182:in `org.jruby.javasupport.JavaMethod.invoke_static'
>        from JavaClass.java:371:in `org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute'
>         ... 17 levels...
>        from Main.java:183:in `org.jruby.Main.runInterpreter'
>        from Main.java:120:in `org.jruby.Main.run'
>        from Main.java:95:in `org.jruby.Main.main'
> Complete Java stackTrace
> java.lang.NullPointerException
>        at
> org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75)
>        at
> org.apache.hadoop.hbase.util.Writables.getHRegionInfo(Writables.java:1
> 19)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMetho
> d.java:196)
>        at
> org.jruby.javasupport.JavaMethod.invoke_static(JavaMethod.java:182)
>        at
> org.jruby.javasupport.JavaClass$StaticMethodInvoker.execute(JavaClass.
> java:371)
>        at
> org.jruby.internal.runtime.methods.SimpleCallbackMethod.call(SimpleCal
> lbackMethod.java:81)
>        at
> org.jruby.evaluator.EvaluationState.callNode(EvaluationState.java:571)
>        at
> org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:
> 207)
>        at
> org.jruby.evaluator.EvaluationState.localAsgnNode(EvaluationState.java
> :1254)
>        at
> org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:
> 286)
>        at
> org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533
> )
>        at
> org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:
> 201)
>        at
> org.jruby.evaluator.EvaluationState.whileNode(EvaluationState.java:179
> 3)
>        at
> org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:
> 387)
>        at
> org.jruby.evaluator.EvaluationState.blockNode(EvaluationState.java:533
> )
>        at
> org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:
> 201)
>        at
> org.jruby.evaluator.EvaluationState.rootNode(EvaluationState.java:1628
> )
>        at
> org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:
> 356)
>        at
> org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:164)
>        at org.jruby.Ruby.eval(Ruby.java:278)
>        at org.jruby.Ruby.compileOrFallbackAndRun(Ruby.java:306)
>        at org.jruby.Main.runInterpreter(Main.java:238)
>        at org.jruby.Main.runInterpreter(Main.java:183)
>        at org.jruby.Main.run(Main.java:120)
>        at org.jruby.Main.main(Main.java:95)
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of 
> Stack
> Sent: Friday, May 27, 2011 12:43 AM
> To: user@hbase.apache.org
> Subject: Re: wrong region exception
>
> Robert:
>
> Looks like script already exists.  Check bin/check_meta.rb.  If you pass it --fix it
should plug the hole.  Read the head of the script for how to run it.
>
> Good luck,
> St.Ack
>
> On Thu, May 26, 2011 at 1:06 PM, Robert Gonzalez <Robert.Gonzalez@maxpointinteractive.com>
wrote:
>> I sent the meta.txt to your saint.ack@gmail .com account due to the attachment.
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of 
>> Stack
>> Sent: Thursday, May 26, 2011 1:35 PM
>> To: user@hbase.apache.org
>> Subject: Re: wrong region exception
>>
>> On Thu, May 26, 2011 at 8:06 AM, Robert Gonzalez <Robert.Gonzalez@maxpointinteractive.com>
wrote:
>>> It looks like an entire region is missing, here is the online table:
>>>
>>> urlhashv2,7FF1A5BF839C37078083B4F8267008F6,1303028235302.b0d55566fd0
>>> e
>>> 0 2ae98541618396aa7b1.
>>> c1-s03.atxd.maxpointinteractive.com:60030
>>> 7FF1A5BF839C37078083B4F8267008F6
>>> 80116D7E506D87ED39EAFFE784B5B590
>>> urlhashv2,8031483E0B3B7F587020FCBB764272D9,1305226123483.3ed065ad87f
>>> 8
>>> 9 aece6b994dd31b42b2a.
>>> c1-s33.atxd.maxpointinteractive.com:60030
>>> 8031483E0B3B7F587020FCBB764272D9
>>> 8041346D0B05617FA4B9152BFE9B18B9
>>>
>>> One ends at 80116D7E506D87ED39EAFFE784B5B590, but the next one doesn't start
there.
>>>
>>
>> So, make sure you actually have a hole.  Dump out your meta table:
>>
>> echo "scan '.META.'"| ./bin/hbase shell &> /tmp/meta.txt
>>
>> Then look ensure that there is a hole between the above regions (compare start and
end keys... the end key of one region needs to match the start key of the next).
>>
>> If indeed a hole, you need to do a little surgery inserting a new missing region
(hbck should fix this but it doesn't have the smarts just yet).
>>
>> Basically, you create a new region with start and end keys to fill the hole then
you insert it into .META. and then assign it.  There are some scripts in our bin directory
that do various parts of this.  I'm pretty sure its beyond any but a few figuring this mess
out so if you do the above foot work and provide a few more details, I'll hack up something
for you (and hopefully something generalized to be use by others later, and later to be integrated
into hbck).
>>
>> St.Ack
>>
>

Mime
View raw message