hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joost Ouwerkerk <jo...@openplaces.org>
Subject Re: WrongRegionException: How do I recover?
Date Thu, 05 Nov 2009 00:07:42 GMT
I investigated following your guidance, Stack.  Unfortunately I am not
seeing evidence of double assignment. It looks more like a case of missing
assignment.  There appear to be key ranges that are not represented in the
.META. table.  So, I have a region that handles keys AAA to BBB, and the
next region handles DDD to EEE.  Now when I try to access key CCC, I get
routed to the region that handles AAA to BBB, presumably because my key is
after AAA and before DDD.  Then HRegion.checkRow fails because the requested
key is outside the region's range.

Consider this error:

org.apache.hadoop.hbase.regionserver.WrongRegionException:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
of range for HRegion
crawled_pages,r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fbasil-in-the-grove,
startKey
='r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fbasil-in-the-grove',
getEndKey()
='r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Feast-broward',
row
='r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fhavana-hideout'

As the error points out, the requested row is outside the range for the
region.  In the .META. table, the next region starts at
'r:http:\x2F\x2Fcom.xxx.yyy\x2Frestaurants\x2Fpashas-3'.  The request row
falls after one region's End key, and before the next region's Start key.

jo

On Wed, Nov 4, 2009 at 4:56 PM, stack <stack@duboce.net> wrote:

> Meta is giving out the wrong address for a region?  Do a scan of .META.  It
> might be easier dumping the scan into a file so you can grep around:
>
> echo "scan '.META.'" | ./bin/hbase shell --format-width=300 &>
> /tmp/meta.txt
>
> Grep in here for the region that contains the row you are looking for.
>  What
> does it have for info:server?  Go to that regionserver (UI or log).  Is it
> carrying the region?  If not, thats what the WRE is about.
>
> For same region, grep its name in master log (hopefully you have DEBUG
> enabled).
>
> Whats its history?  Could it have been assigned to one server and then
> another?
>
> If so, close the region in both places.  Type 'tools' in the shell to see
> doc. on "close_region" command.  You can pass it server to pass the close
> message to.  Close in both places.
>
> If its a double-assignment issue, our name for above phenomeon, suggest you
> upgrade to 0.20.1.  It has at least one pointed fix for this scenario
> (HBASE-1878).
>
> St.Ack
>
>
> On Wed, Nov 4, 2009 at 12:35 PM, Joost Ouwerkerk <joost@openplaces.org
> >wrote:
>
> > HBase has started throwing WrongRegionExceptions at me when trying to
> > access
> > certain regions.  I'm guessing that the META table has somehow gone out
> of
> > sync with reality.  I've tried compacting and I've tried restarting, but
> > the
> > problem does not go away.  The errors are always on the same regions.
>  Has
> > anyone else seen this and succeeded at getting their table back into
> > working
> > order?
> >
> > *Example get:*
> >
> > org.apache.hadoop.hbase.regionserver.WrongRegionException:
> > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> > out
> > of range for HRegion
> >
> >
> crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084,
> >
> >
> startKey='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F',
> >
> >
> getEndKey()='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fhermosa-beach\x2Fall-cuisines\x2Ftags\x2Foutdoor-dining\x2F',
> >
> >
> row='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F'
> >    at
> > org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1522)
> >    at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1554)
> >    at
> > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1622)
> >    at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2278)
> >    at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
> >    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >    at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >    at java.lang.reflect.Method.invoke(Method.java:597)
> >    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> >    at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> >
> > *Example put:
> > *
> > put 'crawled_pages','r:
> > http://com.xxxx.yyyy/restaurants/all-areas/inglewood/all-cuisines/',
> > 'curi:test','test'
> > NativeException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException:
> > Trying to contact region server Some server, retryOnlyOne=true, index=0,
> > islastrow=true, tries=4, numtries=5, i=0, listsize=1,
> >
> >
> region=crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084
> > for region
> >
> >
> crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084,
> > row
> >
> >
> 'r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F',
> > but failed after 5 attempts.
> > Exceptions:
> >
> >    from org/apache/hadoop/hbase/client/HConnectionManager.java:1119:in
> > `process'
> >    from org/apache/hadoop/hbase/client/HConnectionManager.java:1200:in
> > `processBatchOfRows'
> >    from org/apache/hadoop/hbase/client/HTable.java:605:in `flushCommits'
> >    from org/apache/hadoop/hbase/client/HTable.java:470:in `put'
> >    from org/apache/hadoop/hbase/client/HTable.java:1761:in `commit'
> >    from org/apache/hadoop/hbase/client/HTable.java:1742:in `commit'
> >    from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
> >    from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
> >    from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >    from java/lang/reflect/Method.java:597:in `invoke'
> >    from org/jruby/javasupport/JavaMethod.java:298:in
> > `invokeWithExceptionHandling'
> >    from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >    from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call'
> >    from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call'
> >    from org/jruby/ast/CallOneArgNode.java:57:in `interpret'
> >    from org/jruby/ast/NewlineNode.java:104:in `interpret'
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message