hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: NoServerForRegionException, TableNotFoundException and WrongRegionException
Date Mon, 17 Aug 2009 21:21:47 GMT
Please update to the head of 0.19 trunk, or better update to 0.20 trunk --
espeically if you are testing.  Issues described below have been addressed.

How many regions do you have in your table?  Are all going to one
regionserver because you only have one region?

Yours,
St.Ack


On Mon, Aug 17, 2009 at 12:19 PM, Marc Limotte <mslimotte@gmail.com> wrote:

> I'm seeing a nice variety of Exceptions from HBase and could use some
> pointers about what to do next.
>
> This is a new map/reduce program, updating about 550k rows with around a
> dozen columns on a very small cluster (only 4 nodes... as we're still
> testing and it doesn't have to support production yet).  Hbase Version
> 0.19.1.
>
> I ran the job and it seems to make some progress, and then dies after
> several hours, reporting "NoServerForRegionException: No server address
> listed in .META. for region TABLEX,,1250526695078".  I retried it a few
> times with the same result.  I also noticed that the load is not well
> balanced, all requests seemed to be going to one node.  I adjust
> hadoop-site.xml with the addition of these two entries:
>
>    <name>hbase.hregion.max.filesize</name>
>    <value>33554432</value>
>
>   <name>hbase.client.retries.number</name>
>    <value>5</value>
>
> And restarted hbase (and hadoop to be safe).  Re-ran and got the same error
> in the M/R job.
>
> *I thought I'd try dropping the table, since it's a new table and I can
> recreate it.  But that gives another exception:
> *
> hbase(main):002:0> disable 'TABLEX'
> NativeException: org.apache.hadoop.hbase.TableNotFoundException:
> org.apache.hadoop.hbase.TableNotFoundException: TABLEX
>    at
>
> org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:129)
>    at
>
> org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:70)
>    at
>
> org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
>    at
>
> org.apache.hadoop.hbase.master.TableOperation.process(TableOperation.java:143)
>    at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:691)
> ...
>
>
> *And now I see this exception in the HBase logs:
> *
> org.apache.hadoop.hbase.regionserver.WrongRegionException:
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> out
> of range for HRegion .META.,,1250280235390, startKey='',
> getEndKey()='TABLEX,,1250219949252',
> row='TABLEX,840.56098.0544,1250526661861'
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1788)
>    at
>
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1844)
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1912)
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1244)
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1216)
> ...
>
>
> *As a test, tried a "count"...
> *
> hbase(main):007:0* count 'TABLEX'
> NativeException: org.apache.hadoop.hbase.client.NoServerForRegionException:
> No server address listed in .META. for region TABLEX,,1250526695078
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:548:in
> `locateRegionInMeta'
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:478:in
> `locateRegion'
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:440:in
> `locateRegion'
>    from org/apache/hadoop/hbase/client/HTable.java:114:in `<init>'
>    from org/apache/hadoop/hbase/client/HTable.java:97:in `<init>'
>    from sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
> ...
>
>
> *Also saw a thread somewhere that suggested doing a major compaction.  Did
> that.  It returns almost immediately.  Not sure if that's normal or not...
> no perceivable impact from doing this, though.*
>
> hbase(main):013:0> major_compact '.META.'
> 0 row(s) in 0.0220 seconds
> hbase(main):014:0>
>
> Not sure what else to try?  Is there a way to force removal of the table in
> question?  Is there something else I should be looking at?
>
> Marc
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message