hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: recovery from regionserver death
Date Wed, 22 Jun 2011 17:45:48 GMT
Hadoop and HBase versions please.

(no you shouldn't have to do anything special)

J-D

On Wed, Jun 22, 2011 at 9:44 AM, Robert Gonzalez
<Robert.Gonzalez@maxpointinteractive.com> wrote:
> How does one recover when a regionserver dies?  We have this problem periodically and
we basically have to restart hbase or all our jobs die with these type of errors:
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
c1-s35.blablabla.com:60020 for region urlhashv4,F4657B47F9881A42AF88864EC5EA9B27,1307217134729.4fa3defeeaeb59dc56f7ce6f155b2a0b.,
row 'F471203BA4FF5DD2BD2549308FD81F4A', but failed after 10 attempts.
> Exceptions:
>
>
> Then eventually this results in a general failure with Wrong Region exceptions and the
whole table seems to go corrupt.  The errors one sees at the regionserver level are:
>
> 2011-06-22 10:32:35,559 WARN org.apache.hadoop.hbase.regionserver.HRegion: File hdfs://c1-m01:54310/hbase/urlhashv4/d3c3f27ac1ce7a2dff35ddf367fe779d/recovered.edits/0000000000097403816
is zero-length, deleting.
> 2011-06-22 10:32:35,563 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Failed delete
of hdfs://c1-m01:54310/hbase/urlhashv4/d3c3f27ac1ce7a2dff35ddf367fe779d/recovered.edits/0000000000097403816
> 2011-06-22 10:33:19,769 WARN org.apache.hadoop.hbase.regionserver.HRegion: File hdfs://c1-m01:54310/hbase/urlhashv4/9d0d6214bebdefd5466d0e6918c3630c/recovered.edits/0000000000097403669
is zero-length, deleting.
> 2011-06-22 10:33:19,770 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Failed delete
of hdfs://c1-m01:54310/hbase/urlhashv4/9d0d6214bebdefd5466d0e6918c3630c/recovered.edits/0000000000097403669
>
>
> Shouldn't the master detect deaths and rebalance the regions to other regionservers?
 Or is there a manual way to do this without having to restart the whole thing?
>
> Thanks,
>
> Robert Gonzalez
> Maxpoint Interactive
>
>
>
>

Mime
View raw message