hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miklós Kurucz <mkur...@gmail.com>
Subject hbase.client.retries.number = 1 is bad
Date Mon, 03 May 2010 22:12:16 GMT

I'm using a fresh version of trunk.
I'm experiencing a problem where the invalid region locations are not
removed from the cache of HCM.
I'm only using scanners on the table and I receive the following errors:

2010-05-03 23:42:52,574 DEBUG
org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at
2010-05-03 23:42:52,574 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache
hit for row <http://hu.gaabi.www/jordania/(041022)_jord-155_petra.jpg>
in tableName Test5: location server, location region
name Test5,http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg,1272896369136
SEVERE: Trying to contact region server for region
row 'http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg',
but failed after 1 attempts.
java.net.ConnectException: Connection refused

Which is expected as the regionserver was offline for
hours at that time.
The cause of this problem is that I set hbase.client.retries.number to
1 as I don't like the current retry options.
In this case the following code at HConnectionManager.java:1061
   callable.instantiateServer(tries != 0);
will make scanners to always use the cache.
This makes hbase.client.retries.number = 1 an unusable option.

This is not intentional, am I correct?
Am I forced to use the retries, or is there an other option?

Also I would like to ask, when is it a good thing to retry an operation?
In my experience there exists two kinds of failures
1) org.apache.hadoop.hbase.NotServingRegionException : region is offline
This can be due to a compaction, in which case we probably need to
wait for a few seconds.
Or it can be due to a split, in which case we might need to wait for minutes.
Either case I would not want my client to wait for such long times
when I could reschedule other things to do in that time.
It is also possible that region has been transfered to an other
regionserver but that is rare compared to the other cases.

2) java.net.ConnectException : regionserver is offline
This is solved as soon as the master can reopen regions on an other
regionserver, but still can take minutes.
Anyway this exception is also rare(usually)

Best regards,

View raw message