hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miklós Kurucz <mkur...@gmail.com>
Subject hbase.client.retries.number = 1 is bad
Date Mon, 03 May 2010 22:12:16 GMT
Hi!

I'm using a fresh version of trunk.
I'm experiencing a problem where the invalid region locations are not
removed from the cache of HCM.
I'm only using scanners on the table and I receive the following errors:

2010-05-03 23:42:52,574 DEBUG
org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at
'http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg'
2010-05-03 23:42:52,574 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache
hit for row <http://hu.gaabi.www/jordania/(041022)_jord-155_petra.jpg>
in tableName Test5: location server 10.1.3.111:60020, location region
name Test5,http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg,1272896369136
SEVERE: Trying to contact region server 10.1.3.111:60020 for region
Test5,http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg,1272896369136,
row 'http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg',
but failed after 1 attempts.
Exceptions:
java.net.ConnectException: Connection refused

Which is expected as the 10.1.3.111:60020 regionserver was offline for
hours at that time.
The cause of this problem is that I set hbase.client.retries.number to
1 as I don't like the current retry options.
In this case the following code at HConnectionManager.java:1061
   callable.instantiateServer(tries != 0);
will make scanners to always use the cache.
This makes hbase.client.retries.number = 1 an unusable option.

This is not intentional, am I correct?
Am I forced to use the retries, or is there an other option?

Also I would like to ask, when is it a good thing to retry an operation?
In my experience there exists two kinds of failures
1) org.apache.hadoop.hbase.NotServingRegionException : region is offline
This can be due to a compaction, in which case we probably need to
wait for a few seconds.
Or it can be due to a split, in which case we might need to wait for minutes.
Either case I would not want my client to wait for such long times
when I could reschedule other things to do in that time.
It is also possible that region has been transfered to an other
regionserver but that is rare compared to the other cases.

2) java.net.ConnectException : regionserver is offline
This is solved as soon as the master can reopen regions on an other
regionserver, but still can take minutes.
Anyway this exception is also rare(usually)

Best regards,
Miklos

Mime
View raw message