Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of mkurucz@gmail.com designates
 74.125.82.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=mRG0Uj20tFHlKMJtyC3hEVmrOPqWp6K1hKcXmKZJD87zz6T9f8S6nSxwAC4yEITd1q
         Sg5F8SNDQrHunNR22pSSQpATn5BuzKCu0rawS4ijxbItEsInTmAJMKCMa81Zg0jp6kn5
         Wvw2URuoHWr6GY1H3/EMw5Qgnv10tucPLr+Wk=
MIME-Version: 1.0
Date: Tue, 4 May 2010 00:12:16 +0200
Message-ID: <r2tf01f65c21005031512p56e18a1bw21fea1000252253a@mail.gmail.com>
Subject: hbase.client.retries.number = 1 is bad
From: =?ISO-8859-1?Q?Mikl=F3s_Kurucz?= <mkurucz@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi!

I'm using a fresh version of trunk.
I'm experiencing a problem where the invalid region locations are not
removed from the cache of HCM.
I'm only using scanners on the table and I receive the following errors:

2010-05-03 23:42:52,574 DEBUG
org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing
internal scanner to startKey at
'http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg'
2010-05-03 23:42:52,574 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache
hit for row <http://hu.gaabi.www/jordania/(041022)_jord-155_petra.jpg>
in tableName Test5: location server 10.1.3.111:60020, location region
name Test5,http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg,1272896369136
SEVERE: Trying to contact region server 10.1.3.111:60020 for region
Test5,http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg,1272896369136,
row 'http://hu.gaabi.www/jordania/\x28041022\x29_jord-155_petra.jpg',
but failed after 1 attempts.
Exceptions:
java.net.ConnectException: Connection refused

Which is expected as the 10.1.3.111:60020 regionserver was offline for
hours at that time.
The cause of this problem is that I set hbase.client.retries.number to
1 as I don't like the current retry options.
In this case the following code at HConnectionManager.java:1061
   callable.instantiateServer(tries != 0);
will make scanners to always use the cache.
This makes hbase.client.retries.number = 1 an unusable option.

This is not intentional, am I correct?
Am I forced to use the retries, or is there an other option?

Also I would like to ask, when is it a good thing to retry an operation?
In my experience there exists two kinds of failures
1) org.apache.hadoop.hbase.NotServingRegionException : region is offline
This can be due to a compaction, in which case we probably need to
wait for a few seconds.
Or it can be due to a split, in which case we might need to wait for minutes.
Either case I would not want my client to wait for such long times
when I could reschedule other things to do in that time.
It is also possible that region has been transfered to an other
regionserver but that is rare compared to the other cases.

2) java.net.ConnectException : regionserver is offline
This is solved as soon as the master can reopen regions on an other
regionserver, but still can take minutes.
Anyway this exception is also rare(usually)

Best regards,
Miklos