hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samir Ahmic (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-14431) AsyncRpcClient#removeConnection() never removes connection from connections pool if server fails
Date Mon, 14 Sep 2015 17:44:46 GMT
Samir Ahmic created HBASE-14431:

             Summary: AsyncRpcClient#removeConnection() never removes connection from connections
pool if server fails
                 Key: HBASE-14431
                 URL: https://issues.apache.org/jira/browse/HBASE-14431
             Project: HBase
          Issue Type: Bug
          Components: IPC/RPC
    Affects Versions: 1.1.2, 1.0.2, 2.0.0
            Reporter: Samir Ahmic
            Assignee: Samir Ahmic
            Priority: Critical

I was playing with master branch in distributed mode (3 rs + master + backup_master) and notice
strange behavior when i was testing this sequence of events on single rs: /kill/start/run_balancer
while client was writing data to cluster (LoadTestTool).
I have notice that LTT fails with following:
2015-09-09 11:05:58,364 INFO  [main] client.AsyncProcess: #2, waiting for some tasks to finish.
Expected max=0, tasksInProgress=35
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: BindException: 1 time, 
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
	at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:186)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:951)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:571)
	at org.apache.hadoop.hbase.util.SimpleClient.main(SimpleClient.java:62) 

After some digging  and adding some more logging in code i have notice that following condition
in  {code}AsyncRpcClient.removeConnection(AsyncRpcChannel connection) {code} is never true:
if (connectionInPool == connection) {
causing that  {code}AsyncRpcChannel{code} connection is never removed from {code}connections{code}
pool in case rs fails.

After changing this condition to:
if (connectionInPool.address.equals(connection.address)) {
issue was resolved and client was removing failed server from connections pool.
I will attach patch after running some more tests.

This message was sent by Atlassian JIRA

View raw message